The fund was originally suggested by a cross-party group of MPs on the Northern Ireland Affairs Committee.
They have described the government's response as unacceptable.
Former Libyan leader Colonel Muammar Gaddafi supplied arms to the IRA and, during the Troubles, Libyan Semtex plastic explosive was a key weapon in the group's bombing campaign.
MPs on the Northern Ireland Affairs Committee, which investigated the issue for many months during the last parliament, originally called for compensation for victims.
They also wanted a special fund set up and Libyan assets in the UK frozen.
'Tea and sympathy'
Conservative and Labour Party politicians sit on the committee, along with DUP MPs Gregory Campbell, Jim Shannon, Ian Paisley and the independent MP Lady Hermon.
The Northern Ireland Secretary, James Brokenshire, said compensation claims were a "private" matter.
Establishing such a fund was not a viable option, the government added.
Muammar Gaddafi was toppled as Libya's leader in 2011
Mr Paisley, the MP for North Antrim, said he had had enough of "tea and sympathy".
If the government was so convinced about payments eventually coming from the Libyan authorities, the victims should receive "payments in lieu", he said.
"I note the ongoing dialogue ministers have with Libyan counterparts on the issue, which is welcome," he said.
"I now invite them to formally update the committee on a regular basis until the matter is resolved."
The Northern Ireland Affairs Committee also includes Labour MPs Kate Hoey and Conor McGinn, who are both from Northern Ireland.
Ms Hoey, a long-serving member of the committee who has campaigned for victims' compensation, said the government response was "as unsurprising as it is unacceptable".
Kate Hoey described the government's response as unacceptable
"They are telling people to seek justice on their own, to bear the cost and overcome the language barrier of obtaining compensation directly from the Libyan government," she said.
"There is a duty to represent the victims, just as the US and German governments fought for compensation for their citizens."
According to the committee, the government does not consider it to be in the UK national interest to use political or financial support to Libya as leverage to secure compensation for victims.
'Position of cowardice'
Kenny Donaldson, of the campaign group Innocent Victims United, said the government's "dismissive" response was typical of its attitude towards terrorist victims.
"This position is the position of cowardice and is both politically and morally indefensible," he said.
"The UK government repeatedly pushes its chest out when terror attacks happen - there is the pomp and ceremony, there are the strong and well crafted words of repudiation of that terrorism and violence at the time.
"But what happens after for those left devastated?
"The answer is very little."
A government spokesperson said: "The government notes the potential challenges of making additional UK resources available specifically to the victims of Gaddafi-sponsored IRA terrorism, which would need to be considered carefully against government support to victims of terrorism more generally, including in Northern Ireland."
The government also said the issue of compensation has been raised with the Libyan authorities at the highest level.