Joe Hockey's colleagues and political allies have rounded on the Treasurer, questioning his judgment and the quality of advice he is receiving after a disastrous gaffe suggesting poor people "don't have cars or actually don't drive very far".

Mr Hockey has lost three key staff from his 17-person office in the nine months since the federal election. They include the second and third most senior people in the office, who have left since the budget and were charged with overseeing the budget process and providing crucial political and economic advice.

Fairfax Media contacted more than a dozen of Mr Hockey's supporters in the ministry and party machine; political staff; and staff to former treasurer Peter Costello on Thursday about this week's gaffe. All expressed surprise, concern and dismay about Mr Hockey's remarks.

Furious ministerial colleagues turned on the Treasurer over comments they variously described as "stupid and wrong", a "bad example of how to make a point" and "loose language".

Several people close to Mr Hockey's office, who asked not to be named, said he was taking advice from an increasingly small circle of advisers - particularly chief of staff Grant Lovett and press secretary Mike Willesee.

The loss of the experienced trio of deputy chief of staff Creina Chapman, who moved to the corporate sector soon after overseeing the first budget; economics adviser Tony Pearson, who took a sabbatical; and media adviser Tony Ritchie, who joined the NSW Police media team, suggested power was increasingly concentrated in the hands of too few people.

One observer suggested the office "lacked direction", and that Mr Hockey lacked an adviser with "a hard political edge".

"Joe is freelancing, being dumb with his words, though it's not Rudd office dysfunction," the source said.

A second observer said Mr Hockey "could probably strengthen his office", adding the Treasurer needed a political adviser in the mould of David Gazzard, the hard-nosed former political adviser to Mr Costello.

On Friday, Education Minister Christopher Pyne said he "wasn't going to cop any criticism of Joe Hockey" and said the Treasurer had the government's ''full support''.

He then refused - six times - to back the Treasurer's comments on the fuel excise's impact on poor people.

"The simple fact is that this is the guy who has driven the abolition of the carbon tax, the cost of which fell heaviest on low-income households," he told the Nine network, adding that Mr Hockey had also opposed Labor's move to scrap a self-education tax deduction and the changes to fringe benefit tax changes for cars.

"What he is talking about is the fuel excise changes, which are a 40 cents a week impost on households on average."

But Labor transport spokesman Anthony Albanese swiftly pointed out that Mr Pyne had "had six opportunities to support what he said and you refuse to do so".

"What he said was rubbish, if you are in an outer suburb or a regional community you have no choice but to drive and this is the guy who cut all funding for public transport in the budget," Mr Albanese said.

"He [Mr Hockey] should go to western Sydney today and have a look at people driving all the way in to the city, or all the way into their workplace in any capital city from the outer metropolitan areas, they spend a higher proportion of their income than high income earners."

Mr Hockey's gaffe has capped a difficult three months in which the Treasurer has been under fire for puffing on a cigar and, days later, dancing with his son on budget night before delivering wide-ranging budget cuts, taking a holiday to Fiji during a key parliamentary sitting, complaining that everyone in the media was against him and co-operating with a biography that revealed he wanted an even tougher budget.

The North Sydney MP went on a radio blitz on Thursday in an attempt to limit the fallout from the gaffe.

"The fact of the matter is that I can only get the facts out there and explain the facts; how people interpret them is up to them," he told Fairfax Radio station 2UE.

Asked if he realised if his comments sounded callous, Mr Hockey said: "I'm sorry if that's the case but the fact is that the Labor Party says that it's an unjust initiative, unfair initiative, higher income people aren't paying enough, well here is an initiative where higher income people pay on average three times the amount of lower income households in the fuel excise."

Mr Hockey's claim that higher-income households pay more in fuel tax is correct in absolute terms but the claim the tax is progressive has been debunked by research from the Parliamentary Library, the Grattan Institute and consumer group One Big Switch that shows poorer people spend a higher proportion of their income on fuel.

South Australian Liberal Cory Bernardi said the Treasurer's comments were a "distraction" and "we don't need distractions".

"Those in the lower socio-economic group tend to spend more, as a percentage of their income, on transport and the basic necessities of life than do those who are wealthier."

Queensland Liberals Teresa Gambaro and Ian Macdonald and NSW Nationals senator John Williams also took thinly veiled swipes at the comments.

The federal opposition used the three-month anniversary of Mr Hockey's unpopular first budget, much of which remains blocked by the Senate, to step up its political attack on the Treasurer.

Treasury spokesman Chris Bowen labelled the comments "insulting to the intelligence of the Australian people".

With Latika Bourke

Follow us on Twitter