Tony Abbott's hand-picked Indigenous advisers Noel Pearson and Warren Mundine have led a blistering backlash to the Prime Minister's claim that remote communities are "lifestyle choices," describing the remark as "wrong", "disappointing" and "hopeless".

But ministers including Malcolm Turnbull have closed ranks around the Prime Minister, with the undeclared rival to Mr Abbott asking that the comments not be used to generate another "let's-give-Tony-Abbott-a-belting occasion".

Mr Abbott sparked the fury on Tuesday when in response to a question about taxpayers subsidising Indigenous Australians who choose to live in remote communities, the Prime Minister described them as "lifestyle choices".

The remark was immediately condemned as overlooking the cultural connection Indigenous Australians have to their land as well as the Native Title implications if they move. It follows Mr Abbott's comment last July in which he described Australia as "unsettled" until the British arrived.

Mr Abbott was being asked about funding cuts to remote communities in Western Australia and if they amounted to penalties for Indigenous Australians living in remote townships in an interview with the ABC in Kalgoorlie.

"It's the responsibility of every Australian adult to look for work if you are capable of work, and, yes, while you are looking for work, the government will pay you unemployment benefits, but what we can't do is endlessly subsidise lifestyle choices if those lifestyle choices are not conducive to the kind of full participation in Australian society that everyone should have," Mr Abbott said.

Lawyer and activist Mr Pearson described the statement as "disappointing and hopeless".

"I'm just bitterly disappointed to hear this deranged debate go on in the sub-standard manner in which it's being conducted," he said.

"For this debate to be conducted in such a off-the-cuff manner through policy thought bubbles is, I think, a disgraceful turn of events," Mr Pearson told the ABC.

Mr Mundine, who chairs the Prime Minister's Indigenous Advisory Council, said Mr Abbott's remark overlooks the cultural connection First Australians have with the land.

"That is a complete misconception of what it is and he's wrong in that regard," Mr Mundine said. "It is not about a lifestyle, it is not like retiring and moving for a sea change. It is about thousands of years' connection, their religious beliefs and the essence of who they are."

Mr Mundine said native title laws were also an impediment to people moving out of indigenous communities because it would make it more difficult to prove their connection to the land.

He also said there were ways to overcome educational barriers, including sending children to regional boarding schools during the week.

Mr Mundine said he was frustrated by the Prime Minister's gaffes in the area of Indigenous affairs because, of all the prime ministers and politicians he had worked with, he found Mr Abbott to be the one most genuinely committed to real change for indigenous Australians.

"[Mr Abbott and John Howard] seem to be men trapped in their time but that's not to say that [Mr Abbott's] not one of the most generous and committed prime ministers in this area," Mr Mundine said.

The pair's scathing criticism is particularly devastating for Mr Abbott who wants to be known as the "Prime Minister for Aboriginal Affairs" given Mr Pearson and Mr Mundine are his hand picked advisers on the issue.

"We have been mates for a long time and we are of like minds when it comes to most Indigenous issues," Mr Abbott said of Mr Mundine in 2013

"I am convinced that Indigenous policy and administration will be so much better if it is closely informed by the real, lived experience of indigenous people."

Mr Abbott has also cited Mr Pearson whom he describes as his "friend" in saying the Government needs to "empower" Indigenous communities.

"We need to empower communities, but we can't really empower communities without also empowering the individuals, the people, the families that make them up," Mr Abbott said in 2013.

But the Prime Minister was unrepentant in the face of the new backlash and said he was "very comfortable" with his "credentials" in Indigenous policy.

"All Australians are free to live where we choose but inevitably there are some limits to what we can reasonably expect of the taxpayer to support these choices," he said.

"I don't think there are too many Australian prime ministers who took the national government for almost a week to a remote Aboriginal community. I did it last year, it is my intention to do it again this year," he told reporters in Mt Gambier.

While previous missteps from the Prime Minister have been used to undermine his leadership and judgment, ministerial colleagues launched a strong defence on Wednesday. Undeclared leadership rival Malcolm Turnbull said: "It's important to put that issue on the table, talk about it rationally without turning it into a let's-give-Tony-Abbott-a-belting occasion, as often people like to do."

Treasurer Joe Hockey said Mr Abbott's comments were neither wrong nor insensitive. "Don't be hard on him about it because it is the life that people are having that we are trying to deal and what they choose to have as well," he told the ABC.

Cabinet minister Christopher Pyne said Labor's demands that Prime Minister apologise were "bizarre and hysterical."

Trade Minister Andrew Robb Mr Abbott's remark was "certainly not racist," as has been claimed by Greens Senator Rachel Siewert.

Parliamentary Secretary to the Prime Minister on Indigenous Affairs Alan Tudge said while connection to the land "is very important for many people," it's hard to maintain a "strong culture if people aren't educated".

Indigenous Labor senator Nova Peris said Mr Abbott had showed his "true colours" with his remarks and exposed himself as an "insincere pretender". "He is effectively saying Aboriginal people's personal and spiritual beliefs and the decisions they make don't matter," Senator Peris told Fairfax Media.

"In the Territory, Aboriginal people call him 'The Gammon Man,'" she said. In the Northern Territory the term "gammon" means something that is false or not serious.

Opposition Leader Bill Shorten said Mr Abbott was stuck in the 1950s and wanted to move indigenous Australians off their land. The Liberal Indigenous West Australian MP Ken Wyatt was contacted for comment but did not return calls.