Theresa May was plunged into a new crisis on Saturday night after the government’s social mobility adviser revealed he and his team were quitting, warning the prime minister was failing over her pledge to build a fairer Britain.

In a major blow to No 10, Alan Milburn, the former Labour cabinet minister who chairs the government’s social mobility commission, said he and all three of his fellow commissioners were walking out including leading conservative Gillian Shephard.

The move will be seen as a direct challenge to May’s Downing Street vow to place fairness and social justice at the heart of her premiership.

In his resignation letter, seen by the Observer, Milburn warns that dealing with Brexit means that the government does not seem to have the necessary bandwidth to ensure the rhetoric of healing social division is matched with the reality.

I have little hope of the current government making the progress I believe is necessary to bring about a fairer Britain, he tells the prime minister. It seems unable to commit to the future of the commission as an independent body or to give due priority to the social mobility challenge facing our nation.

The resignations come with the prime minister already under pressure, as she faces crunch Brexit talks and questions over the future of her most senior minister, Damian Green.

Milburn warned that failing to deal with the inequalities that fuelled the Brexit vote would simply lead to a rise of political extremes. In a devastating assessment of the lack of progress, Milburn said: The worst position in politics is to set out a proposition that you’re going to heal social divisions and then do nothing about it. It’s almost better never to say that you’ll do anything about it.

It’s disappointing at least that the government hasn’t got its shoulder to the wheel in the way it should to deal with these structural issues that lead to social division and political alienation in the country.

In America for 30 years real average earnings have remained flat. Now here the chancellor is predicting that will last for 20 years. That has a consequence for people, but a political consequence as well. It means more anger, more resentment and creates a breeding ground for populism.

It is understood that Shephard, the former Tory education secretary and deputy chair of the commission, will also resign.

She is said by friends to be absolutely livid with the way in which the commission has been treated. The social mobility commission, set up by Nick Clegg under the coalition government, advises ministers on the issue and monitors progress. Its most recent report last week warned of a striking geographical divide, with London and its surrounding areas pulling away while many other parts of the country are left behind.

In his resignation letter, Milburn states that, while his term as chair had now come to an end, he had decided to walk away rather than reapply for the role. He said that a repeated refusal to properly resource and staff the commission, an obsession with Brexit and an absence of policy had led to his decision.

The other remaining commissioners Paul Gregg, director of the centre for analysis of social policy at the University of Bath, and David Johnston, the chief executive of the Social Mobility Foundation – are also understood to be quitting.

The dramatic resignations will sting the prime minister, who used her first speech in Downing Street after taking office to vow to tackle social injustice and inequality. When it comes to opportunity, we won’t entrench the advantages of the fortunate few, we will do everything we can to help anybody, whatever your background, to go as far as your talents will take you, she said.

Milburn’s letter to the prime minister states: I do not doubt your personal belief in social justice, but I see little evidence of that being translated into meaningful action.

The need for political leadership in this area has never been more pressing. Whole communities and parts of Britain are being left behind economically and hollowed out socially. The growing sense that we have become an ‘us and them’ society is deeply corrosive of our cohesion as a nation.

The 20th-century expectation that each generation would do better than the last is no longer being met. At a time when more and more people are feeling that Britain is becoming more unfair rather than less, social mobility matters more than ever.

Milburn said that key posts on the commission had gone unfilled for years, while the number of commissioners had been allowed to dwindle from 10 to four. Attempts to hire new commissioners failed. He said protracted discussions about the commission’s role, remit and resources remained unresolved.

It reflects wider concerns in Whitehall that the size of the Brexit policy challenge is preventing meaningful work on pressing domestic issues.

While Milburn’s resignation letter praises Justine Greening, the education secretary, for having shown a deep commitment to the issue, it adds: It has become obvious the government as a whole is unable to commit the same level of support.

Milburn said that last month’s budget had been a damp squib, adding that urgent action was needed to help stagnant wages and close the attainment gap in education between poor students and their more affluent classmates.

Asked if his resignation was a political act to damage the Conservatives, Milburn, a former ally of Tony Blair, said: I’ve served Gordon Brown on this, and he was hardly a political ally; I’ve served David Cameron on this, and he was hardly a political ally; and now I’ve served Theresa May. The issue is bigger than party politics.