He says they "increasingly feel like they are on the wrong side of a profound unfairness".
This is why, he suggests, young people turned out in record numbers to vote in the general election.
They were particularly worried about their finances, job security and housing prospects, he added.
Mr Milburn, who heads the commission that monitors progress towards improving social mobility, made the comments as he launched a new exploration of public attitudes to it.
This new Social Mobility Barometer was based on an in-depth survey of 4,723 UK adults.
Some 51% of the 18- to 24-year-olds polled for the barometer said they thought where people ended up was determined by their background and who their parents were.
This compared with 40% of those polled who were aged 65 and over.
Overall nearly half of people said they felt background determined chances of success.
And four-fifths of those surveyed said there was a large gap between the social classes in Britain today.
Also, although 47% said they were better off financially than their parents, this dropped to 24% for 25- to 49-year-olds.
Meanwhile, just a fifth of 18 to 24-year-olds believed they had a better level of job security than their parents.
Mr Milburn said: "Young people increasingly feel like they are on the wrong side of a profound unfairness in British society - and they are unhappy about it.
"The barometer finds that half of young people think the situation is getting worse, with only 30% of 18- to 24-year-olds believing it is becoming easier to move up in British society."
He added: "The feelings of pessimism young people are expressing are borne out by the facts they are experiencing.
"Those born in the 1980s are the first post-war cohort not to start their working years with higher incomes than their immediate predecessors.
"Home ownership, the aspiration of successive generations of ordinary people, is in sharp decline, among the young especially."
Mr Milburn warned: "Britain's deep social mobility problem, for this generation of young people in particular, is getting worse not better."
The academics' union, the University and College Union, described the poll results as depressing and said young people had seen "their pay fall, the jobs market remain incredibly difficult, tuition fees rocket and support to stay on at college disappear".
And Sir Peter Lampl, chair of the Sutton Trust charity, which promotes social mobility, said: "The commission's barometer should be a wake-up call for policymakers.
"Political rhetoric needs to be translated into real polices to level the playing field and improve opportunities for young people, particularly for those from the most disadvantaged families."