Mr Gordhan will be replaced by Malusi Gigaba, said a statement issued late on Thursday by the president's office.
Earlier this week, President Zuma recalled Mr Gordhan from planned events in the UK.
Sfiso Buthelezi will become Deputy Finance Minister, replacing Mcebisi Jonas.
President Zuma also made a number of other changes in the cabinet.
"I have directed the new ministers and deputy ministers to work tirelessly with their colleagues to bring about radical socioeconomic transformation and to ensure that the promise of a better life for the poor and the working class becomes a reality," President Zuma's statement said.
Last October, Mr Gordhan was charged with fraud - but the charges were later dropped.
He has described the allegations as politically motivated.
Pressure has been growing on President Jacob Zuma to step down
Mr Gordhan has been seen as standing up to President Zuma in cabinet and has warned against corruption becoming rampant.
The South African Communist Party, an ally of the governing African National Congress, had earlier lodged a formal objection to plans to dismiss Mr Gordhan, who is widely respected internationally.
Many senior ANC figures also opposed the finance minister's removal.
Meanwhile, the Democratic Alliance, South Africa's main opposition party, said it would call a vote of no-confidence in President Zuma in parliament.
Mr Gordhan's recall from the UK caused South Africa's rand to lose nearly 5% of its value against major currencies earlier this week.
Pressure has been growing on Mr Zuma to step down amid numerous allegations of corruption.
He denies the claims.