Flight MH128, bound for Kuala Lumpur, returned "due to a disruptive passenger", an airline statement said.
The flight landed safely and the passenger was arrested, police said, adding it was not terror-related.
Police said he threatened passengers with a device but authorities "quickly ascertained" it was not an explosive.
Malaysia's Deputy Transport Minister Abdul Aziz bin Kaprawi described it as a "powerbank", or mobile charger, according to the AFP agency.
The suspect was a 25-year-old Sri Lankan man who was released from a psychiatric facility earlier on Wednesday, said Victoria Police Chief Commissioner Graham Ashton.
The man had been living in Melbourne on a student visa, Mr Asthon said, clarifying earlier information he was an Australian citizen.
Police initially treated it as a possible terrorism incident before determining it was "a case involving a mental health issue", the commissioner said.
One eyewitness, Andrew Leoncelli, told Melbourne's 3AW radio station that 10 minutes into the flight he could "hear this idiot saying that he wanted to go in to see the pilot".
The man refused to sit down, and threatened to "blow the plane up", said Mr Leoncelli, a former professional Australian Rules football player.
"The staff screamed out 'I need some help, I need some help'. So I jumped up, undid my buckle, and approached him," he said.
A photo posted to Instagram by passenger David Henderson: "MH128 boarded by armed police. No one hurt"
Mr Leoncelli said the man ran down the plane towards the back, but was grabbed and disarmed of a black object "the size of a watermelon" which had two antennae and what appeared to be a smartphone input.
The man was restrained, but the object remained on the plane for over an hour until passengers were allowed to leave, he added.
Malaysia Airlines said the Airbus A330 was in the air for just 30 minutes of its eight-hour flight time before landing.
Heavily armed police boarded the plane just before midnight local time (14:00 GMT) and placed the man under arrest.
Passengers were allowed out on to the tarmac after a long wait
"We are obviously concerned for the passengers and crew," said Victoria Police Superintendent Tony Langdon.
"It would have been a very traumatic experience for them."
Mr Ashton defended what local media said was a 90-minute delay in getting the 337 passengers and crew off the plane.
He said authorities had waited until they felt confident there were no explosive devices on board.
"Decisions had to be made about what was the safest way to get passengers off the plane," he said.
Both the airline and Australian authorities will investigate, Malaysia Airlines said.
A number of flights were diverted following the incident, but the airport confirmed it was operating as normal.