The song emerged last week and has been widely shared on social media.
One line in it asks: "Is there still freedom of expression in the country?" and refers to a "doctor" who can't tolerate criticism.
The information minister said President John Magufuli "loves" the song.
He also suggested that it be "improved" to take on other issues, such as tax evaders, corrupt businesspeople and drug traffickers, said Information Minister Harrison Mwakyembe.
Nay wa Mitego's arrest came days after President Magufuli warned media to be careful about exercising freedom.
"Media owners, let me tell you: 'Be careful. Watch it. If you think you have that kind of freedom - not to that extent,'" Mr Magufuli said.
Police earlier said the rapper would be questioned "for releasing a song with words that malign the government".
He was reportedly held at a hotel in Morogoro, before being transferred the 190km (120 miles) to the commercial capital Dar es Salaam.
Critics say Mr Magufuli is showing he is intolerant of dissent
Other lines in the song said: "Who are you now? Don't you want to listen to advice? Don't you want criticism?"
The lyric is addressed to someone he calls "a doctor specialising in lancing boils".
The phrase "lancing boils" has been used repeatedly by Mr Magufuli to refer to people he deems obstructive.
Mr Magufuli, who came to power in 2015, presents himself as a no-nonsense man of the people, unafraid of sacking corrupt or incompetent officials.
Recent opinion polls have suggested he is popular, but critics accuse him of an impulsive style of governing and being intolerant of dissent.
Another line in Nay wa Mitego's song goes: "I see you handed a club to a madman," which has been interpreted as referring to an official appointed by the president who has been accused of heavy-handed behaviour.
The BBC's Sammy Awami in Dar es Salaam says the rapper is popular for his scathing lyrics, attacking not only the government but even his fellow artists.
On his social media accounts he uses the name "NayTrueboy", saying he always speaks his mind and tells the truth.