The "Yes" votes stood at 51.7 percent after 95 percent of ballots had been opened, state-run Anadolu news agency said, with the lead narrowing in the final stages of an increasingly tight count.
A "Yes" vote would replace Turkey's parliamentary democracy with an all-powerful presidency and may see Erdogan in office until at least 2029, in the most radical change to the country's political system in its modern history.
The outcome will also shape Turkey's strained relations with the European Union. The NATO member state has curbed the flow of migrants - mainly refugees from wars in Syria and Iraq - into the bloc but Erdogan says he may review the deal after the vote.
In Turkey's three biggest cities - Istanbul, Izmir and the capital Ankara - the "No" camp appeared set to prevail narrowly, according to Turkish television stations.
Speaking to reporters in Ankara, Deputy Prime Minister Veysi Kaynak said that the "Yes" camp had not won as many votes as expected, but was still ahead nationwide.
Earlier in the day a crowd chanted "Recep Tayyip Erdogan" and applauded as the president shook hands and greeted people after voting in a school near his home in Istanbul. His staff handed out toys for children in the crowd.
"God willing I believe our people will decide to open the path to much more rapid development," Erdogan said in the polling station after casting his vote.
"I believe in my people's democratic common sense."
The "Yes" percentage of the vote - which stood at 63 percent after around one quarter had been opened - eased as the count came further west towards Istanbul and the Aegean coast. Broadcaster Haberturk said turnout was 86 percent.
The opposition People's Republican Party (CHP) said a last-minute decision by the electoral board to accept unstamped ballots as valid votes put the vote in question.
"We will pursue a legal battle. If the irregularities are not fixed, there will be a serious legitimacy discussion," CHP deputy chairman Bulent Tezcan said.
The referendum has bitterly divided the nation. Erdogan and his supporters say the changes are needed to amend the current constitution, written by generals following a 1980 military coup, confront the security and political challenges Turkey faces, and avoid the fragile coalition governments of the past.
"This is our opportunity to take back control of our country," said self-employed Bayram Seker, 42, after voting "Yes" in Istanbul.
"I don't think one-man rule is such a scary thing. Turkey has been ruled in the past by one man," he said, referring to modern Turkey's founder Mustafa Kemal Ataturk.
Opponents say it is a step towards greater authoritarianism in a country where some 47,000 people have been jailed pending trial and 120,000 sacked or suspended from their jobs in a crackdown following a failed coup last July, drawing criticism from Turkey's Western allies and rights groups.
"I voted 'No' because I don't want this whole country and its legislative, executive and judiciary ruled by one man. This would not make Turkey stronger or better as they claim. This would weaken our democracy," said Hamit Yaz, 34, a ship's captain, after voting in Istanbul.
Relations between Turkey and Europe hit a low during the referendum campaign when EU countries, including Germany and the Netherlands, barred Turkish ministers from holding rallies in support of the changes. Erdogan called the moves "Nazi acts" and said Turkey could reconsider ties with the European Union after many years of seeking EU membership.
Kurdish militants overnight killed a guard in an attack on a vehicle carrying a district official from Yildirim's ruling AK Party in southeast Turkey's Van province, security sources said.
They said Kurdistan Workers Party (PKK) militants staged the attack in Van's Muradiye district. A second of the AKP official's guards was wounded and 17 people were detained over the attack, they added.
In the southeastern Diyarbakir province, two people were killed and one wounded in a gunfight in a village schoolyard which was being used as a polling station, other security sources said. The cause of the clash was not immediately clear.
On Saturday, Erdogan held four rallies in Istanbul, urging supporters to turn out in large numbers and saying it "will be a turning point for Turkey's political history".
Erdogan and the AK Party enjoyed a disproportionate share of media coverage in the buildup to the vote, overshadowing the secular main opposition CHP and the pro-Kurdish Peoples' Democratic Party (HDP).
CHP leader Kemal Kilicdaroglu has accused Erdogan of seeking a "one-man regime", and said the proposed changes would put the country in danger.
Proponents of the reform argue that it would end the current "two-headed system" in which both the president and parliament are directly elected, a situation they argue could lead to deadlock. Until 2014, presidents were chosen by parliament.
The government says Turkey, faced with conflict to the south in Syria and Iraq, and a security threat from Islamic State and PKK militants, needs strong and clear leadership to combat terrorism.
The package of 18 amendments would abolish the office of prime minister and give the president the authority to draft the budget, declare a state of emergency and issue decrees overseeing ministries without parliamentary approval.