The 350-strong Japanese military contingent, which has been based in South Sudan's capital Juba for the past five years, has been mostly helping with infrastructure construction.
"The first group is leaving today," Daniel Dickinson, spokesman for the U.N. peace keeping mission called UNMISS, told Reuters moments before the troops were due to board their flight at Juba International Airport.
Dickinson said the Japanese contingent would leave in three batches and that the group that was due to leave on Monday had 68 troops.
Deadly violence, often driven by ethnic hatred has gripped South Sudan since 2013 when fighting broke out between forces loyal to President Salva Kiir and his political foe Riek Machar whom he sacked as vice president.
A 2015 peace deal ended that conflict and Machar was early last year restored to his vice presidency position but persistent animus between the two men finally exploded into fresh fighting in July.
Episodes of mass violence especially in the equatorial region of south Sudan have been growing, with surviving civilian witnesses reporting government troops turning up in towns and starting shooting rampages.
The UK has described the killings and other atrocities in South Sudan as genocide although the U.N. has not yet made that determination.
South Sudan government spokesman, Ateny Wek Ateny, said they welcomed Japanese troops' departure because "the government of South Sudan is able to control the country."