Speaking at a security conference in Singapore, he said such moves undermined regional stability.
China's territorial claims in the resource-rich South China Sea are contested by several nations.
At the same time, Gen Mattis praised Beijing's efforts to restrain North Korea's missile and nuclear activity.
His comments came shortly after the UN Security Council expanded targeted sanctions against North Korea in response to a series of missile tests conducted this year.
The council voted unanimously to back the sanctions after weeks of negotiations between the US and China.
In his speech at the annual Shangri-La Dialogue forum, Gen Mattis said: "We oppose countries militarising artificial islands and enforcing excessive maritime claims.
"We cannot and will not accept unilateral, coercive changes to the status quo."
President Donald Trump and other senior US officials have repeatedly stated that they would protect its interests in the South China Sea, a key shipping route.
During his nomination hearing earlier this year, Secretary of State Rex Tillerson warned that the US was "going to have to send China a clear signal that first the island-building stops, and second your access to those islands also is not going to be allowed."
In response, the Chinese foreign ministry said Beijing would "remain firm to defend its rights in the region".
Gen Mattis sought to reassure that America was not turning its back on Asia
But in Singapore Gen Mattis also struck a positive note on US-China relations, saying that while competition between the two countries "is bound to occur, conflict is not inevitable".
The biggest question amongst Asian delegates attending the forum has been how much of a role the US will continue to play in this increasingly tense region, the BBC's Karishma Vaswani in Singapore reports.
She adds that Gen Mattis sought to reassure his peers that the US was not turning its back on Asia.
What is the South China Sea dispute?
Rival countries have wrangled over territory in the South China Sea for centuries, but tension has steadily increased in recent years.
Its islets and waters are claimed in part or in whole by Taiwan, China, Vietnam, the Philippines, Malaysia and Brunei.
Beijing has been building artificial islands on reefs and carrying out naval patrols in waters also claimed by these other nations.
Although the previous US administration of Barack Obama insisted it was neutral, it spoke out strongly against the island-building and sought to build ties with, and among, the South East Asian nations whose claims overlap those of China.
In July 2016, an international tribunal ruled against Chinese claims, backing a case brought by the Philippines, but Beijing said it would not respect the verdict.
The frictions have sparked concern that the area is becoming a flashpoint with global consequences.