The strike was in support of Afghan troops fighting insurgents who attacked Kabul airport with rockets shortly after US Defence Secretary Jim Mattis arrived on a visit.
It is not clear how many civilians were killed or injured.
The US recently confirmed it would send 3,000 extra troops to Afghanistan.
"Tragically, one of the missiles malfunctioned, causing several casualties," the Nato mission in Afghanistan, known as Resolute Support, said in a statement.
"Resolute Support deeply regrets the harm to non-combatants. An investigation into the attack and the malfunctioning ammunition has begun."
The rocket attack on the airport was claimed by both the Taliban, which said they had targeted Gen Mattis's plane, and their rival, the Islamic State group.
Gen Mattis held talks with Nato chief Jens Stoltenberg and Afghan President Ashraf Ghani about plans to strengthen Afghanistan's military.
Support from US-led Nato troops would give Afghan forces a "compelling battlefield advantage over anything the Taliban stands to mass against" it, Gen Mattis said.
He said the US would not allow "a merciless enemy to kill its way to power".
US combat operations against the Taliban officially ended in 2014, but more than 8,000 US special forces remain in the country supporting Afghan troops.
Sixteen years after the US-led invasion, the Afghan government still only controls about 60% of the country.