AFP reports citing officials.
The latest round of talks in Brussels -- the eighth and last, EU foreign affairs chief Catherine Ashton said afterwards -- was aimed at normalising relations between Serbia and breakaway Kosovo, paving the way to eventual EU membership.
But after more than 12 hours of talks, the two sides left without striking any deal.
"The gap between the two sides is very narrow but deep," Ashton, who chaired the talks, said in a statement.
"This is the last time we will meet formally," she added, saying "a number of proposals were put on the table."
Serb Prime Minister Ivica Dacic and his Kosovo counterpart Hashim Thaci will now return home for consultations "and will let me know in the next few days of their decision."
Ashton gave no details of the proposals discussed and did not explain what decision she expected from the two sides.
"I wish them a good journey home and every possible success in reaching a conclusion," she said in her brief statement.
Serbia's Dacic said separately that despite the lengthy meeting, "we do not have an agreement at this moment."
Holding out the prospect of further exchanges, however, Dacic told Serb media: "We still have a certain amount of time ahead to reach the solution."
According to the website of Serbian state broadcaster RTS, Kosovo Premier Thaci said it would be "possible to continue talks next week if Serbia agrees with our principles."
It was not immediately clear if he expected these to be under EU auspices, while Ashton's remarks suggested that no formal meeting in Brussels is envisaged.
The main sticking point in the talks, which got under way late last year, has been the future of the Serb minority living in Kosovo, especially the north, since it declared independence in 2008.
Majority ethnic Albanian Kosovo has since then won recognition from around 100 countries, including the United States and most EU member states.
Serbia and Serbs living in Kosovo continue to reject Pristina's declaration of independence, with the focus on the some 40,000 Serbs in the north.
Both sides earlier agreed to establish an association of Serb municipalities, but Kosovo refuses to grant it the executive and judicial powers demanded by Serbia.
Serbian President Tomislav Nikolic on Monday said recent statements from Kosovo were "not encouraging".
"We must ensure the rights (for the association) are guaranteed not only by an agreement between Belgrade and Pristina, but also by the international community, primarily through the EU in its role as a mediator," Nikolic said.
Kosovo fears in turn that a large degree of autonomy for the north runs the risk the area would eventually break away.
Progress at the talks had been seen as essential to keeping the momentum going, with Ashton saying Monday she expected them to be "conclusive".
Washington meanwhile stressed the importance for Kosovo and Serbia to "move forward both in democratic terms, in economic terms and on their path for European integration."
The European Commission is due to issue a report on the negotiations on April 16, which will be forwarded to EU leaders for consideration at their end-June summit.
Serbia was hoping that once a deal was agreed, it would be given a date to start EU accession talks at the summit in June, while the European Commission has said Kosovo could get an agreement on an association accord at the same time.