An agreement at 27 fell through after British Prime Minister David Cameron demanded concessions that Germany and France were not willing to give, one of the officials said.
The decision means Britain could now be left outside the tent as up to 25 EU member states - not including Denmark which has an opt-out from the euro -- push ahead with deeper integration, including much tighter debt and deficit rules among the euro zone countries, the diplomats said.
Germany and France want to change the EU treaty to enforce much stricter rules among the euro zone states and others due to join, with the aim of preventing a repeat of the debt crisis afflicting the single currency area for the past two years.
While Britain could still reverse its position, such a move could expose Cameron to criticism at home, with a strong eurosceptic stream in his Conservative party determined that Britain should take a tough line with Europe and win back powers that critics say have been surrendered to Brussels.
The danger for Cameron is that if up to 25 countries do push ahead with deeper integration, it could involve discussions over changes to the single market and financial regulation, both of which could have a profound impact on the British economy.
"We've always said we would do it at 17 if it didn't work at 27. That's what happened," one senior EU diplomat said.
Another added that a new treaty would likely involve the 17 euro zone states plus the eight non-euro zone countries that are expected to join the single currency in the future.
"Cameron was clumsy in his manoeuvering," another senior EU diplomat said.
British diplomats were not immediately available to comment.
As well as discussing treaty change during more than 10 hours of talks on the first day of a two-day summit, EU leaders also debated steps to strengthen their financial resources to tackle the debt crisis.
One diplomat said the leaders had agreed that the euro zone's permanent bailout fund, the European Stability Mechanism, would have a capacity capped at 500 billion euros, rather than earlier expectations that it could top that figure.
It was also agreed that the ESM would not be granted a banking licence, as had originally been proposed by European Council President Herman Van Rompuy, the diplomat said.
The leaders also agreed to explore the idea of providing bilateral loans to the International Monetary Fund totalling 200 billion euros, with 150 billion of that coming from the euro zone, to bolster IMF resources to tackle Europe's debt crisis.