Her conservative CDU/CSU bloc has seen its worst result in almost 70 years but will remain the largest in parliament.
Its current coalition partner, the social democratic SPD, says it will go into opposition after historic losses.
The nationalist AfD has won its first seats and is set to be the third party, a result that sparked some protests.
Dozens of demonstrators gathered outside the right-wing, anti-Islam party’s headquarters in Berlin, some with placards saying “Refugees are welcome”.
Protests were also held in Frankfurt and Cologne.
While her alliance has remained the largest party, it is the worst result for the alliance between the Christian Democrat (CDU) and the Christian Social Union (CSU) since 1949, when national elections were held in Germany for the first time after World War Two.
Addressing supporters, Mrs Merkel, who has been in the job for 12 years, said she had hoped for a “better result”.
She added that she would listen to the “concerns, worries and anxieties” of voters of the Alternative for Germany (AfD) in order to win them back.
Mrs Merkel also said her government would have to deal with economic and security issues as well as addressing the root causes of migration – one of the main reasons behind the AfD’s result.
“Today we can say that we now have a mandate to assume responsibility and we’re going to assume this responsibility calmly, talking with our partners of course.”
The result is disastrous for Mrs Merkel, BBC Berlin correspondent Jenny Hill says.
The chancellor is being punished, our correspondent adds, for opening Germany’s door to almost 900,000 undocumented refugees and migrants.
The Social Democrats (SPD), led by Martin Schulz, also had its worst election result since 1949. He said the outcome meant the end of the “grand coalition” with Mrs Merkel’s alliance.
“It’s a difficult and bitter day for social democrats in Germany,” Mr Schulz told supporters. “We haven’t reached our objective.”
With the possibility of an alliance with the SPD rejected, Mrs Merkel’s options are narrow, and the process of forming a new coalition could take months.
Six parties will be in the German parliament for the first time since the 1950s.
The most likely scenario is a “Jamaica” coalition, so-called because of the colours of Jamaica’s flag. It includes the black CDU/CSU, the yellow, business-friendly Free Democrats (FDP) – who are returning to parliament after a four-year hiatus – and the Greens.
It is not a marriage made in heaven, as the Greens want to phase out 20 coal-fired power plants and the FDP disagree, but it is the only formation that would guarantee enough seats in the new Bundestag, German broadcaster ZDF says.
All parties have rejected working with the AfD.
The party has capitalised on a backlash against Mrs Merkel’s policy towards migrants and refugees, many of them from war-torn, mainly Muslim countries like Syria.
Its programme is heavily anti-immigrant, and particularly anti-Islam. It called for a ban on minarets and considered Islam incompatible with German culture.
Prominent AfD figure Frauke Petry said on Twitter (in German) that Germany had experienced an incomparable “political earthquake”. The party’s performance was better than forecast in opinion polls.
Meanwhile, Beatrix van Storch, one of the party’s leaders, told the BBC that the result would change the political system in Germany, giving “a voice” to the people she said were not represented in the last parliament.
“We’ll start debates on migration, we’ll start debates on Islam, we’ll start debates on ever closer union.”
AfD calls for stricter asylum rules to curb abuse of the system, including vetting of claims in countries of origin that are deemed “safe”.