Washington (CNN) -- It was a common refrain during the House and Senate late-night votes to avert the fiscal cliff. Senator after senator, congressman after congressman lamented the fact that the legislation didn't "do more," "go bigger" or that it was "far from perfect."
Political watchers believed the fiscal cliff negotiations were the perfect time for President Barack Obama and House Speaker John Boehner to hatch a "grand bargain" -- a deal that would have included both large increases in tax revenue and major cuts in government spending. At the time, both men looked better positioned to deliver a bipartisan plan. Boehner seemed to have a firmer hand on his caucus leading up to the talks, and the president was coming off a hard-fought re-election win.
Those hopes, however, proved empty.
What passed both the House and Senate was noticeably scaled back and far from grandiose.
This is not the first time a grand bargain has fallen apart. Obama and Boehner worked together in an effort to negotiate a path toward fiscal health in 2011 -- another so-called "grand bargain" -- but the efforts ultimately ended in acrimony. In an interview with CNN's Jessica Yellin, Boehner described it as his "greatest disappointment."
With two grand bargain failures between the president and the speaker, is it really possible for Congress to go big and pass meaningful, bipartisan legislation as the nation approaches hitting its debt ceiling? Members of Congress and political watchers alike say the prospect seems bleak. A combination of lawmakers lacking courage, the threat of primary challenges and nobody wanting to put skin in the game makes it incredibly difficult to get a big, bipartisan deal done.
According to Nathan Gonzales, deputy editor of The Rothenberg Political Report, the biggest deterrent to "going big" is the next election.
"With anything big, members will get a lot of what they like and also a lot that they don't like," Gonzales said. "And in the realm of political campaigns, which focuses on the negative, it is easy to put the negative parts of a bill into a campaign ad and get a lot of attention."
The last few elections have proven this. Outside political groups have, in many situations, funded political hopefuls to challenge congressional incumbents. A whopping $293.5 million in outside money was spent in 2012 congressional races, according to the Federal Election Commission. In the 2010 congressional races, according to the FEC, $153 million was spent by outside groups, and 54 incumbents lost re-election, many of them Democrats.
One difficult vote "doesn't create primary defeat, but it could cause an outside group to get involved, to put the financial backing behind a challenger and all of a sudden make the member's electoral life more miserable," Gonzales said. "This is a fairly new phenomenon. The kind of immediacy with which someone can come out of nowhere, be funded by an outside group and get to a high stage is much quicker than before."
Doubt in getting a grand bargain done is not limited to those on the outside, though. Retiring Republican Rep. Steve LaTourette, R-Ohio, agreed that a grand bargain agreement is not doable.
"No one has the courage to go big because when you go big you are going to make some people mad," LaTourette said shortly before Tuesday night's vote to avert the fiscal cliff. "If I was a Democrat, once you start to go big, the AARP, AFL-CIO is going to come after you, and when you are a Republican, the tax people come after you."
LaTourette has a history with trying to go big. He was the key Republican advocate for the Simpson-Bowles deficit reduction and budget plan, which ultimately failed in the House. He even proposed bringing the plan to the floor at a Republican conference meeting Tuesday afternoon.
Sen. Dick Lugar, R-Indiana, knows this all too well.
Indiana Republican voters kicked Lugar out of office in the party primary in May after 36 years in the Senate. Richard Mourdock, a longtime GOP operative, benefited from manpower and money from tea party activists and outside groups determined to topple the six-term veteran.
Lugar was attacked over questions about whether he even lived in Indiana, for his seniority and for his penchant for bipartisanship. The Club for Growth, a conservative pro-growth group, perhaps played the biggest role, releasing a barrage of ads that framed Lugar as out of step with his own party. The group endorsed Mourdock in mid-February and spent close to $1.5 million on television ads -- close to what Lugar's own campaign spent on TV spots. But it also collected more than $300,000 from donors and gave it to the Mourdock campaign, making their total spending on the race close to $2 million.
Lugar, who voted for Obama's economic stimulus package and the bank and financial institution bailout in 2008, proved that while one tough vote may not hurt you, a few put together can cause headaches.
"The bottom line is a grand bargain or going big would require compromise and the most influential outside groups on both sides are not fans of compromise," Gonzalez said. "Those two factors are at odds with each other."
Rep. Chris Van Hollen, D-Maryland, though hopeful that it is still possible, acknowledged that there are major impediments to big, bipartisan legislation.
"If everybody can give a little bit, then you can get an agreement," Van Hollen said. "The whole idea behind that is shared responsibly. If everybody has got skin in the game, then you can do a large agreement."
CNN's Ashley Killough, Dana Bash and Deirdre Walsh contributed to this report.