By Barbara Starr and Larry Shaughnessy
WASHINGTON (CNN) - Defense Secretary Leon Panetta was packed up and ready, as he says, to "get the hell out of town." But it looks like he can't step aside just yet from the top job at the Pentagon.
The Senate on Thursday dashed Panetta's hopes of quickly confirming his intended replacement, Chuck Hagel, before the start of a one-week congressional recess.
Proponents failed to muster enough support in a procedural vote to end debate on Hagel's appointment and push the nomination toward a concluding vote.
Democrats are calling Republican opposition a filibuster, while GOP members say they simply want more time to address concerns.
Hagel's confirmation process has been rocky and some Republicans still want answers about his finances. Others are demanding more information from the White House on an unrelated issue – the deadly terror attack last September on the U.S. diplomatic mission in Benghazi, Libya.
Panetta is eager to step aside and return to California following a half-century in public service, much of it at the highest levels of the U.S. government. He's been leading the Pentagon for most of the past two years.
"My office is packed up. Sylvia is packing at home. I'm ready to go," Panetta said at a ceremony on Thursday honoring former Secretary of State Hillary Clinton.
"The second best Valentine's present would be to allow Sylvia and I to get the hell out of town at the end of the day," Panetta joked.
While Panetta still planned to head home for the weekend, he's going to be on the job for a bit longer.
"Secretary Panetta will remain secretary of defense and will maintain all his authorities until a new secretary is sworn in," even if that does not happen until Congress returns from its recess, a senior agency official told CNN.
Panetta will travel to the NATO summit in Brussels next week, but the extent of his duties at his Pentagon office beyond that is not clear.
Panetta on Thursday paid a final visit to the CIA, which he led before moving to the Defense Department. He also privately visited Section 60 at Arlington National Cemetery where many of those killed in Iraq and Afghanistan are buried.
Republicans signaled that they are willing to let Hagel's nomination proceed following the recess.
But if the Hagel fight ultimately becomes protracted, an administration official said it might then have to decide between asking Panetta to stay on longer than planned or having his top deputy, Ashton Carter, run the agency on an acting basis.
Separately, Tom Hagel said his brother has no intention of backing out of the the nomination.
"Knowing him, not only will he not withdraw, but he will be motivated to fight harder," Tom Hagel told CNN.
A source very close to Chuck Hagel who is not part of his confirmation team told CNN that he would never withdraw on his own.
"The only way he will withdraw is if President Obama asks him to," the source said.
Hagel, a former Republican senator from Nebraska, now feels the nomination fight "has become so personal, that it makes him stand even firmer," the source said.
CNN's Dana Bash contributed to this report