WASHINGTON (CNN) -- The Democratic leader in the Senate promised Sen. Arlen Specter he would retain his seniority when he jumped from the Republican to the Democratic Party, Specter told CNN on Wednesday, but faced "pushback" from other Democratic senators. "Sen. [Harry] Reid said that I would maintain my committee assignments and that my seniority would be established as if I'd been elected in 1980 as a Democrat," the long-serving Pennsylvania senator told CNN's Dana Bash. The full Senate voted Tuesday to strip Specter of his seniority, dropping him to the bottom of the pile on every committee he sits on. The action came on a resolution -- passed on a unanimous voice vote -- that set out committee assignments for the entire Senate. Specter suggested other Democratic senators had objected to his moving ahead of them in the all-important seniority ranks. "The caucus has some concerns, some people who would be passed over, and we're going to work it out," he said. "... I'm confident that Sen. Reid's assurances on my seniority will be fulfilled." Sen. Debbie Stabenow, a Michigan Democrat, suggested Specter had irritated his new colleagues by telling The New York Times the Minnesota courts should "do justice" by declaring Republican Norm Coleman the winner of a bitterly disputed Senate race against Democrat Al Franken. "I think it made many members very upset," Stabenow said in answer to a CNN question about whether Specter would have maintained his seniority if he had not made the comment. "... It was definitely something that concerned everybody. Yeah." Specter's comments to the newspaper appeared about the same time the Democrats were making their decision. Sen. Barbara Mikulski, a Maryland Democrat, phrased the concerns of the caucus more generally. "We have people here who have worked very hard to achieve their seniority on their committees and subcommittees, and we should keep the same leadership we've had," she told CNN. "It's not personal to Sen. Specter." Reid tried to defuse the brewing dispute in comments to CNN's Wolf Blitzer on Wednesday. "Arlen is a senior member of the Senate and that is significant. We can try to work something out with the individual chairmen and I'm certainly doing that," he said. He urged Democrats to focus on the big picture. "I think that everyone should just try to relax and remember that he is a Democrat. We're doing our best to make him happy, and as a Democrat, I think he is," Reid said. Reid's spokesman, Jim Manley, separately said, "There was no miscommunication." Specter jumped from the Republican Party to the Democratic Party last week, putting the Democrats within reach of a 60-seat "supermajority" that could make it all but impossible for Republicans to block Democratic legislation. There is more at stake in the seniority debate than Specter's ego or bragging rights. The old saw "rank hath its privileges" holds true in the U.S. Senate as in few other places. With seniority comes the ability to influence legislation. Specter has been citing his seniority on the powerful Appropriations Committee -- the one that doles out money -- as he hits the campaign trail as a Democrat. "My senior position on appropriations has enabled me to bring a lot of jobs and a lot of federal funding to this state," Specter said at a town hall meeting on Monday. Over and over, he made a point of telling an auditorium filled with medical faculty and staff about the hundreds of millions of dollars he delivered to the Keystone State, thanks to the power he's accumulated in his 29 years in the Senate. "Pennsylvania has a big interest in my seniority, a big interest," he said. As it now stands, Specter is junior to Montana's Sen. Jon Tester, who has been in the Senate since 2007. And he lost his top position on the Senate Judiciary Committee, just as it gears up to question President Barack Obama's yet-to-be-named nominee to replace Justice David Souter. Specter was in the driver's seat when the Senate Judiciary Committee grilled the two newest Supreme Court justices back in 2005. But he will be the very last to speak this time -- after even Sen. Ted Kaufman of Delaware, who has been a senator for all of four months. In fact, only two of the 18 other senators on the committee have been in the upper house longer than Specter -- and he has been in the Senate longer than seven other committee members put together. Back in 2005, the Pennsylvania senator was in the driver's seat when the Senate Judiciary Committee grilled the two newest Supreme Court justices. But now, when the Judiciary Committee questions President Obama's yet-to-be-named nominee to replace Justice David Souter, Specter will be the very last to speak -- after even Sen. Ted Kaufman of Delaware, who has been a senator for all of four months.