House Republican leaders said Thursday they're ready to go to court against President Obama if he doesn't scuttle his plan to move the census into the purview of the Oval Office, saying it's an unconstitutional abuse of power. House Republican Conference Chairman Mike Pence, R-Ind., also called on Obama to withdraw his nominee to head the Commerce Department, Republican Sen. Judd Gregg of New Hampshire, if Obama didn't have the confidence in him to lead the Census Bureau. Gregg has been a long time opponent of increased funding to the bureau. Under Obama's plan, the director of the U.S. Census Bureau, who has yet to be named, would report to White House senior management in addition to the Commerce Department, which oversees the bureau. A Senate committee has scheduled a hearing next month on the potential change. Republicans on the House Energy and Commerce Committee are also pushing for an investigation. GOP leaders sent Obama a letter to the White House on Wednesday demanding a reversal of the plan. "If the president doesn't acquiesce to our letter, then we will seek the courts," said Rep. Darrell Issa, R-Calif., a ranking member of the House Oversight and Government Reform Committee, said at a news conference Thursday. "Ultimately I don't think there's any question among federal courts about whether or not this is a personal power of the presidency or whether or not executive privilege would be waived if he started doing functions like this," Issa said. A spokesman for Issa told FOXNews.com that the lawmaker wouldn't initiate a lawsuit but would lend his support to any individual or group that did. At the news conference Thursday, House Republican leaders announced the formation of a census task force to keep an eye on developments. Republicans displayed a large placard with a 2006 quote from White House Chief of Staff Rahm Emanuel that read, "If you think redistricting is always partisan and political which it is...it's going to be on steroids this time." Census numbers determine everything from government pay-outs to how many people represent each state in Congress. Past censuses have sparked fights over issues as varied as how to ensure remote population groups are counted accurately to how such terms as "poverty" are defined. The controversy began when Obama nominated Gregg to head the Commerce Department. Gregg once voted for a broader budget measure that would have abolished the agency, and he opposed increased funding for the 2000 census. Gregg's record raised concerns about his commitment to an accurate census count, a priority for minority groups that have historically been undercounted. Gregg's nomination initially pleased Republicans because he has opposed increased funding to the census and once supported abolishing the agency. But now they have begun to question his silence. "If President Obama doesn't trust Sen. Gregg to oversee a fair and accurate census, he should withdraw the nomination," said GOP conference chairman Mike Pence, R-Ind. The White House sought to soothe those concerns in a statement late last week reassuring that the census director would "work closely with White House senior management." That in turn sparked an uproar from Republicans, who accused the White House of injecting partisan politics into the census and seeking to cut out agency professionals in favor of political operatives. The White House issued a statement Wednesday, emphasizing Obama's commitment to a "complete and accurate count through a process that is free from politicization" even while seeking to explain that no real change was being made to the census director's chain of command. "As they have in the past, White House senior management will work closely with the census director given the number of decisions that will need to reach the president's desk," said the statement from White House spokesman Benjamin LaBolt. Rep. Carolyn Maloney, D-N.Y., who has pushed legislation to create an independent census agency, complained about the move by House Republicans, saying their "answer is to have a press conference and create a tempest over the Census Bureau, even before the president has had a chance to unpack his bags."