Read our COVID-19 research and news.

Census planners are using digital geographic information systems to help cut the costs of the 2020 count.

© Lee Roth/Roth Stock/PR Newswire, Courtesy of U.S. Census Bureau

Money, politics, and abandoned homes: Why the 2020 Census might be in jeopardy

John Thompson took the reins of the U.S. Census Bureau in August 2013 with orders from Congress to dramatically lower the projected cost of the 2020 decennial census. His response was an ambitious plan to modernize procedures that would also shave $5.2 billion off a projected $17.8 billion price tag. If successful, Thompson’s plan meant that the 2020 count would come close to matching the $12.3 billion cost of the 2010 census.

Four years later, those projected cost savings appear to be in serious jeopardy. Annual budgets have fallen far short of what’s needed to fully implement Thompson’s plan, and internal costs have risen.

Hanging in the balance is the fate of the census itself, the nation’s largest civilian exercise, which is required by the U.S. Constitution. Thompson is also gone, and his 30 June departure has only added to the uncertainty at the government’s oldest and largest statistical agency.

This story, as well as companion stories to be published this week on ScienceInsider, takes an in-depth look at the myriad political, technical, and budgetary issues that are confronting the Census Bureau, which generates some of the world’s most widely used demographic data.

A re-engineered census

Why does it cost upward of $12 billion to get simple demographic information—age, gender, race/ethnicity, and household relationships—about every U.S. resident? The answer lies in how the Census Bureau carries out the mandate it was given in 1789 to conduct an “enumeration” of every person in each state for the purpose of apportioning the number of seats in the U.S. House of Representatives. Over time, the government has come to rely on data from the census and its longer annual offshoot, the American Community Survey (ACS), to allocate half-a-trillion dollars through health, welfare, housing, infrastructure, education, and other federal programs.

John Thompson retired as head of the Census Bureau in June.

U.S. Census Bureau

Close to two-thirds of the country’s population is fairly easy to count, it turns out. That large cohort—which probably includes anyone reading this story—respond promptly to the agency’s decennial query. But connecting with that final third of the population is what really drives up the cost. (The decennial census is actually a survey of the U.S. population and housing units, although that dual focus derives from tradition rather than from the original intent of Congress.)

Thompson confronted the challenge head-on by proposing to re-engineer the three major components of the census:

  • the master address file that underpins everything else;
  • self-response rates; and
  • the massive workforce deployed to count those who don’t respond to the agency’s initial solicitation.

To pull off a complete and accurate 2020 nose count, that agency has to perfect what it calls “operations and systems.” There are some 35 of these, including internet and telephone systems for answering census questions, the traditional mail-in paper form, hand-held devices to help field workers track down those who haven’t already responded, and the use of information obtained from other government agencies or third parties—including geospatial data from satellites—to update the master file of 143 million addresses. Some systems and operations have changed little since the 1970 census, Thompson notes.

Upgrading those operations and systems for the 2020 census could save $5.2 billion, Thompson has estimated. It’s a coincidence, he adds, that the figure conforms to a congressional directive to do the 2020 count at the price of 2010’s. “I never actually promised that,” he says. “I just knew we could do things more efficiently. And I wanted a number I could stand behind, not one that we just pulled out of the air.”

False savings

His strategy assumed that Congress would provide the funding in the years leading up to Census Day on 1 April 2020 to ensure that all of the moving parts meshed. But that hasn’t happened. Since 2015, Congress has appropriated about $170 million less overall than the bureau has estimated it needs to stay on track for a modernized count. And the trend is going in the wrong direction.

Will the big bucks be there?

Tight budgets to date contrast sharply with what the 2020 Census will need in the next few years, according to a 2016 estimate from then-Census Bureau Director John Thompson.

(Data) U.S. Census Bureau

(Graph) D. Malakoff/Science

For instance, the $800 million request from President Donald Trump for 2018 is $112 million below what Thompson had estimated 2 years ago would be needed for that year. And it appears likely that Congress will go along with that request, although it may not happen until well after the 1 October start of the next fiscal year.

With Census Day less than 1000 days away, Thompson says the agency is running out of time to recover from those skimpy budgets. The bureau “is moving ahead to do a complete and accurate census in 2020,” he asserts. “But Congress is, at some point, going to have to pony up the money that’s needed to do a good census.”

What that means is spelled out in the agency’s budget request to Congress for fiscal year 2017, submitted by the former President Barack Obama's administration in February 2016. The numbers may be scary to fiscal conservatives: $912 million needed in 2018, $2.05 billion in 2019, and $6.15 billion in 2020.

The good news, Thompson notes, is that those numbers are, respectively, $877 million, $1.6 billion, and $2.5 billion less than they would have been for those years absent a re-engineered census. But Congress shouldn’t be looking for additional savings, he warns: “Those numbers are not going down.”

In the meantime, its recent tight budgets have forced the Census Bureau to curtail, delay, or cancel several activities designed to test or implement improvements. In some cases, it has also meant ditching planned improvements and falling back on tried-and-true methods, including a method of first contacting millions of residents in rural and remote areas that Census officials had hoped to replace with a more efficient approach.

“As with everything else, we have to balance the risk versus our mitigation approaches,” Lisa Blumerman, associate census director and head of the 2020 Census, explained after a 11 July project management review, a quarterly update of the state of the 2020 census held at the agency’s headquarters in Suitland, Maryland. “Our focus is on procedures designed to ensure that we can conduct a complete and accurate census.

The Census Bureau headquarters in Suitland, Maryland.

U.S. Census Bureau

Census advocates don’t blame the bureau for taking that cautious strategy. But they say the approach is penny-wise and pound-foolish. “The Census Bureau is inadequately funded,” argued Representative José Serrano (D–NY), the top Democrat on the House spending panel that oversees the agency, at a markup last month of a 2018 spending bill. “The Trump administration has shortchanged the important planning and testing tools that are needed now to prevent higher costs in the future. Without this crucial testing now, initial response rates will drop and costs will rise.”

But his words, and those of other Democrats, fell on deaf ears. The panel’s Republican majority rejected Serrano’s amendment to add $384 million to the agency’s overall $1.5 billion request. On 13 July the full appropriations committee approved a bill that would provide only a $51 million boost in 2018 for the account that includes the 2020 census and ACS.

“I can’t do any more”

This fall the Census Bureau will issue a new estimate for overall cost of the 2020 census that will almost certainly be much higher than the earlier $12.5 billion price tag. But how much higher is anybody’s guess. And some of the increase can’t be blamed on Congress and the new administration: This spring Census officials disclosed that a new information technology system being used to manage the rest of the census’s interlocking systems will cost at least $309 million more than initially budgeted.

But Thompson won’t be around to find out the impact of those and other changes to his master plan. On 15 May he announced that he was stepping down as director and retiring from government on 30 June. (Thompson spent 27 years at the Census Bureau before joining the private sector in 2002, returning in August 2013 to lead the agency.)

He announced his departure less than a week after disclosing the IT cost overrun to congressional overseers. His boss, Secretary of Commerce Wilbur Ross, had already launched a review of all the cost estimates for the re-engineered census, saying he didn’t have confidence in numbers generated by an appointee of the Obama administration.

The implied partisanship at the government’s leading statistical agency may have been the last straw for Thompson. As director he made little headway in winning over conservative Republican legislators who have long complained that the 70-some question ACS, which in 2005 replaced the long form of the decennial census, is unduly burdensome and an invasion of privacy (see stories here, here, and here). And the agency’s recent budgets have certainly complicated his plans for 2020.

“You work and you work,” Thompson told ScienceInsider last week, “and at some point you take stock and you say, ‘I’m at the point where I’ve done all I can do and I can’t do any more.’ It’s time to go.”

Thompson says Ross never raised the option of his staying on, although a 2012 law aimed at minimizing disruptions during the 10-year census cycle would have allowed Trump to appoint Thompson to a second 5-year term. (Thompson was 4 months into a 1-year extension of his first term when he announced his departure.)

“The new person will be nominated by the current president and confirmed by the current [Republican] Senate,” Thompson says. “So he or she will have a real mandate. I’m not going to get into any finger pointing with Wilber Ross,” adds Thompson, who this week started a new job as head of the Commission of Professional Associations on Federal Statistics in Washington, D.C. “I respect him, and I know he wants to have a good census. And I’ll leave it at that.”

Cutting back

Ross has appointed two highly respected career Census Bureau employees—Ron Jarmin and Enrique Lamas—to lead the agency until a permanent director is in place. But they will have limited clout in dealing with Congress and the White House at a time when the agency needs a strong advocate.

One of the biggest challenges facing any director is riding herd on the wildly fluctuating budget for the decennial census. Over the current 10-year cycle, for example, it will rise from a low of $66 million in 2012 to a projected peak of $6.2 billion in 2020. At the same time, the long ramp-up gives Census officials some leeway in scheduling the iterative tests needed to ensure the seamless integration of individual systems and operations.

Census officials have been able to use that flexibility to absorb initial budget shortfalls for 2020. But they hit the wall in September 2016, when Congress failed to pass a 2017 budget on time and instead froze spending indefinitely at 2016 levels. On 18 October, Blumerman announced that she was canceling two field tests scheduled for April 2017, one in Puerto Rico and one on two Indian reservations in Washington state.

“Given the current uncertainty about FY [fiscal year] 2017 funding, the Census Bureau will not continue expending resources to prepare for the FY 2017 field tests, only to receive insufficient resources to conduct them,” her memo reads. “It is a risk we are unwilling to take.” Continued planning for the tests, she added, “would all but guarantee wasted efforts and resources.”

Her memo said that the agency still needs the information the canceled tests would have provided, and it raised the prospect of incorporating the tests into the final dress rehearsal for the 2020 census. Census officials had chosen three locations, with a combined 700,000 households, for the so-called End-to-End Census Test in April 2018. Each location offered a different set of characteristics that can be important in conducting the census. For example, Providence county in Rhode Island offered an East Coast, urban area with a varied population. Three counties in southern West Virginia represented a rural and mountainous setting. And Pierce county in southwestern Washington state featured a large military base.

Six months later, however, Census officials decided that the full dress rehearsal would take place at only one site, in Providence. The news was revealed in the president’s 2018 budget, and the unspoken but clear implication was that there wasn’t enough money in that request to conduct a three-site test. At this month’s project management review, Blumerman praised the demographics of the single remaining site and said, “We feel very comfortable that Providence county will allow us to test what needs to be tested in the End-to-End test.”

An internet option

The 2020 census will be the first one that can be completed over the internet. Previous versions consisted of paper questionnaires—some 400 million were printed in 2010—that needed to be filled out and mailed back. The paper version will still be available, although the agency plans to print only 140 million copies this time around. And paper will still be the only option for some 10% of the U.S. population living in rural and remote areas or without mailable addresses. For the first time, the public will also be able to complete the questionnaire by calling a toll-free number that will be staffed by trained operators offering assistance in several languages.

In 2020, census organizers anticipate that offering internet and phone options will boost self-response rates and greatly reduce the number of paper forms.

U.S. Census Bureau

Census officials hope that most people will choose the internet option. Over the past 3 years it has run small-scale tests of the cloud-based operation in several cities that have gone relatively smoothly. But those tests can’t simulate what will happen after the census “goes live,” when officials say the website could receive as many as 2 million simultaneous hits.

The key to a high self-response rate is publicity. The Census Bureau conducts a massive communications campaign in the run-up to every census with help from thousands of local and regional governments, civic organizations, and corporate partners. But Census officials have canceled the advertising campaign that was to accompany the 2018 dress rehearsal, citing a tight budget. That decision eliminates the only scheduled large-scale test of what public service messages and other outreach efforts are most effective in raising self-response rates, as well as the opportunity to tap some of those partnerships before the buildup to 2020.

Getting good addresses

Another challenge facing the bureau is making sure it has reliable addresses. As part of preparations for the 2010 census, some 150,000 people walked more than 6.5 million kilometers to identify every housing unit in the country. For the 2020 count, Thompson has projected the agency could save almost $1 billion, and maintain its high standards, by tapping into readily available, high-resolution geospatial data. In particular, Census officials set a goal of having in-house employees verify 75% of the nation’s housing units by poring over satellite imagery and other data from commercial mapping services.

Identifying abandoned residences is a key part of an efficient census.

U.S. Census Bureau

But last fall, citing a shortage of funds, Census officials canceled the second phase of that two-step in-house exercise, the part aimed at resolving discrepancies that had been identified by an initial analysis. The Census Bureau estimates it will now need to physically verify 30% of the units, a rise of five percentage points. The Government Accountability Office has estimated that the additional field workers could cost an extra $22 million, although it says the agency hasn’t tracked the current operations well enough to know how much additional canvassing will be needed in 2019.

Puerto Rico has traditionally presented a special challenge because of its nonstandard system of addresses. The agency had scheduled a field test this spring involving 123,000 housing units in the San Juan area in hopes of resolving those differences. The exercise would also have tested local attitudes toward using the internet self-response option, as well as any logistical problems related to offering Spanish versions of all three self-response options.

But the Puerto Rico test was canceled because of the budget shortfall. Also canceled were two scheduled tests this spring on two Native American reservations in Washington state, that were designed to measure the agency’s ability to use tribal enrollment data in enumerating residents as well as how the new electronic system of updating addresses works in areas without unique addresses.

Off the beaten track

Another recent change with potentially significant budgetary implications involves the agency’s interactions with some 12 million housing units in rural and remote areas. They are places without regular mail service and unique, city-style addresses, or those that are simply really hard to reach. These areas are not covered in the address canvass exercises done in advance of the actual census.

In past censuses, the first step was to have field workers update the list of housing units in their assigned blocks. Then they would knock on the door of those residences and hope that someone answers who is willing to be enumerated. If not, they would leave a note saying a census worker had been there and would be returning.

That operation was known as update/enumerate (U/E). One variation, called update/leave (U/L), involved updating the address but not knocking on any doors. Instead, the enumerator left a paper questionnaire, with a cover letter instructing the residents to mail it back.

Field workers will use handheld devices for the first time in 2020.

U.S. Census Bureau

The re-engineered census called for the elimination of U/L and the sole use of U/E in those areas. Only this time the field worker would verify or update the address electronically, presumably on a tablet, before knocking on the door. At that point, the census taker will leave information if no one is home, giving residents an option to respond through the internet or telephone, but also letting residents know that the census taker will return otherwise. The operation envisioned additional door-knocks for U/E households, unless the residents have answered the questions.

However, this month Blumerman announced the agency had decided to scrap that plan and, instead, would restore U/L for all but 500,000 of the 12 million units. The major difference, she explained, was that field workers will not be doing any enumeration after updating the address. “We will simply be leaving the package,” she said.

Blumerman said she could not comment on the cost implications of the new approach, pending release of the new life-cycle estimate, nor whether cost was a factor in the agency’s decision. But outside experts speculate that the agency may be banking on a high rate of self-response through the internet option, thereby reducing the percentage of households that would end up in the more costly nonresponse follow-up (NRFU) pool.

The proposed new U/E system was never tested in the field, notes Terri Ann Lowenthal, a census consultant and former congressional staffer based in Stamford, Connecticut, and that no such test appears in the revised schedule of remaining tests.

Another reason for its demise could be unanticipated logistical challenges. Atri Kalluri, who manages IT systems for the decennial census, told reporters this month that it isn’t possible for field workers to do both update and enumerate operations on the same mobile device because they require different software platforms that can’t run on the same device. He said census officials also don’t want to burden field workers with carrying two different devices, that is, a tablet and a smartphone. So any enumeration would have had to be done the old-fashioned way, with paper rather than electronically.

When they don’t answer

High self-response rates are the Holy Grail for census takers. But the reality is that rate has been dropping steadily, from nearly 80% in past decades to below 63.5% in 2010. And those nonrespondents are harder to find, thanks in part to economic disruptions and a growing population of immigrants.

“There is undoubtedly a lot of distrust of government among the immigrant community,” Thompson acknowledges. “So it’s going to take a real effort, especially through the partnerships, to get the word out and explain to the community why it’s important to be counted and that the census is confidential.”

There are two major components of NRFU. One is being able to weed out bogus addresses, that is, those attached to housing units that are vacant or that don’t exist at all. The second is getting the necessary information from valid addresses.

The Census’s traditional approach to that challenge was through field operations. That requires not just workers—600,000 in 2010—but also the physical infrastructure—12 regional centers and nearly 500 local offices—to manage them. The 2020 census hopes to cut those numbers in half, and save a projected $2.5 billion.

Some of the savings will come through automating procedures like recruitment, training, and payroll. But the real innovation for 2020 is the use of so-called administrative records. That refers to all the demographic information already on file with U.S. government agencies.

By and large, those records won’t obviate the need for people to answer the census because they are not sufficiently detailed to meet the bureau’s data quality standards. “Remember, they are not trying to do a census using administrative records,” Thompson says. “What they are trying to do is use administrative records where they can help the Bureau apply its resources more effectively.”

One such application is in reducing the number of viable addresses. Thompson estimates that Census may be able to eliminate up to half of the country’s vacant units—some 14 million in 2010—through the use of commercial and government records. That pruning, he says, will both reduce the number of units that enumerators must visit and improve the odds that someone will answer their knock on the door. In 2010, for example, enumerators visited a single address as many as six times in pursuit of the information.

“No time to lose”

Although he’s no longer in charge, Thompson says he has full confidence in the abilities of the current 2020 census leadership team. “They don’t need me to do a good job,” he says with a laugh.

But Thompson is deeply worried about forces beyond their control. “The FY 2019 budget will be really critical for the Census Bureau,” he says. “They have been deferring and deferring things. But nothing can be deferred in 2019. There’s simply no more time to lose.”