The U.S. Census Bureau is proud of the high response rate on its American Community Survey (ACS), an annual sampling of the U.S. population begun in 2005 to replace the long form of the decennial census.
But those results don’t come easily—or cheap. Despite a law requiring people to participate in the survey, only about half of the 295,000 households chosen each month actually complete the 72-question ACS on their own. For the rest, the agency takes a series of steps to badger or cajole them into completing the survey, including home visits. That extra effort eventually bumps up the response rate to an impressive 97%.
Such diligence by a federal agency would normally win praise from Congress. But several Republican legislators regard the survey, which provides policymakers and social scientists with an exquisitely detailed portrait of the country and is used to determine how to distribute half a trillion dollars in federal aid, as a form of harassment and an invasion of privacy.
“My office continues to hear from constituents saying that someone is parked outside their house or they are knocking on the door,” Senator James Lankford (R–OK), one of the survey’s most vocal opponents, told census officials last week at a hearing on the bureau’s plans for the 2020 decennial census that also touched on ACS. “They don’t want to complete this, they think it should be voluntary, and we need to find a way so that it’s not oppressive.”
Next month, census officials will try to address some of those concerns by softening some of the messages about participation on survey materials mailed to households. However, previous research suggests that a kinder, gentler ACS may actually lower the initial self-response rate, driving up follow-up costs and potentially harming the quality of the survey.
A multistep process
The ACS is an annual sampling of 3.5 million households, about 2.5% of the United States total. And the process is carefully orchestrated. Residents are notified by mail that they’ve been chosen to participate. A second letter tells them how to fill out the survey online. On the outside of the envelope, next to the address, is a message that reads “YOUR RESPONSE IS REQUIRED BY LAW.”
Several days later a postcard arrives, thanking those who have submitted the survey and encouraging the procrastinators to do likewise. Then those who still haven’t responded get a package with a paper version of the 28-page questionnaire inside. As with the second mailing, this fourth missive states on the outside that participation is mandatory.
Together, those pleadings generate a response rate of 50% to 55%, says ACS director James Treat. But that’s not nearly good enough for a survey designed to reflect a scientifically valid cross section of the country, he says.
At this point, the Census Bureau gets a bit more aggressive. Operators call residences with a known phone number, and ask if residents would be willing to complete the survey by phone. Up to 10% agree, talking with one of a few hundred people at three call centers around the country.
That leaves some 35% of the households still unaccounted for. To reach them, the bureau sends out 3000 agents to knock on doors. Hoping to catch residents at home, they may arrive early in the morning or in the evening. But deploying that full-court press is expensive. Each visit, Treat says, costs the bureau roughly $170.
Next month, however, the bureau is going to change things up in a bid to reduce concerns about the survey. Officials will remove the stern messages on the envelopes of the second and fourth mailings sent to 24,000 recipients, less than 10% of the May sample. Then they will measure the impact on self-response rates.
Planners are hoping not to repeat history. “Ten years ago we tested a similar mailing that didn’t talk about participation being mandatory [and] the self-response rate dropped by 20 percentage points, from about 50% to 30%,” Treat says. If that happens again, Treat says, “it will mean more follow-up phone calls or visits. And that will make the survey more expensive.”
Lankford already thinks the ACS is too expensive as well as too intrusive. Last week he gave Census Bureau Director John Thompson an earful on both issues while presiding over a hearing on the 2020 census. (The committee’s chair, Senator Ron Johnson [R–WI], was absent for most of the 2-hour hearing.)
“Is there anything on the ACS that’s already available from other sources? What data can we buy from a company?” he asked Thompson. “I don’t know of another survey that costs as much as we’re spending on ACS. We need to find a more efficient way to do it.” Speaking to ScienceInsider after the hearing, Lankford asserted that a company “would be able to find individuals who are willing to answer those questions voluntarily, and probably for half the price.”
But Lankford’s real ire was directed at how the Census Bureau pursues residents who need to be reminded to fill out the ACS. “So it’s not just the length of the questionnaire and the type of questions on the survey,” Lankford scolded Thompson. “It’s also the way the individual is treated, and the repetitive appearances at their door.”
An unhappy respondent
Andrea Stockton, who lives outside Boise and runs a nursery school in a local church, says she had never heard of the ACS when she was notified in December 2013 that she had been chosen. She thought the first letter from the bureau was junk mail, she recalls, and her friends confirmed her suspicions by labeling it a “scam.”
The local Chamber of Commerce told her to call the regional census office, she says, but she remained dubious after talking to someone on the phone. It was only after she faxed a few pages of the survey to a relative who was an attorney, who said it looked legitimate, that she decided to comply with the stern warning that participation was not voluntary.
Even so, she was incensed by questions asking about her income, household bills, and commuting patterns. Although she dutifully answered them, she then crossed out her answers and wrote “not applicable” across every question she considered inappropriate. She says she never received another letter from census officials after mailing in the form.
Stockton says she wouldn’t have felt any kindlier toward the ACS if the mailings had omitted the warning. Asked if she resists other government forms, she replied, “the IRS knows how much I make. But I don’t have a choice. The Census Bureau is a different arm of the government, and I just don’t think they have any business asking me those questions. I don’t even want my neighbors to know when I leave the house for work.”
Looking for improvements
Thompson said he can only go so far to address the concerns of Stockton and other residents that the survey is inappropriate, but he says that the agency is trying to make the experience less onerous. “We understand there are concerns about the length of the ACS and the perception that the language on the envelope is upsetting,” he told Lankford. “We are looking to see if we can make the questionnaire shorter, by asking some questions every other year if the information is not needed on an annual basis. … We are also looking to see if we can use administrative records to answer some of the questions on the form.”
In the long run, Thompson said, the best solution is a better informed public. “We are looking at how we can change our messaging, to do a better job of communicating to people why the ACS is important to their community.”
Lankford agrees that census officials can do better. “You get a postcard and it looks just like a piece of junk mail,” the legislator told ScienceInsider after the hearing. “It says you’ll be getting another piece of mail. And when that comes, it looks like another piece of junk mail, but on the outside it says: You are required by law to open this and go online and fill this out. A lot of people don’t even think this is real.”
Thompson said the results from the field test should be available before the end of the year, and possibly even by this summer. But those results aren’t likely to placate Lankford, who plans to continue to push for changes to ACS.
“I understand that there’s a constitutional mandate to do a census every 10 years. But in between, the ACS is just providing helpful information,” he says. “Yes, some of the questions are required by Congress … but the method of doing so is not required.”