Read our COVID-19 research and news.

A lot is riding on this question from the American Community Survey.

A lot is riding on this question from the American Community Survey.

U.S. Census Bureau

Can Question No. 12 survive? Researchers fight to retain a question about college degrees on American Community Survey

A proposal to drop a question about college education from a large annual government survey would make it a lot harder for the National Science Foundation (NSF) to track trends in the U.S. scientific workforce. Social scientists hope to persuade the Obama administration to reject the idea, which stems in part from criticism of the survey by some members of Congress.

Each year, the Census Bureau uses the American Community Survey (ACS) to collect housing and demographic information from some 3.5 million people. The ACS debuted in 2005 as a way to keep track of national trends that occur between the decennial census, a tally of every household in the country. Many federal agencies and private firms use the ACS data to track trends and plan programs. NSF, for example, uses the question that has been proposed for elimination—which asks respondents to identify their college major—to create a sampling pool for a more detailed survey that illuminates trends in the science and engineering workforce.

But some members of Congress dislike the ACS. They believe the government has no business asking Americans about the number of flush toilets in their homes or when they leave for work in the morning, or even how much they earn. Some legislators have gone even further, arguing that the 72-question ACS should be drastically shortened or even eliminated because it’s an unnecessary burden on the public.

In response to that criticism, the Census Bureau this year carried out the first stem-to-stern review of the survey. A review panel graded each question on two key criteria: Did a law or regulation require the government to collect that information? And how much time and effort did it take to answer? The bureau also tried to measure the value of the information to the federal government and the broader audience of users.

Seven questions scored low enough to be axed from the survey beginning in 2016, the bureau announced on 31 October in the Federal Register. Six of the choices appear to be uncontroversial—five questions pertain to a person’s marital status, and a sixth asks residents whether there is a business or medical office on their property. (Update:  Some social science groups are unhappy that these questions have been targeted.)

And then there’s the bureau’s decision to flag Question No. 12, which asks respondents who have completed college to list their undergraduate major. (Question No. 11 asks people about their level of education, from no schooling through a Ph.D.) The proposed elimination of that question has struck a nerve with the U.S. statistical and social science research community, and with the thousands of organizations that cite the data for their own purposes. There is a bipartisan political consensus that the country needs more tech-savvy workers, they say, and Question No. 12 is the most comprehensive, timely, and cost-efficient way to collect the data that policymakers need to achieve that goal.

The bureau’s decision is actually a proposal to an interagency panel that will make a final recommendation next spring to the White House Office of Management and Budget (OMB). The Federal Register notice started the clock on a 60-day comment period. The U.S. statistical community hopes its responses will persuade the Obama administration to reverse what it sees as an unintended yet potentially damaging consequence of a well-meaning exercise in good government.

Keeping the question is “a no-brainer,” says Norman Bradburn, a senior fellow at NORC, an independent research organization at the University of Chicago in Illinois. “There is at present no other source for this information,” says Bradburn, a former head of NSF’s social and behavioral sciences directorate, which includes the National Center for Science and Engineering Statistics (NCSES) in Washington, D.C. “To produce it by a special survey would be extremely costly and provide data of much less quality and utility.”

Degrees of difficulty

Another opponent of dropping the question is Patrick Jankowski, vice president of research at the Greater Houston Partnership. When Jankowski tries to woo high-tech companies to the Texas city, data from Question No. 12 allow him to brag that the metropolitan area has a higher concentration of graduates with degrees in science and engineering than the country as a whole. The question also generates a demographic breakdown of the region’s population, for example, how many women aged 25 to 39 hold degrees in science and engineering. He says such data can be very useful in tailoring the city’s pitch to prospective employers, to other scientists and engineers thinking of relocating to Houston, and to businesses who wish to reach a particular population.

“It’s b---s--- to say that it’s a burden on someone to answer that question,” Jankowski says. “Anybody who has gone to college is proud of the degree they received.”

In fact, Question No. 12 scored very low on the bureau’s scale of the “burden” it imposes on the public: It takes the average respondent 9 seconds to answer, officials estimate. “It’s easy for people to provide the answer, the response rate is high, and it’s not a sensitive question,” explains James Treat, chief of the ACS office in Washington, D.C.

The problem is that Question No. 12 also scored low on the “benefits” side of the bureau’s ledger. In large part, that’s because Congress hasn’t explicitly mandated collection of the information. But officials at NSF, which waged a successful campaign to have the question added to the ACS, say that Congress has repeatedly made it clear it wants the agency to collect such information in a cost-effective manner.

The charge goes back to NSF’s founding in 1950 as a research agency that would also be a “central clearinghouse for the collection, interpretation, and analysis of data on scientific and engineering resources.” Since then, Congress has reinforced that mission several times in legislation.

One of NSF’s key information products, the biennial Science and Engineering Indicators, draws upon Question No. 12 to help paint an exhaustive picture of global trends in science and engineering. The data are also essential for a congressionally mandated report on women, minorities, and persons with disabilities in science. NCSES uses the answers to create its own survey of college graduates. Its definition of the scientific workforce includes both persons with so-called STEM (science, technology, engineering, and mathematics) degrees and those holding jobs requiring STEM-related skills but who lack a STEM degree.

NSF once obtained data on college majors from the decennial census. But the long form disappeared after the 2000 census. NSF fielded its own survey in 2003 to identify the subset of scientists and engineers in the overall U.S. workforce, at a cost of $17 million. So in 2005 Lynda Carlson, then head of NSF’s statistical shop, began a campaign to place an equivalent question on the ACS and eliminate the need for a stand-alone NSF survey.

Carlson initially was told that legislation would be needed ordering the Census Bureau to add the question to the ACS. But after the relevant committees balked at her request, she got OMB to agree that asking about undergraduate fields of degree was in the national interest. After extensive field tests, the question first appeared in 2009. Since then, the arrangement has worked out so well that NSF was able to save another $4 million by canceling another survey on recent college graduates that it had used to fill in the blanks between each decennial census.

Question No. 12 provides “the only way we have to take a broad look at the entire U.S. population,” says John Gawalt, the current NCSES director. Gawalt reacted “with disbelief” when the question appeared on the Census Bureau’s hit list, he says, adding that he finds it “ironic” that the bureau actually carries out the survey of college graduates on behalf of NSF.

“The Census Bureau is fully aware of the importance of the question,” Gawalt says. “And we’ve been encouraged to comment on the benefits that weren’t captured in its initial analysis.”

Political considerations

Social scientists aren’t blaming the Census Bureau for its decision to put Question No. 12 on a hit list. Instead, they believe that census officials agreed with them on the question’s value but that political realities have forced their hand.

“The Census Bureau does not want to do this,” asserts Katherine Smith, executive director of the Council of Professional Associations on Federal Statistics in Alexandria, Virginia, which represents those who rely on the quality and availability of federal statistics and which strongly opposes dropping Question No. 12. “They were forced to go through a process, and this is how it turned out. But I don’t think they are any happier with the result than we are.”

The Census Bureau “has to show Congress that it is trying to save money,” Bradburn explains. Given that many legislators favor reducing the size and scope of bread-and-butter surveys like the ACS, the obvious solution was to propose dropping some questions.

The Census Bureau’s Treat says he has no dog in the fight except to make sure that the ACS content review is carried out with integrity. “There are no quotas” for the number of questions that must be dropped, he says, “and in the end I don’t really care about the fate of any particular question.” At the same time, he says the Federal Register notice gives the community a golden opportunity to make its case for retaining Question No. 12 and, by extension, the ACS itself.

“I know there’s a lot of angst in the community right now,” Treat says. “But I think there’s a lack of understanding that the survey is under attack. So I encourage everybody to respond to the notice. The more responses we get, the better understanding there will be about the value of collecting this information.”

Update, 1:20pm 11/14/2014: This story has been updated to reflect concerns about the proposal to remove questions related to marital status.