young detective

Credit: woolzian/iStockphoto

Studying the science of science

In theory, the scientific method works like this: Researchers ask a question, construct a hypothesis, collect data, evaluate their results, and—ta da!—the world gains valuable scientific insights. In practice, of course, it doesn’t always work that way, and some scientists are taking it upon themselves to go beyond their core research areas to study where the system can go wrong.

Ferric Fang, a professor of laboratory medicine and microbiology at the University of Washington, Seattle, added retractions and scientific misconduct to his research agenda because of his experience as editor-in-chief of the journal Infection and Immunity. In 2010, 3 years after he had taken the helm, the journal retracted five papers by researcher Naoki Mori after they were found to contain manipulated images of various types of protein and DNA gels. In that process, Fang says, “it became apparent to me that the current system was not really well-equipped to recognize skillful misconduct.” In the following months, other journals that published Mori’s papers found similar issues. In the end, more than 30 of Mori’s papers were retracted. “That was the end of my period of naiveté,” Fang says.

Of the several dozen papers Fang has published in the last 5 years, about a third have focused on metascience issues, including investigations into the causes and consequences of retracted research. Now that he’s doing it, Fang thinks of his metascience research as a necessary component of being an active scientist. “As the editor-in-chief of Infection and Immunity, I felt that it was an opportunity—maybe even a responsibility—to weigh in on important issues of the day,” he says.

Fang is among a small yet tenacious group of scientists who study the culture of academic science alongside their other lines of research. These projects offer few quantifiable career benefits, such as grants, awards, or promotions. Nonetheless, scientists who take on this work say that these projects can lead to acquiring new skills and knowledge that help them in their roles as researchers. The real motivation, though, as well as the potential reward, lies in a desire to improve the culture of the scientific community.

Getting started

Like Fang, anthropologist Kathryn Clancy’s foray into metascience arose from personal experience. In 2011, a close friend told Clancy, an assistant professor of anthropology at the University of Illinois, Urbana-Champaign, that she had been sexually assaulted while doing fieldwork as a graduate student. Clancy discussed the issue on her personal blog, sharing two women's stories, which touched off a conversation that reverberated across the science blogosphere and social media. Two graduate students who saw Clancy’s posts invited her to submit a proposal for a symposium on the ethics of field site management to a conference, but reviewers rejected her abstract—something that rarely happens to invited symposia—because her data came “only from blogs.” Clancy thought that quantitative data would give her work in this area more clout, so she and several other researchers collaborated to conduct a more formal survey and interview study, which they called the Survey of Academic Field Studies (SAFE). They found that 64% of the women who participated reported being harassed or assaulted in the field. Clancy is currently working on several follow-ups to the SAFE study and has also published a paper about the importance of discussing race and identity in the social sciences.

Kathryn Clancy

Kathryn Clancy

Credit: L. Brian Stauffer/University of Illinois at Urbana-Champaign

Clancy was also predisposed to think beyond her core research. Her experience as a labor and women’s rights activist during her graduate school years convinced her that, if she were to continue along the academic track, she needed to do it “the right way,” she says. That way, she believes, requires being reflective. “I’m not just going to do research as a scientist; I’m also going to do critical research on science,” she says.

For Jeremy Yoder, a postdoctoral fellow in the Department of Forest and Conservation Sciences at the University of British Columbia, Vancouver, his metascience research about LGBTQA (lesbian, gay, bisexual, trans*, queer, or asexual) scientists’ experiences was “the intersection of personal interest and serendipity,” he says. Yoder, a gay man, found a lack of quantitative data about the representation and experiences of LGBTQA individuals in science. Understanding these scientists’ perspectives could help identify solutions to improve working conditions for underrepresented or marginalized groups, and Yoder realized he could step up and fill that gap. “It occurred to me that it would be a big step forward to put up an online survey,” he says. So, in 2013, he teamed up with Allison Mattheis, an education researcher whom he met when both were at the University of Minnesota, to set it into motion.

The results from these types of metascience research projects have the power to improve the culture and practice of science. Recognition of the barriers faced by scientists from underrepresented groups can inform programs designed to recruit and retain those scientists; understanding the prevalence and impact of deceitful research can lead to more effective policies to prevent them. So, if weighing in on issues of the day is an important part of being a scientist, then why aren’t more scientists doing it?

Opportunities and risks

There are no prestigious awards or grants for researchers conducting this metascience work, but for some practitioners, the experience has created career opportunities. Clancy says that she has been invited to give talks about her metascience research, and Yoder’s work earned him a diversity award last year from the University of Minnesota, where he began his metascience research. In addition, taking on these projects demonstrates strong work ethic, excellent time management, a passion for advancing science, and extended analysis skills—all highly desirable skills in an academic candidate.

Ferric Fang

Ferric Fang

Credit: UW Creative

Plus, metascience work can sharpen researchers’ soft skills, such as running a lab, Fang points out. “I don’t think [my retraction work] makes anybody more likely to award me a grant, or to accept my paper,” he says, “but it’s definitely helped my career by making me a more critical mentor and scientist. I’m much more conscious than I used to be about the cultural environment I establish in my lab and how supportive I am.” He makes it clear to his students that he values clean work above all else, even if it does not yield the sexy results so many labs are looking for. Fang believes that creating an environment where students are encouraged to maintain a reasonable work-life balance and high ethical standards is good for trainees and sets the stage for generations of quality science.

Fang also says that his metascience research has made him a more rigorous scientist. After encountering fabricated images, he’s especially careful in evaluating the presentation of data and analyses in papers he reads. “I’m more critical in looking at figures in papers, and not being quite so trusting that whatever I see must be true.”

But there are just as many negative career effects that can arise from pursuing metascience work. One is the risk that colleagues may perceive metascientists as less dedicated to their other lines of research. Traditional scientific norms prescribe that scientists, especially those in the dawn their careers, should focus on a specific, narrow area of research. Yoder says that although he didn’t hesitate in taking on his LGBTQA project, which he began as a postdoc, he has definitely considered potential negative consequences for his image as a scientist. “In the back of my head, I don’t want [this project] to look like this is distracting from my mainline research focus,” he says.

Jeremy Yoder

Jeremy Yoder

Courtesy of Jeremy Yoder

Now that he is applying to tenure-track faculty positions, Yoder has thought carefully about how his metascience work fits in with the other aspects of his career. His LGBTQA work informs his teaching and mentoring, he says, and shows his dedication to improving diversity in science. But he recognizes that everyone may not see it that way, and he is prepared to encounter institutions that don’t appreciate the full range of his research. But, he says, “if a search committee looks at my CV and thinks that I shouldn’t be involved in [LGBTQA] work, I probably don’t want to be on faculty there anyway.”

Metascientists also flout another common norm: that “good” scientists shy away from the limelight. For better or worse, metascience research often receives widespread attention not only among scientists, but also among media and the general public. “The brighter you shine in the public eye, the more disdainfully your colleagues may look upon you,” Clancy explains, citing Carl Sagan as an example of a brilliant scientist who was adored by the public but shunned by his peers. When the “Sagan Effect” strikes scientists popular to the general public, they are perceived by their colleagues as less serious, and their work is seen as less rigorous.

Clancy says that although some scientists have thanked her for her work exposing the prevalence of sexual harassment and assault in fieldwork, not all have been supportive. “Unfortunately, many of my colleagues don’t see this research as research,” she says. “People think of me as an activist instead of a scholar, instead of realizing those are intricately entwined.”

“Recently, a colleague told me that I’m really ‘buzzy’ right now,” she continues. “But that same colleague never invites me to collaborate on research, even when I’m an expert and would be perfect for [that project].” She has been approached to collaborate on further sexual harassment and assault research, but not her other lines of research. “I thought gaining some level of notoriety for [the SAFE paper] would make it so that people would reach out to me or want to collaborate with me on my other research as well, but that hasn’t happened.”

Is it right for you?

Rather than tallying up the potential career pros and cons, metascientists are motivated by a desire to improve the culture of the science community, as well as their own research. Because this research often stirs criticism, practitioners must have a thick skin and clear sense of purpose. Fang says that he has encountered pushback from colleagues who believe that work that is critical of the scientific process gives ammunition to science deniers. Fang understands that point of view, but doesn’t let it get in the way of his work. “I see myself quoted by creationists, climate change deniers,” he says. “Honestly, I just ignore it.” Hiding criticisms is not productive, he says, and won’t improve science. “The best way to deal with the situation as we have it today is to confront it in a forthright and transparent manner, and to do everything we can to make it more robust, more reliable, and to identify problems that lead to less reliable or less valid research.” He sees his metascience work as an important piece of that effort.

Still, aspiring metascientists should be aware of the potential consequences and prepare accordingly. This is especially true for students. Clancy points out that graduate school can be a particularly “exploitative and vulnerable time,” during which doing controversial metascience research could be difficult, especially if advisers or other colleagues in positions of power are not supportive of the work. That’s not to say that it’s impossible for graduate students to participate. When a student approached Clancy to collaborate on a follow-up project, Clancy was initially nervous that such a project could interfere with that student’s career, but, after seeing the student’s passion, she ultimately agreed to it. “It was so important to [the student] that she do it that I was like, ‘Okay, then, I’m in,’” Clancy says. Clancy has also brought one of her own students into a related project and hopes that she can help shield her mentees from any backlash to their work.

Researchers at all career stages should also consider whether they have the time and effort to dedicate to a new project. Scientists must find a way to slot in metascience between myriad other responsibilities: existing research projects, teaching, mentoring, grant writing, sleep, to name a few. Plus, they generally receive no additional funding to investigate these questions. Rather, these projects are labors of love that some scientists choose to make time for. “I guess that’s what nights and weekends are for,” Fang says of his metascience research. Yoder echoes this sentiment, noting that he has managed to pull pieces of his LGBTQA project together on nights and weekends. For the moment, the follow-ups he would like to do have taken a backseat to other projects and job applications. He thinks it’s highly likely that he will continue with the project eventually, but it will depend on his other responsibilities, he says.

Clancy suggests that scientists could make a list of pros and cons to evaluate how taking on such work could affect their careers. Some aspects to consider include the researcher’s seniority, gender, and race, all of which can complicate the politics of speaking out. Fang, Yoder, and Clancy all acknowledge that their positions of relative privilege helped pave the way for them to boldly pursue metascience research. Fang points out that he had tenure when he began his retraction research, so unlike early-career academics, his work was less likely to affect future positions or job security. For Clancy—a white, tenure-track professor—“privilege allowed me to be radical and loud and angry in a way that I think, looking back, I’ve learned is not as possible for people with different identities.” And Yoder, too, feels he is in a “fairly safe place” to discuss issues in science. “I’m a reasonably productive white man, even if I’m a gay white man. I don’t know this in any certain way, but I suspect that there’s a different inflection for me having a side project like this than for someone in a group where things like how much time are you spending on your research is policed a lot more.” He has also been lucky to work with advisers who provide an atmosphere of intellectual freedom and fully support all the facets of his research program, he notes.

The tools to succeed

Conducting metascience research requires reaching outside academic comfort zones. Although scientists are well trained in the methodologies and analyses popular in their subfield, new projects often lead them into uncharted territory. “There are definitely new skills I’ve had to learn along the way, and I’ve had many people who have been generous with allowing me [to] consult them,” Fang says. Yoder says that he, too, has learned a lot from collaborators, and that Mattheis’s expertise vastly improved their project. “I could’ve done a lot of this work without talking to Alison first, but I could not necessarily have produced work that was publishable in a peer-reviewed journal,” he says. “If you are, like me, someone who studies things that are not humans and you’re interested in studying humans, you absolutely need to talk to people who do that for a living.”  

If you are, like me, someone who studies things that are not humans and you’re interested in studying humans, you absolutely need to talk to people who do that for a living.

–Jeremy Yoder

Researchers have the training and tools to learn about a new field and the associated methodologies, but it’s much harder to find colleagues willing to dedicate precious time and resources to new, exploratory projects. The process of finding metascience collaborators, like the projects themselves, is often a mix of serendipity and shared passion. Yoder and Mattheis were friends who happened to have a shared interest in surveying scientists who identified as LGBTQA. Fang and frequent collaborator Arturo Casadevall, a microbiologist and immunologist at Johns Hopkins University in Baltimore, Maryland, met on the editorial board of Infection and Immunity and found they were both interested in metascience issues. “The way it works is one of us has an idea about something and fires off an email; then we start putting things to paper, we read together, share ideas,” Fang says. Clancy’s collaborations, too, have come from like-minded scientists in other fields who have approached her with new ideas.

In the end, passion is what pushes these projects forward. Researchers press on despite potential career hurdles, making extra time to pursue work they believe can make a difference. Clancy, who is up for tenure next year, says she has considered the possibility that her metascience research could affect her tenure review, but that it never made her afraid to jump into the work she feels is so important. “If you don’t act as the person you want to be your whole life from the beginning, waiting until you’ve jumped through this particular hoop isn’t suddenly going to turn you into a different person,” she says. “What was more scary to me was, ‘I’ll just wait until I have tenure to do XYZ.’ As a labor organizer, I met a lot of faculty who said that, and when they got tenure, they didn’t change.” She also feels strongly that this work was the natural product of her years of activism and labor organizing and her academic interests. “For me, starting this research project was a way of coming full circle. This was me being the researcher I wanted to be—who I was supposed to be,” she says.

Ultimately, metascientists all share the conviction that their work is important, despite what others may think. Whether or not other scientists are interested in pursuing their own metascience research, they will inevitably be affected by metascience research findings, Fang says. Clancy points out that this type of work has staying power and can make contributions to the field for years to come. “Even if I leave academia, the SAFE paper will be out there and racking up citations, and will still be read by many grad students,” she says.

At its root, metascience represents the purest form of scientific inquiry: digging deeply into a problem out of pure curiosity. “Being a scientist is, at the most fundamental level, about being able to study what’s exciting to you,” says Yoder. “If you see something like this and you know how to follow up on it, you should do that.”

Search Jobs

Enter keywords, locations or job types to start searching for your new science career.

Top articles in Careers

Follow Science Careers

dancing shoes