Rather, Yi Rao and colleagues at Peking University in Beijing, China, showed that male mice will cheerfully mount both male and female mice, as long as their brains are deficient in one chemical messenger: serotonin. The paper, published in Nature, received plenty of mediacoverage. Now, two other research groups report seemingly opposite findings: Male mice with no serotonin in their brains still prefer female mice to males. The researchers contend that serotonin is about social communication and impulsive behaviors, not sex.

Mounting behavior aside, sexual preference in mice is not about “turning mice gay.” It never has been. Instead, it’s about the role that a single chemical can play in animal behavior. And it’s about what, exactly, those behaviors really mean.

Serotonin serves as a messenger between cells. It plays important roles in mood. Serotonin-related drugs are used to treat some forms of migraine. And of course, serotonin plays a role in the psychedelic effects of recreational drugs such as hallucinogens. So when the Peking University group set out to show serotonin’s role in sexual preference, they attacked it from several angles. They used mice that had been genetically engineered to lack the brain cells that usually produce serotonin. They used a chemical to deplete serotonin in the brains of normal mice. And they created another strain of mice that lacked the enzyme that makes serotonin in the brain.

In all three sets of animals, the researchers saw the same result. The male mice with no or low serotonin in their brains approached males and females with seemingly equal enthusiasm. Two years later, the Rao group doubled down. Using almost the same methods, they showed that female mice lacking serotonin in their brains will mount other females and will also accept male mounting, they reported in the Proceedings of the National Academy of Sciences.

This was not the creation of gay mice. Rather, for male mice deficient in serotonin, the sex of the other mouse simply no longer mattered. But is it really that simple? In a very general sense, increasing serotonin in the brain produces some of the side effects seen with serotonin-based antidepressants in humans, which include decreased libido and inhibited ejaculation. In contrast, drastic decreases in serotonin are associated in animals with increases in mounting. So is removing serotonin from the brain a deletion of sexual preference? An increase in libido? Something else? Serotonin has roles in many other behaviors and states, including aggression, feeding, impulse control and more.

Other scientists decided to find out for themselves. Mariana Angoa-Pérez, a molecular neurobiologist at Wayne State University School of Medicine in Detroit,  made her own version of the knockout mouse that the Rao group used, deleting or knocking out the enzyme that makes serotonin in the brain. But her male mice showed a marked preference for female urine and female mice, not the lack of sexual preference seen in the Rao group’s experiments.

 “Serotonin is related to hyper- or hyposexuality, but that’s different from sexual preference,” says Angoa-Pérez. “There are so many receptors and brain areas. It’s not a black-and-white issue.”

In particular, Angoa-Pérez and her collaborator Donald Kuhn, also at Wayne State, note that mounting behavior can be a sign of aggression and dominance between male mice. “Any behaviors we see related to sex are aggression-related,” Kuhn explains. He says that their experiments have shown that their low-serotonin mice are extremely impulsive and aggressive, “they just can’t control themselves.” They impulsively explore new environments, willingly try new foods (something normal mice are very shy about) and of course, willingly attack new cage mates. Kuhn and Angoa-Pérez published their findings on sexual behavior February 23 in PLOS ONE.

Now another study corroborates Angoa-Pérez and Kuhn’s findings. Daniel Beis and colleagues at the Max-Delbrück Center for Molecular Medicine in Berlin, bred their own version of the serotonin enzyme knockout mice. As in the other group’s study, the knockout males preferred female mice, and were hyperaggressive toward, well, everything. Beis concluded that decreasing serotonin in the brain produces deficits in social, not sexual, behavior. “I don’t like to say that serotonin is responsible for a single function, but that it’s modifying the system,” he says. “Stories cannot be so easily told as they sometimes are.”  Beis and his colleagues published their findings March 25 in Biology Letters. 

Rao, the lead author of the 2011 study, is fully confident in the results from his lab showing that brain serotonin regulates sexual preference. He says that the differences in the findings are probably the result of different experiment methods. “Three [of my] students have carried out these experiments before we were confident of the results,” he writes in an email. “Furthermore, the [Beis] paper did not use the standard methods used by us and others to perform the right experiments. Either they should have explained that our methods were wrong or they [should] repeat exactly what we did rather than doing experiments with different methods and analysis.”

The experimental results seem fairly contradictory. But it doesn’t necessarily mean that any one of those studies is wrong. “The data undoubtedly indicates the involvement of the brain serotonin system,” says Vladimir Naumenko, a behavioral geneticist at the Institute of Cytology and Genetics in Novosibirsk, Russia. But at the same time, he says, serotonin can be involved, without being the last word in sexual preference.

He also notes that differences in the methods used by the groups could play a role in the different experimental results. He explains that Beis looked at aggressive behavior, while Rao’s group did not. The Rao group used a mouse strain called C57Bl/6J, while the Beis group used a different one, C57Bl/6N, and the Kuhn group worked with a blended mouse, a mix of the C57Bl/6J mice and a strain called sv129. It may be a matter of just a few letters, but strains of mice can be strikingly different in behavior and have different levels of important brain chemicals. Finally, each group created their own knockout mouse to perform the studies. It is possible that different mouse strains — and knockouts — could result in dissimilar behaviors, from sniffing to mounting and everything in between. 

The studies raise the question of how, exactly, scientists should interpret mouse behavior, especially in the presence or absence of particular genes. Brain serotonin differences certainly could affect sexual behavior in mice. But it may not mean much, if anything, for humans. “I would say from a cultural perspective, results from animals are hardly transferable to such sophisticated things as human sexuality,” Beis says.

Mounting behavior in mice could have sexual intentions. It could have aggressive intentions. It could even mean something else. Time spent sniffing male or female could be a function of sex drive, or it could be the focused whiff of a less-sensitive nose. Odd behaviors could arise in mice that have never had serotonin, as their brains adapt to fill the gap. Many more mice will mount before scientists figure out exactly what role serotonin plays.

The seemingly inconsistent findings don’t mean that one scientist is wrong. As scientists continue to work on the sexy side of serotonin, they may discover new subtleties to mouse sexual behavior that change our knowledge of who mounts who and why. Wrestling over serotonin and sexual preference isn’t about “turning mice gay.” It’s about science working as it should — fumbling, somewhat gracelessly, toward understanding.