ANN ARBOR—Darwin referred to the origin of species as "that mystery of mysteries," and even today, more than 150 years later, evolutionary biologists cannot fully explain how new animals and plants arise.

For decades, nearly all research in the field has been based on the assumption that the main cause of the emergence of new species, a process called speciation, is the formation of barriers to reproduction between populations.

Those barriers can be geographic—such as a new mountain, river or glacier that physically separates two populations of animals or plants—or they can be genetic differences that prevent incompatible individuals from producing fertile offspring. A textbook example of the latter is the mule; horses and donkeys can mate, but their offspring are sterile.

But now a University of Michigan biologist and a colleague are questioning the long-held assumption that genetic reproductive barriers, also known as reproductive isolation, are a driving force behind speciation. Their study was published online publication in the Proceedings of the National Academy of Sciences on Sept. 2.

"Most research on the formation of species has assumed that these types of reproductive barriers are a major cause of speciation. But our results provide no support for this, and our study is actually the first direct test of how these barriers affect the rate at which species form," said Daniel Rabosky, assistant professor in the U-M Department of Ecology and Evolutionary Biology and a curator of herpetology at the Museum of Zoology.

Rabosky and Daniel Matute of the University of Chicago reasoned that if genetic barriers to reproduction are a leading cause of new species, then groups of organisms that quickly accumulate those genes should also show high rates of species formation.

Photo of two cinnamon-chested bee-eaters superimposed on an evolutionary tree showing two-thirds of known bird species. Rabosky and Matute used the tree to test whether the ability of bird species to interbreed with other species is related to the rate at which they formed new species. Tree illustration and bird photo by Daniel Rabosky

They tested that idea by comparing speciation rates to genetic indicators of reproductive isolation in birds and fruit flies. They chose birds and fruit flies because extensive data sets on interspecies breeding experiments exist for both groups. The researchers used evolutionary tree-based estimates of speciation rates for nine major fruit fly groups and two-thirds of known bird species.

Rabosky and Matute created computer models to carry out the comparison, and the results surprised them.

"We found no evidence that these things are related. The rate at which genetic reproductive barriers arise does not predict the rate at which new species form in nature," Rabosky said. "If these results are true more generally—which we would not yet claim but do suspect—it would imply that our understanding of species formation is extremely incomplete because we've spent so long studying the wrong things, due to this erroneous assumption that the main cause of species formation is the formation of barriers to reproduction.

"To be clear, reproductive barriers are still important on some level. All sorts of plants and animals live together in the same place, which couldn't happen without reproductive barriers. But our results question whether genetic reproductive barriers played a major role in how those species formed in the first place."

While speciation is often defined as the evolution of reproductive isolation, the new findings suggest that a broader definition may be needed, Rabosky and Matute conclude.

Over the last decade or so, the pursuit of the genetic underpinnings of speciation has led to reports on the discovery of a handful of "speciation genes," defined as genes that contribute to reproductive isolation between species. The studies include a 2008 Nature paper that reported the first speciation gene in a mammal.

But if the findings of Rabosky and Matute prove to be more widely applicable to other organisms, speciation genes probably play a minimal role in the formation of species, they said.

"The whole enterprise of finding 'speciation genes' is potentially irrelevant to understanding the origin of species," Rabosky said. "But our study is certainly not going to be the final word on this. If anything, our results indicate that a lot more data will be needed before we can conclusively link the mechanisms we usually study in the lab to the patterns of species formation we see in the natural world."

The researchers speculated that part of what is missing from speciation studies is, paradoxically, extinction. Some researchers have suggested that speciation might be limited primarily by factors associated with the persistence of new species. These models propose that it is relatively easy for a species to split into new species, but the vast majority of new species do not persist over geological timescales.

The research reported in PNAS was supported by the Miller Institute for Basic Research in Science at the University of California-Berkeley, the National Science Foundation and the Chicago Fellows Program at the University of Chicago.