About Research 2
In today's great flood of media items on scientific topics including the environment, not only journalists but also governments, interest groups, general writers and websites (including Ozcranes!) quote science to support their claims or policies. Are they using the most reliable sources, and quoting them fairly?
Some stories, claims or even major policy will turn out to be citing anecdotes: interesting, but not research (Science Tour Part 1). Some do quote research studies, though. Here in Part 2 of the Science Tour, we compare the places these research studies are published: talks at conferences, some government or NGO reports, scientific books, scientific journals, and more. How do ‘peer-review’ and ‘grey literature’ relate to reliability?
What is..
..‘Peer Review’
A kind of scientific quality control where scientists open up their work for scrutiny by other experts, before publishing. Independent expert reviewers, mostly volunteers, are known as ‘referees’: so a ‘refereed journal’ is the same as ‘peer-reviewed’.
This can be a pretty gruelling process. Think of Carroll D. Littlefield (our scientist from the Abstract in Part 1) wading through swamps year after year, tracking broken eggshells and predator scats, dodging perhaps angry crane parents.. then there's the analysis phase, writing up, and submitting for publication. Scientists and research students would certainly empathise with these quotes from the Sense About Science peer-review Working Party –
Few outside the scientific world can be expected to know about the immense effort, and even pain, that is experienced before research is ready to be published as a peer-reviewed paper in a scientific journal… Scientists submit their research findings to a journal, which sends them out to be assessed for competence, significance and originality, by independent qualified experts who are researching and publishing work in the same field (peers). This is known as ‘peer review’. Despite its extensive use and recognition among scientists in assessing the plausibility of research claims, in the rest of society very little is known about the existence of the peer-review process or what it involves.
Is peer-review the same as editing? No, most scientific books and conference proceedings are not peer-reviewed. They have editors, sometimes with major input, but this is in-house vetting, not external and independent. The same applies to supervision of academic dissertations (theses).
There's about 11,200 peer-reviewed science journals. The most influential are the 8,500 (76%) that are ‘indexed’ (make up the scientific article databases). Size doesn't matter – peer-reviewed, indexed journals may be small or large, fortnightly or annual, commercial or fully voluntary. Citations are also used to indicate prestige, although the Working Party notes this can be manipulated.
You can download a summary or the full (pdf 470kb) Working Party report, covering many issues surrounding peer review, its crucial role in the process of publishing and testing new scientific results, its problems and its future. There's also a brochure from the UK Science Media Centre, ‘Peer Review in a Nutshell’ (pdf 107kb).
Peer Review and the Acceptance of New Scientific Ideas: Discussion paper from a Working Party on equipping the public with an understanding of peer review. May 2004, Sense About Science (UK). Available online, as above.
TOP..‘Grey literature’
Grey is about universal availability. Academic journals are not grey literature. Is this because they are peer-reviewed? No, only some scientific books are peer-reviewed but most libraries will define them as non-grey. Is ‘grey’ low quality? Not necessarily at all, eg a scientific journal article will be just one fraction of a (grey) dissertation or (grey) technical report that holds much extra material. Some grey literature is prestigious.
Types of scientific research literature
| Type | Peer-reviewed | Grey |
|---|---|---|
| Science journal | 11,200 (76% indexed) | No |
| Science Book | A few | No |
| Conference talk or program | No | Yes |
| Conference Proceedings | A few | Yes |
| Dissertation or thesis | No | Yes |
| Government report | No | Yes |
| Interest Group report | No | Yes |
| Technical paper | No | Yes |
Availability works like this: scientific journals and books are marketed universally (albeit sometimes not-for-profit) and anyone with a (very) good budget or library, can subscribe to, or read any science journal or book. Also in theory, journals keep their archives for ever, giving long-term continuity. But items like government agency reports and conference proceedings (papers collected into a volume after a conference) have to be specially ordered and have a limited life on the market. Some are confidential and never released publicly, despite containing important data and conclusions.
The limited availability of ‘grey’ publications makes writers wary of citing (quoting) them in support of their research unless absolutely necessary, as readers may have trouble tracking down the source to query or extend the research. But (non-grey) out-of-print books can be equally hard to find!
Crane research
← Common, or Eurasian Crane Grus grus (International Crane Foundation). Journal abstracts for many research papers on the Common Crane are available via the ECWG (European Crane Working Group) website
Literature types aren't mutually exclusive, there's a natural progression through the life of a research project. A PhD student studying cranes might give a talk to a conference with an abstract in the program; the final dissertation may be up on the university website, or the International Crane Foundation Digital Library; key findings of the research may be written up for scientific journals (abstracts usually available free on the web); the researcher may contribute a chapter to an edited scientific book; then might be hired to do a report on cranes for a government agency or wetland conservation organisation. All are very valuable resources, but it's fair to say that articles in peer-reviewed, indexed journals cited by later authors, will have the most widespread circulation and scientific standing. Over the years much key crane research has appeared only in hard-to-obtain edited conference proceedings, perhaps very frustrating for all concerned. But there are more and more submissions to science journals and some proceedings are now published on-line at the International Crane Foundation.
Finally: Checking source quality
You can apply the Science Tour principles anywhere people are quoting science to support their statements – the media, websites including Ozcranes.net, government reports and NGO documents. Are claims based on studies using scientific methods; have the principal findings of these studies appeared in peer-reviewed publications (which); do citations in the studies really support their arguments; finally is the media (or whoever) fairly quoting the studies, or stretching them beyond the original?
Of course governments must govern and make decisions every day, and lobby organisations have their campaigns, and so on. But you may be surprised how much policy, legislation and opinion on scientific topics is not backed by science, although some might be, in time.
«Back to Ozcranes Research Home
