Taking the wider view

The ‘quality’ of a research paper is a tricky thing to quantify – journal impact factors and citations have gone some way to do this, but they don’t take into account how wide an audience can be for scientific research. That’s where alternative metrics come in says Cat Williams.  

Your annual ‘Top 100’ highlights 2018’s most popular scientific research papers… what does this mean, and are there any trends?

Altmetric publishes the Top 100 every December, and the list is populated based on the online attention data that we have collected for research publications published in the preceding 12 month period. These attention data are captured from a range of online sources – it might be, for example, that a research paper has been mentioned in or linked directly from an online news article, or that it’s been referenced on a Wikipedia page. Our system pulls all of these ‘mentions’ together and calculates an ‘Altmetric Attention Score’ based on the volume and type of mention.

The Top 100 list showcases the papers that have received the most attention from all of the sources we track; so arguably those which have achieved the most reach amongst academic and broader audiences. Due to the public-facing nature of this engagement, we typically see topics that resonate with a broader audience dominating the list – big global challenges like climate change, health and dietary issues that people experience for themselves, and also a mixture of research that reflects the public agenda of the time; the devastating impacts of Hurricane Maria in Puerto Rico and the spread of misinformation online topped the 2018 list.

So, how exactly do you track the popularity of research papers? Our system is automated but complex! First, we ensure that the places where research papers are hosted online are indexed by our system. This includes academic publisher websites, but also institutional repositories, and an increasing number of preprint servers, where papers are often posted prior to formal publication.

At the same time, we have systems that monitor a range of online sources, including thousands of mainstream media outlets, public policy sites, patent offices, blogs, social media, Wikipedia, YouTube. We look for links back to the domains that host research publications within those sources (and in the case of news and policy, also employ text-mining to pull out author and journal names, which can then be matched to their research paper). These instances are what we call ‘mentions’, and once they’ve been found our system pulls them all together, scores them collectively based on the source of the attention, volume, and author of the mention, and then presents all of the information via our ‘details pages’, where users can explore all of the original mentions of the research paper. Combined, these data are called ‘altmetrics’ – alternative metrics that help tell a story about a piece of published research. A coloured visualisation also helps people see at-a-glance which sources the paper has received attention from.  

Why should scientists pay attention to alternative metrics? The impact factor and academic citations have long been used as a proxy for the quality of a research paper, but actually they only tell a small part of the story. In order to truly understand the influence and potential impacts of research, it’s important to take a much wider view. This is where altmetrics come in – they help capture and collate the attention an item has received from audiences beyond the academy. For the authors of these papers this is invaluable; they can see who is talking about their research and why and use this information to demonstrate its value to funders and other stakeholders.

Importantly, altmetrics aren’t a measure of quality (only actually reading the paper can really provide that), but they can be useful for determining its reach. Globally, there is an increased focus on making publicly-funded research and academic expertise more accessible to a wider audience, and altmetrics have a big role to play in understanding the extent to which this is being achieved.

Does this suggest the days of more traditional measures of academic impact coming to a close? Presumably, the traditional journals with their cherished impact factors been resistant to this new way of calibrating author success?

Citations certainly still have an important part to play in academic discourse, and in how we understand the influence of a piece of research. That said, academic publishers big and small have actually been amongst the first and the fastest to adopt altmetrics and display them for their articles, seeing them as a useful way of providing authors and readers with additional context and insights into how a publication has been received, often long before any citations have accrued.

Altmetrics also have the benefit of offering insights at the individual article level – something that journal-level impact factors are widely-acknowledged as failing to accurately represent. Crucially, altmetrics are not intended to replace more traditional bibliometrics such as citations, but to act as complementary data that can help us better understand how diverse audiences are engaging with academic work.  

What else can the scientific community get out of altmetrics? One of the most powerful aspects of altmetrics are the opportunity they provide for people from different geographies, backgrounds and levels of expertise to connect. Within the underlying data and detail of exactly who is discussing their work, researchers might be able to identify potential new collaborators, or even public groups who might benefit from further engagement.

Aside from offering them new ways to better demonstrate the value of their work to gain funding, being able to show that they are taking an active part in sharing their research more widely can also lead to new career opportunities and a more positive public profile for themselves and the research organisations that they belong to.

Perhaps most importantly, altmetrics offer a new way for someone to see and show that they are making a difference and encourage the scholarly community to focus on sharing their valuable findings with as many people as possible.   

Cat Williams is COO at Altmetric, Digital Science – a data science company that provides altmetrics to publishers, institutions, researchers, funders and other research organisations around the world.