One of the grim ironies of the covid-19 pandemic is that the world has shown an unprecedented interest in data — and yet there has never been so much uncertainty around official statistics. Facebook and Twitter have become catalogues of terrifying charts, as people around the globe have tried to work out how quickly the SARS-CoV-2 virus is spreading. Meanwhile, academics and journalists have come up with lots of inventive ways to quantify the impact of the pandemic. The Economist has, for example, published analyses of tourist flows, footfall in major global cities and internet usage.

Yet we still don’t know the numbers that matter most. Because countries have different testing capabilities, and many carriers show no symptoms, we don’t know how many people the virus has infected. More worryingly, we don’t even know exactly how many people it has killed. As I reported in early April, the official covid-19 tolls in many countries exclude victims who were not tested before passing away. It often takes several days to establish and report the cause of death, creating a lag in the data. And if hospitals are overwhelmed, people with other conditions might not get the intensive care they need to survive. As a result, the numbers of deceased that get read out each night on the news are smaller than the true number of fatalities that the virus has already caused.

During medical crises such as this one, a better way to measure mortality is to look at a region’s “excess deaths”: the number of people who have died of all causes, compared with the average for the same period in previous years. I have been using two approaches to do this. The first is to take publicly available data about mortality, produced regularly by a handful of statistical bureaus, and estimate how many more people have died than usual in a selection of places. The second is to use figures from EuroMOMO, a network of epidemiologists in 24 European countries, which covers 350m people in its weekly reports. As the pandemic continues, I will update all of these datasets on our excess-mortality tracker, which is available free to all readers.

How I calculated The Economist’s estimates

One of the biggest challenges when it comes to measuring the spread of the virus is that every country collects its data differently. This is particularly true for the official covid-19 figures. For example, France has conducted a considerable amount of testing in care-homes, and includes fatalities from such institutions in its daily tolls, whereas almost all of the Netherlands’ reported deaths have occurred in hospitals.

Yet the data about deaths from all causes, which might seem more straightforward to collect, also vary in quality and frequency across countries. Spain’s Carlos III Institute, its national epidemiology centre, is publishing daily figures for each region — but these have a large lag, and are increased retrospectively when newer data come in (see chart below). By contrast, Britain’s Office for National Statistics produces a weekly total of the number of death certificates registered in its system, which are not revised in future. So the comparisons between countries are not entirely apples-to-apples. We have also had to use a simple method for our baseline of “expected deaths”: an average for the same week in previous years (ideally the past five years, but sometimes two or three, depending on availability).

Moreover, some countries have reported only partial mortality data, which cover a non-random selection of regions or victims. In Istanbul and Jakarta, the two non-Western places currently in our tracker, I am relying on tallies of the number of people who have been buried, as reported by local departments. These figures will not pick up everybody who has died, but they are still much higher than the official covid-19 tolls. Turkey does not even give regular breakdowns of covid-19 deaths by region, so we had to estimate that Istanbul had half of the national share, based on a report from April 1st. Given how slowly many developing countries publish their all-cause mortality data, I will probably have to use a similar approach to find numbers in South America, Africa, Asia and the Middle East.

Even some rich nations do not have complete data. Italy’s National Institute of Statistics (ISTAT) has released figures on all deaths for just 1,700 of the country’s 7,900 municipalities (or comuni) — only the ones that have both shown at least a 20% increase this year and claim to have accurate records. This makes it impossible to come up with a national tally of excess deaths in the European country that was hit hardest. However, because these comuni cover 7.2m of the 10m people in Lombardy, the worst-affected Italian region, I was able to produce an estimate for that area. I found a snapshot map of official covid-19 deaths for those comuni at the end of March, and used that to predict what share of Lombardy’s deaths these 7.2m people suffered each week.

All of this data-cleaning is fiddly and time-consuming, especially because the underlying spreadsheets are in different formats and languages. Whenever new data are published, I check them carefully, in case the definitions have changed. New York’s official death toll, for example, has started to include people who “probably” died of covid-19, according to the symptoms on their death certificates. To make this process more organised, I wrote a script in R, the programming language used most often by data journalists at The Economist, to do the cleaning for me. It is about 1,000 lines long, and spits out CSV files in the same format for each country. Those go into a Github repository, which Martín González, one of our brilliant interactive journalists, uses to make the dynamic charts for our tracker. We hope that, at some point, once we have made the code presentable, to make this resource publicly available — keep checking our GitHub.

How EuroMOMO comes up with its estimates

Although I have been able to find mortality numbers for many European countries, not all of them make these data publicly available in a timely fashion. Germany, for example, has only released nationwide figures up to March 13th (at the time of writing), since like Italy it struggles to coordinate lots of regional offices.

To get around this, I’ve collaborated with EuroMOMO’s epidemiologists. They collect mortality data on a weekly basis across Europe, which we will include on our tracker. And while they don’t break down the absolute numbers of fatalities by country, they do provide a trend chart for each member state on their website.

In this respect, EuroMOMO is a unique resource. No other continent has such a coordinated system. The United States has a “Pneumonia and Influenza Mortality Surveillance” tool, which publishes weekly total deaths for each state. But states take months to update their figures, which is why we have only included the city of New York in our tracker so far.

While working with EuroMOMO, I spoke to Lasse Skafte Vestergaard, to find out more about how the data are collected and what the limitations might be. Lasse coordinates the network, and helps put together the bulletin that the group publishes each Thursday. He is also an epidemiologist at Denmark’s national institute for infectious disease, and has specialised in influenza, malaria and parasitology.

JT: Lasse, your organisation is the only one that tracks excess mortality across an entire continent. How did this come about?

LSV: EuroMOMO’s network was set up in response to the global influenza pandemic in 2009, to ensure early detection of future public health events. Currently we have 28 partner institutions across 24 countries and regions. We also have representatives from the European Centre for Disease Prevention and Control (ECDC) and the World Health Organization (WHO) Regional Office for Europe, who both support the network activities and our hub financially, participate in technical discussions, and help our active strategy to expand geographical coverage, especially in Eastern Europe. For example, Ukraine has recently started to provide monthly data to the hub. One of the great strengths of our approach has been its consistency and close collaboration between participating countries, running week after week, year after year. Putting together the national data from all of them adds power to detect smaller changes than what individual countries may immediately see.

JT: How comprehensive are your numbers?

LSV: Our data come from national or regional statistical offices, and are typically based on information on deaths recorded in civil registers. Overall, our sample covers a substantial proportion of the European population, with more than 350 million people. However, the fraction covered varies quite a lot between countries, from 95–100% in many to only 15–20% in others. This all depends on local administrative systems and political decisions.

JT: Are there any limitations that people interpreting the numbers should be aware of?

First of all, we are providing mortality estimates, based on certain statistical models and assumptions, not the exact numbers of deaths. There will always be some level of uncertainty around them. Secondly, the coverage in some countries is only for a few selected regions or cities, for example in Germany and Italy. Since we are looking at national-level estimates, the observed figures may not adequately detect much higher mortality levels within smaller parts of a country. Finally, even the best quality reporting is not fully complete, and there are always a few days or weeks of delay. We apply an adjustment function in the statistical algorithms, but still, the estimates provided for the most recent weeks must be read with caution.

JT: Some readers might look at EuroMOMO’s charts and think that the death toll from covid-19 is no worse than from some of the recent flu seasons. How would you respond?

LSV: Given the dramatic mortality signals we are already seeing, especially in some parts of Europe, we can conclude that this pandemic is not just another flu — not at all. It is striking to see large variation, from extremely high excesses in one country to normal levels in neighbouring ones. We will later have to evaluate the effect of the various societal preventive measures, initiated with different speeds and levels of coverage. Then we can conclude what should be recommended in terms of preparedness for future pandemics.

We don’t yet know how long we will keep updating The Economist’s tracker. If the pandemic surges and then abates over several months, as countries go into and out of periods of lockdown, then excess mortality figures will be an important measure of how well social-distancing policies are working, and when they need to be enforced. I will keep digging away at official reports and spreadsheets in various languages, and expanding our ever-more tangled R script. If any readers spot data that we could use, or corrections that we need to make to the existing numbers, those would be very helpful — and I would be happy to hear about them at jamestozer@economist.com.

James Tozer is a data journalist at The Economist. You can follow The Economist Data Team’s work on Twitter.

Measuring the true toll of the pandemic was originally published in The Economist on Medium, where people are continuing the conversation by highlighting and responding to this story.

If, like me, you grew up with a love of writing but a pig-headed ignorance of algebra and algorithms, then the idea of learning to code has probably never crossed your mind. If it has, then it almost certainly repulsed you. Computer programming seems akin to sewage engineering: complicated, vital for society, and so mind-numbingly unpleasant that thankfully other people are paid to do it. When I started to think about becoming a journalist, I stubbornly overlooked college courses with a data science component. Who needs to learn the language of computers to tell a good story?

Working at The Economist has shown me that it can help — a lot. I started here as a paid intern, giving performance insights to our newsroom and social media team, having used some basic web analytics tools for a journalism start-up. I arrived with a patchy knowledge of Microsoft Excel and a couple of introductory books about statistics. Like anybody who has ever interned at a newspaper, I also wanted to write some articles. Number-crunching was a good way in, because (when done properly) it allows you to say something empirical and distinctive. With patient supervision from some of the paper’s statisticians, I was able to come up with a few stories using datasets from governments and a handful of companies.

But what if nobody has collected the data before? Discussions with other journalists about potential stories would often meander towards: “It’s an interesting topic, but gathering numbers on it would require a lot of manual collection or some help from the data team.” Enter the coders. The Economist has several journalists with computer-programming skills. Many of them are adept at “scraping” text and numbers from the internet — that is, writing scripts that open hundreds (or even thousands) of web pages and copy information from them into a spreadsheet. Some of the best off-the-wall articles that we have published, such as these ones about prostitution and drugs on the dark web, were based on web scraping.

Inspired by these stories, I decided to learn how to code. I picked up a few lessons along the way:

1. You don’t have to be a numerical wizard for web scraping to be worthwhile

For many of the articles that I ended up writing with scraped data, I used barely any maths, beyond simple sums and averages. My first story looked at the spread of LGBT venues — nightlife, saunas, community groups and so on — around rural Britain, with a straightforward measure of venues per capita in each county.

2. You don’t have to learn fancy statistical techniques

Many online coding courses promise a future career as a data scientist. (R and Python are the languages of choice among journalists: the former is great for statistics and charts, the latter is more flexible for everything else.) I ended up enjoying the modules about econometrics and machine learning, but took the ones on web scraping first. It is perfectly possible to learn how to code without any maths whatsoever.

3. You will suck at coding for ages, but stick with it

Teaching yourself computer programming is like learning a musical instrument. You will spend most of your time being infuriatingly bad at it, and the realisation that ten-year-old prodigies have already mastered it while you can barely string a few lines together will often make you want to hurl your keyboard against the wall. But once you reach a basic level of competency, you can suddenly do most of the cool things that you want to, albeit clumsily. There is a definite satisfaction in conquering a tricky website with your scraping code. If you focus solely on the basics of your chosen language and then the web scraping aspects, you should get there within 50 hours of practice. (Selenium, my preferred scraping library, takes a little longer.)

4. You can scrape responsibly

Scraping a website involves many automated visits, but most websites will start blocking visits from your IP address when they detect too many. So you should set up your scraper to avoid this. Make sure it covers only as many pages as you need to. Break your scraping tasks into separate sessions. And leave long load times between each page request (10 seconds will usually do it). Many websites have terms and conditions that forbid you from selling on the data, but journalistic research is usually in the clear.

Coding gives you the ability to come up with entirely novel information. I have used scraping to write articles about ski resorts, Donald Trump’s criticism of Google News, Led Zeppelin and several pieces for our sports blog. But scraping can also be a gateway drug to the countless other awesome software packages that languages like Python contain, from astronomy to video-game design. For example, I eventually earned a writing job on our data journalism team thanks to learning statistical modules. These helped me discover interesting patterns in all sorts of data, such as which countries are most likely to have liberal abortion laws, and which are most likely to be good at football.

A little coding knowledge can be a valuable skill for a journalist — and will only become more so.

James Tozer is a data journalist at The Economist.

Getting into a scrape was originally published in The Economist on Medium, where people are continuing the conversation by highlighting and responding to this story.