<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Giulio Bertoli on Medium]]></title>
        <description><![CDATA[Stories by Giulio Bertoli on Medium]]></description>
        <link>https://medium.com/@GiulioBertoli?source=rss-11b5cf5ebbed------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*i71BapqOAFcyTjrw.png</url>
            <title>Stories by Giulio Bertoli on Medium</title>
            <link>https://medium.com/@GiulioBertoli?source=rss-11b5cf5ebbed------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 14:34:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@GiulioBertoli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Covid19 Apples and Oranges: a Data Problem.]]></title>
            <link>https://medium.com/@GiulioBertoli/covid19-apples-and-oranges-a-data-problem-521126089418?source=rss-11b5cf5ebbed------2</link>
            <guid isPermaLink="false">https://medium.com/p/521126089418</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[covid19]]></category>
            <dc:creator><![CDATA[Giulio Bertoli]]></dc:creator>
            <pubDate>Tue, 24 Mar 2020 08:28:34 GMT</pubDate>
            <atom:updated>2020-03-24T13:06:38.751Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*jUZWPRRYCvoPGjB8HFbqWQ.png" /></figure><p>I’ll cut to the chase as I am no big fan of introductions when they’re written just for the sake of it. I have a serious problem with the way the world is looking at the data of this pandemic.</p><p>As I am writing this piece I’m quarantined in Milan, Lombardy, the area in Italy where we registered most cases. Things are not looking good, many people are dying, many got and still get infected. Just last Sunday the Italian government put in place stronger quarantine measures. I’m lucky that I can work from home, but not everyone can. The impact on the economy is truly unpredictable, we still don’t know how bad it will get.</p><p>Basically, in case you didn’t notice, it hit the fan. Big time.</p><p>Back to the point, my problem is simple. Everybody is looking at data. Data here, data there. Number of cases, death rates, daily increment, active cases, averages of this and that and whatever, estimates about peaks and R0 and SIR models and so forth. I also fell for it at the beginning, trying to give a rough estimate of the “peak” in Italy, which I thought it was going to be on last Saturday, guilty as charged. <br>You have experts claiming that in China stuff happened this or that way for one or the other reason, in Germany there are only a few deaths because they have lot of ICU beds, in Italy people are dying because they are very old and young people interact a lot with the old ones, Japan is a statistical anomaly somehow for nearly everybody, South Korea was successful because of the widespread testing strategy, the Netherlands claiming that they don’t need a lockdown based on some kind of model they have of other countries, etc. Everybody is comparing countries, because suddenly everybody feels the need to say something and understands therefore a lot about data and, mostly, about how to interpret it.</p><p>The point I’m trying to make is that nobody has looked for two simple, very simple things, which incidentally is also the reason why you should stop listening to all this information bonanza and worry only about what happens around you. I’ll come back to it. The two things are: <br>1) a common framework; <br>2) testing.</p><p>I dare you to find a single document stating that countries have agreed on the methodology regarding the collection and gathering of the numbers they share. That establishes whether you should or shouldn’t test post-mortem when calculating the death rate. That clearly states what counts as Covid19-death. That makes sure that there is a minimal testing coverage above a certain population threshold. That explains why it doesn’t seem to bother anybody that some countries release the number of tests performed, if they do, each in its own way, whenever they want, sometimes giving rough estimates and in general in the “as-I-damn-please-so-be-thankful-we’re-doing-it” way.</p><p>I am a physicist by formation. The first, and really the FIRST thing they teach you in a physics degree is a simple one: NEVER compare apples to oranges. The first thing you should do when you’re making a comparison is to make sure you are comparing two things in a meaningful way, i.e. those things are measured in the same way. The two numbers should MEAN the same thing.</p><p>I truly get the need to understand how bad this s**tstorm is, believe me, and that we all want to look for data as a guiding principle for the decisions we make. We want to find common ground so we can all agree on what needs to be done, and numbers seem like a good place to start. But we need to agree on the numbers first.<br>This reminds me of a physics urban legend (probably true, never bothered to find out) of two particle physics groups using the +1 and -1 metrics respectively that couldn’t agree on the results of the experiments. Well, we’re never going to agree if we use different metrics.</p><p>The way I see it, the only job of the WHO was to make sure we could get information right. They failed miserably, and let us not forget:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*a40IJi6kLeZIqj8FfSyvow.png" /><figcaption>Absence of evidence mistaken for evidence of absence. Again.</figcaption></figure><p>The point is not to attack the WHO. I don’t exactly know about everything they do, I don’t know if I could do/have done better, so I don’t have a say in it. What I *can* say is that if you supposedly are, or behave like, the guy in charge of making sure stuff like pandemics doesn’t happen, you better NOT mistake absence of evidence for evidence of absence. Because to me that’s pretty much your only job and you should be prepared.</p><p>But let’s say you do make the mistake (as they did) and the pandemic starts, then at the VERY LEAST you should have every single country giving you the numbers according to some standard that you fix, and should have fixed already. Otherwise, just as it is happening today, people will start pondering and look at “data from the WHO” like it’s some kind of Bible, while what they are basing all their models and statistical analysis on is just a bunch of numbers the WHO itself doesn’t even know what they are, most of the time.</p><p>Let us be concrete and take an example: death rates in Italy and Germany. Maaaaaany people have been talking about it, my Twitter feed is giving me a different take on the issue every day, even from some proud social scientists with dubious linear regression skills that somehow feel it is time to show off:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*u0AbTVJ1JxP1mYTQEiUuZQ.png" /><figcaption>How to NOT do linear regression 101.</figcaption></figure><p>If I ever dared to present a regression like this one to my Russian PhD advisor he would have sent me back to primary school. Anyway, other stuff may make more sense, and there is a lot of effort being done online.</p><p>So, about this death rate? Why is it so different? Is it the number of ICU beds, much larger in Germany? The fact that the virus got to older people only in Italy because old people in Germany are locked up in hospices (yes, apparently that could be a major factor)? Is it because Germany is still at the early stage of the outbreak, and numbers will converge eventually? Better testing meant Germany got more mild-symptoms cases? Will we know before it is all over? When it will be over?</p><p>There is one thing that nobody seems to care about: it’s not the same number.</p><p>Italy does post-mortem testing, while Germany assumes that you will be diagnosed before you die (references are at the end of the article, it’s all there folks). How big is this factor? Do we know? Of course we don’t, and people just “<em>assume</em>” this factor is not big when looking at the numbers. Try sell this pitch to a company, “<em>assuming</em>” stuff about your data when comparing to the competitors. Any reasonable manager will stop you right there, show you the door and tell you to never come back. Why? Apples and oranges. <br>So you shouldn’t compare different numbers and that’s it, and each time you read an article stating “The death rate here or there is …” or “Germany is successful compared to others because…” you should ask yourself: what are they talking about? Are they comparing the same number? Did they even check?</p><p>Let’s try for fun to give an estimate of the factor in question: you should first subtract from the number of deaths in Italy the ones that are tested post-mortem. Do you know how to do it without a source? Nope, but <em>let’s assume</em> that many of the people with 3 or more diseases will die after getting Covid19. Their health is already very compromised and they die right after getting the virus, so they haven’t been tested before as <em>it is reasonable to think</em> that they will die of non-respiratory issues. <em>Let’s say</em> you test many of those deaths, just as Italy is doing right now. How big is this share in Italy’s death number? <em>Preliminary</em> analysis from the ISS (Istituto Superiore di Sanità) shows that about 48% of the total number of deaths had 3 or more diseases. Pretty big, right? I already decreased the death rate in Italy by one half. Should I go on? The same analysis shows that only the 0.8% had no previous disease. Does it mean we should cut down the number of 99.2%? No. People are dying of Covid19 here, and that’s a fact. Notice all the “<em>let’s assume, let’s say, preliminary</em>” I just used. This was an exercise to show that without a common framework, basing everything on assumptions, we are getting nowhere: we simply don’t know this factor, and that’s it, and estimates may be more or less reasonable but they will always be off, and we do not know by how much. The reason being that this is a dynamical, multiplicative process.</p><p>Again, I’m not trying to say what should be the number you should look at, the opposite if anything, just that it’s NOT the same number as Germany. Apples and oranges.</p><p>Let me be clear: I hope for all the Germans and for everybody in genereal that the situation there will be better than Italy’s and that only a few people will die, it would be great and I wish there could be only a few deaths everywhere. Germany has all the cards to overcome this, more than anybody else in my opinion, and we should all take example. At the same time, when you look at the reaction of Germany’s RKI (official authority about German data, always somehow lagging of a couple of days with respect to the rest of the world in presenting data and never releasing data on how much they test) when confronted on the issue you find something like (see references): “We just <em>assume</em> everybody who dies gets tested before”. Assuming, are you?</p><p>So, what can we agree on? How do we search for a common framework? One thing that everybody seem to agree on is the number of diagnosed cases. My problem with this is the following: it depends, yet again, on testing. How much, who and when you test is crucial. Let’s look at another piece of work, this time a pretty good one, from Mark Handley:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Ep606D3lkTC4Tj9eaxOFKQ.png" /><figcaption>CoVID 19 Worldwide Growth Rates. See <a href="http://nrg.cs.ucl.ac.uk/mjh/covid19/#en">http://nrg.cs.ucl.ac.uk/mjh/covid19</a>.</figcaption></figure><p>You may have seen this already somewhere else, in slightly different forms. The graph shows a log scale, against time, of the number of cases. The benchmark is Italy, and the plot tries to show how much of an offset other countries have in following Italy’s path. This might be useful for policymakers to know when to go full lockdown or something, as many countries already did after having a look at what’s happening here.</p><p>Sure we can all agree on this, right? It’s just the total number of cases, after all! Well, how about this: this information is almost useless if you don’t have the same number of samples.</p><p>There are countries (Italy, Austria, Canada, South Korea…) that are doing a pretty good job in the way they share data about testing. Some other (Germany, Spain …) are doing a crappy one and you have to search in the news for something people heard at press conferences, but no official number is available. France is doing a weekly update. I’m going to focus on Italy, Germany, France and Spain just to show my point. You could add other countries. Here is a plot that, I believe, can give a rather different interpretation to the whole story:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*N-oVd62E4gCAvrTgNjuA6Q.png" /><figcaption>Snapshot of the 15th of March</figcaption></figure><p>It shows the total number of positive cases versus the number of tests performed up to the same day, the 15th of March. It’s almost 10 days old, because that’s how long ago many countries said something about how much they test, but you get the picture.</p><p>For Germany and Spain I had to interpolate (guilty, I know, but not my fault) because I had only 2 or 3 data points coming from press conferences (rough estimates). Please let me know if you get hands on a good dataset (my guess is: you won’t). For France and Italy I have official numbers. Are you wondering why is France behaving so well in Mark’s plot? You think it is and it will be better than Italy as they had more time to prepare? Nope. They just tested much less, and if you compare you’ll see that when Italy had performed the same number of tests, they actually had less cases than France. But looking at Mark’s plot, it seems as France is just lagging Italy and following its path, doing somewhat better! Again, apples and oranges.<br>So in France, I think, nobody really knows what’s going on, but given that the testing positivity rate is similar to Italy, it appears as if the dynamics is the same, just lagged, and nobody seems to ask: what if the testing positivity ratio is approaching a constant (it is)? If they start testing as much as Italy, what would be the situation there?<br>This orange looked like an apple? It happens, much more than you’d think.</p><p>The plot is made with data from the 15th of march. You can tell that France and Germany have the same number of cases, ~5k. As of today, the 23rd, Germany has almost 25k and France just 16k. Did the situation get out of hand in Germany? No. The claim that Germany (Spain) is just lagging a little more (less) is also clearly misrepresented. A whole different dynamics is going on there, and I won’t go in the details about process dynamics, I think the plot I made speaks by itself. <strong>The “total number” is just a number taken at very different testing capacities, hence it’s capped</strong>. It’s a misrepresentation of reality to say that the virus is spreading similarly, with a lag in time. We just don’t know because we don’t test in the same way and, most importantly, at the same rate. And because of that, even the number “total number of cases” in the WHO countries-data means something else, and should be replaced by “total number of cases given the number of tests”.</p><p>I could go on and tell you more differences in methodology that basically resemble the two above. How Germany has different testing guidelines for test selection (it’s in the references, I’m not making it up). How I purposely never mentioned the US, simply because I fear they don’t have a clue themselves of what is going on there (+8k new cases in one day, as soon as they started testing at a reasonable pace, what does it tell you?). I won’t go on, though, I think I made my point. All I’m saying is just this: <em>don’t compare the numbers unless you go real deep into what they </em><strong><em>MEAN</em></strong>.</p><p>What can we do, then? Should we just say “Screw it, I’m looking at the numbers anyway because that’s all we got”? (Yes, that’s pretty much what everybody does). I’m also doing it, sometimes, they still give me a picture. But let us not forget that this picture is taken out of focus by a three-year-old with a finger on top of the lens.</p><p>I believe there is another solution. Just look at the data <em>within each country, and compare to it internally. </em>As every country is handling things in their own way, you should look at differences among regions of a single country, where you can be sure there is a common framework and sound data. I haven’t done it yet, I may do it in the coming days if I have time. Important differences may be found within the country itself that will surely give you a better look at the phenomenon by actually having numbers that all mean the same thing.</p><p>Local focus vs global, in essence, may come to the rescue. This is also what I claimed at the beginning: look what happens around you, and focus on that. This thing will be solved if, <strong>locally</strong>, everybody acts in order to contain it. And in general, if you really <em>must </em>compare countries to take decisions on what to do, do it very carefully, looking at numbers in the way I did above, asking yourself everything about those numbers. And if you don’t have the time, my line of thought is simple: look at the worst case there is, and prepare for worse. Why risk it, basing yourself on <em>assumptions</em>?</p><p>I do wonder why everybody feels the need to compare countries. Maybe it’s because they cancelled team sports and people have nothing to watch, so they are rooting for their own country (which in some kind of sick way hopes that other countries may go worse then theirs). I don’t know, I’m just <em>assuming</em>.<br>I had hoped that in a truly global problem we would have been able to cooperate and agree on a few basic things. At least on how we count. We failed, in my opinion. But it is nice to see glimpses of humanity: Germany is to receive patients from Northern Italy (after initially block transit of protective masks and suites to Italy, but well), Russia, Cuba and China are sending doctors, and I’m sure there are many other examples of cooperation: let’s have more of that, instead of comparing apples to oranges.</p><p>References in random order and languages, if I skipped something just ask me and I’ll find it for you:</p><p><a href="https://github.com/pcm-dpc/COVID-19">https://github.com/pcm-dpc/COVID-19</a><br><a href="https://www.iss.it/">https://www.iss.it/</a><br><a href="https://mobil.stern.de/amp/gesundheit/covid-19--so-oft-wird-wirklich-auf-das-coronavirus-getestet-9189186.html">https://mobil.stern.de/amp/gesundheit/covid-19--so-oft-wird-wirklich-auf-das-coronavirus-getestet-9189186.html</a><br><a href="https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Falldefinition.pdf?__blob=publicationFile">https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Falldefinition.pdf?__blob=publicationFile</a><br><a href="http://nrg.cs.ucl.ac.uk/mjh/covid19/">http://nrg.cs.ucl.ac.uk/mjh/covid19/</a><br><a href="https://ourworldindata.org/coronavirus">https://ourworldindata.org/coronavirus</a><br><a href="https://www.n-tv.de/panorama/Warum-ist-die-Sterblichkeitsrate-so-niedrig-article21657232.html">https://www.n-tv.de/panorama/Warum-ist-die-Sterblichkeitsrate-so-niedrig-article21657232.html</a><br><a href="https://www.repubblica.it/esteri/2020/03/23/news/coronavirus_germania_pazienti_italiani-252076289/">https://www.repubblica.it/esteri/2020/03/23/news/coronavirus_germania_pazienti_italiani-252076289/</a><br><a href="https://www.theweek.in/news/world/2020/03/23/communist-cuba-sends-its-medical-brigade-to-Italy.html">https://www.theweek.in/news/world/2020/03/23/communist-cuba-sends-its-medical-brigade-to-Italy.html</a></p><p>And thanks to this guy who is basically saying in one tweet what I just said above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*TxLupVYtTLsblQjX-qARfg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=521126089418" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>