2 April 2020
In a post on 31 March 2020 I linked to a New York Times article about US data and estimates based on Kinsa Health’s internet-connected thermometers, which showed a significant drop in fevers vs Kinsa’s forecast. I did say that we should be “cautious about concluding that this represents a decrease in COVID-19 infections.” It turns out that I should have put even more caveats when I shared this article.
A friend who knows much more about Kinsa’s dataset and methodology, and is close to the company, recommends even greater caution. He does not want the comments attributed to him but gave me permission to summarize them; what follows is my summary and any errors are my own:
- The data has been historically accurate on broad trends, correlates with the CDC data, and has the advantage of being released earlier.
- Using the data more precisely — for example, at a narrow geographical level — is more challenging.
- By way of background, there is no precise baseline (“ground truth”) against which to train models. We don’t know exactly who has the flu. So the “percentage of people who are sick” is at best an estimate, which is designed to correlate with the CDC’s metric. It’s consistently calculated so is probably a reasonable proxy for the flu at a national level.
- However, there are a lot of challenges in extrapolating from this to what we want to know about COVID-19.
- First, it is much harder to be accurate at a local level. CDC data is not published locally so the model can’t be changed; and some locations have very few thermometers.
- Secondly, users are not representative; they tend to be younger and tech savvy. That poses a big problem in trying to extrapolate to COVID-19.
- Third, user usage patterns can change over time. Worse yet, they can change in reaction to new stories like this one! So even if past correlations were pretty good, they might not be the same today.
- Finally, flu incidence is more random than one might think, meaning that it’s hard to say what “atypical” levels of fever are.
It’s a helpful reminder that at a moment when more data, more studies, and more so-called experts are being thrust at us than ever before–and when many scientific papers are being distributed as pre-prints before going through the peer review and publication process–we need to be extra-cautious when looking at data and claims.