EvidenceAlerts

COVID-19 COVID-19 Evidence Alerts from McMaster PLUS is a new service that alerts users to current best evidence for clinical care of people with threatened, suspected or confirmed COVID-19 infection.
Visit the site Sign up for alerts
Wynants L, Van Calster B, Bonten MMJ, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020 Apr 7;369:m1328. doi: 10.1136/bmj.m1328. (Systematic review)
Abstract

OBJECTIVE: To review and critically appraise published and preprint reports of prediction models for diagnosing coronavirus disease 2019 (covid-19) in patients with suspected infection, for prognosis of patients with covid-19, and for detecting people in the general population at risk of being admitted to hospital for covid-19 pneumonia.

DESIGN: Rapid systematic review and critical appraisal.

DATA SOURCES: PubMed and Embase through Ovid, Arxiv, medRxiv, and bioRxiv up to 24 March 2020.

STUDY SELECTION: Studies that developed or validated a multivariable covid-19 related prediction model.

DATA EXTRACTION: At least two authors independently extracted data using the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist; risk of bias was assessed using PROBAST (prediction model risk of bias assessment tool).

RESULTS: 2696 titles were screened, and 27 studies describing 31 prediction models were included. Three models were identified for predicting hospital admission from pneumonia and other events (as proxy outcomes for covid-19 pneumonia) in the general population; 18 diagnostic models for detecting covid-19 infection (13 were machine learning based on computed tomography scans); and 10 prognostic models for predicting mortality risk, progression to severe disease, or length of hospital stay. Only one study used patient data from outside of China. The most reported predictors of presence of covid-19 in patients with suspected disease included age, body temperature, and signs and symptoms. The most reported predictors of severe prognosis in patients with covid-19 included age, sex, features derived from computed tomography scans, C reactive protein, lactic dehydrogenase, and lymphocyte count. C index estimates ranged from 0.73 to 0.81 in prediction models for the general population (reported for all three models), from 0.81 to more than 0.99 in diagnostic models (reported for 13 of the 18 models), and from 0.85 to 0.98 in prognostic models (reported for six of the 10 models). All studies were rated at high risk of bias, mostly because of non-representative selection of control patients, exclusion of patients who had not experienced the event of interest by the end of the study, and high risk of model overfitting. Reporting quality varied substantially between studies. Most reports did not include a description of the study population or intended use of the models, and calibration of predictions was rarely assessed.

CONCLUSION: Prediction models for covid-19 are quickly entering the academic literature to support medical decision making at a time when they are urgently needed. This review indicates that proposed models are poorly reported, at high risk of bias, and their reported performance is probably optimistic. Immediate sharing of well documented individual participant data from covid-19 studies is needed for collaborative efforts to develop more rigorous prediction models and validate existing ones. The predictors identified in included studies could be considered as candidate predictors for new models. Methodological guidance should be followed because unreliable predictions could cause more harm than benefit in guiding clinical decisions. Finally, studies should adhere to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) reporting guideline.

SYSTEMATIC REVIEW REGISTRATION: Protocol https://osf.io/ehc47/, registration https://osf.io/wy245.

Ratings
Discipline Area Score
Infectious Disease 7 / 7
Family Medicine (FM)/General Practice (GP) 6 / 7
General Internal Medicine-Primary Care(US) 6 / 7
Hospital Doctor/Hospitalists 6 / 7
Internal Medicine 6 / 7
Public Health 6 / 7
Pediatric Emergency Medicine 5 / 7
Occupational and Environmental Health 5 / 7
Emergency Medicine 6 / 7
Intensivist/Critical Care 5 / 7
Comments from MORE raters

Emergency Medicine rater

The models are at high risk for bias and many have poor discrimination, so front-line clinicians cannot use this information.

Emergency Medicine rater

In the few months since COVID-19 was first diagnosed and described, these authors found 27 publications describing 31 prediction models for the disease. The main finding was that the included studies were at high risk of bias and probably overestimated the accuracy of the models. Models for diagnostic accuracy had higher C statistics than those for prognosis in general, but I would not at this point endorse strict adherence or use of any of these unvalidated models.

Emergency Medicine rater

This is a nice (and necessary) reminder that just because a study/review is published in a peer-review journal, doesn`t mean it`s ready for prime time. In fact, in some cases, it`s possible that early adoption may be harmful. The article makes a good argument that even during a global crisis (e.g., COVID19 pandemic), looking for new ways to do research in a quick turnaround fashion is important. It is just as important to ensure the research that`s published and potentially implemented is reliable.

General Internal Medicine-Primary Care(US) rater

The article reports that all reviewed studies were "appraised to have high risk of bias owing to a combination of poor reporting and poor methodological conduct for participant selection, predictor description, and statistical methods used." I gave it a decent score for newsworthiness because medical providers need to know that many of the hastily written articles may be misleading.

Hospital Doctor/Hospitalists rater

Extremely timely and useful information. Too early to draw conclusions about sensitivity and specificity of prediction models.

Infectious Disease rater

Sound advice to appropriately interpret diagnosis and prognosis prediction models for COVID-19. Until an adequately reported model is published, clinical judgment prevails.

Infectious Disease rater

This is an excellent review of the limitations of current data tools and models for COVID-19. There is more enthusiasm for data analytical tools than quality at this time, and this provides a handy guide to what's out there and what researchers should be doing to improve existing tools.

Intensivist/Critical Care rater

This article is not very interesting for day-to-day practice. Probably more interesting for statisticians.

Intensivist/Critical Care rater

Good idea but data are too sparse, and what is available has significant methodologic concerns that limit clinical uptake of the results. This SR/MA demonstrates a need for more work in the field rather than having any impact on caring for COVID patients.

Intensivist/Critical Care rater

A thoughtful and rigorous systematic review limited by the poor quality of the studies included. It is almost certain, however, that it will be possible to construct predictive models to accurately identify patients presenting with symptomatic illness who have COVID-19, but the model will require laboratory studies and/or CT imaging of the chest.

Intensivist/Critical Care rater

Interesting compilation of prediction models. The variables that fall out in the models are not all that novel - they are pretty intuitive and well known. I do think the most interesting thing is that of these 31 models, only 1 used data from outside of China, which may explain why the prediction models are not very robust right now.

Internal Medicine rater

Even though, this is titled as a "quick" systematic review, it seems premature to analyze studies at this time.

Internal Medicine rater

The main relevance aspect is in being critical about the state of the knowledge on the topic, and limitations on current efforts to create diagnostic and prognostic rules.

Internal Medicine rater

This is such a rapidly moving subject that I believe every systematic review is going to be out of date by the time it is published.

Occupational and Environmental Health rater

Although modelling is important, it is no surprise that numerous models are emerging and most with challenges. This is relevant background, but it is most useful among those whose role is mathematical modelling as opposed to applied front-line decision-making.

Occupational and Environmental Health rater

Indicates problems with recently published models for diagnosis and prognosis in COVID-19. Relevant information about issues with current models, but not to those in direct clinical practice.

Pediatric Emergency Medicine rater

The studies refer to adult data and considering the different clinical picture and prognosis in children affected by C19, are not transferable to children.

Pediatric Emergency Medicine rater

Unfortunately, there was a high percentage of bias in the studies, so interpretation of their results is questionable. The review highlights that so little is known about this condition. Improved longer-term data collection is necessary to provide more accurate prediction models.

Public Health rater

Anyone with fair medical knowledge understands this. More important for non-medical persons to understand.

Public Health rater

This is a systematic review of scientific articles based on predicting the occurrence or progression of COVID-19 in a given patient. This was a good review of current literature on this hot topic and will provide any casual reader a good understanding of what`s out there to use for diagnostic or prognostic purposes. It also provides a needed caution regarding the high risk for bias in these early prediction models, as well as advice on how to conduct a study well. On the whole, I think it will be of interest to most clinicians in any front-line or hospital-based specialty, but these results will be ephemeral: by June 2020 the findings of this paper will be obsolete.
Comments from EvidenceAlerts subscribers

No subscriber has commented on this article yet.