SciELO - Scientific Electronic Library Online

vol.81 número4Conocimientos, actitudes y prácticas sobre covid-19 en argentina. estudio tranversalCaracterísticas clínicas de los pacientes críticos con COVID-19 índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados




  • No hay articulos citadosCitado por SciELO

Links relacionados

  • No hay articulos similaresSimilares en SciELO


Medicina (Buenos Aires)

versión impresa ISSN 0025-7680versión On-line ISSN 1669-9106

Medicina (B. Aires) vol.81 no.4 Ciudad Autónoma de Buenos Aires ago. 2021



Design and development of an early warning score for COVID-19 hospitalized patients

Diseño y desarrollo de un sistema de alerta temprana para pacientes hospitalizados por COVID-19

Iván Huespe1  2  * 

Indalecio Carboni Bisso1 

Nicolás A. Gemelli1 

Sergio A. Terrasa1 

Sabrina Di Stefano1 

Valeria Burgos2 

Jorge Sinner1 

Mailen Oubiña1 

Marina Bezzati1 

Pablo Delgado1 

Marcos Las Heras1 

Marcelo R. Risk2 

1 Hospital Italiano de Buenos Aires

2 Instituto de Medicina Traslacional e Ingeniería Biomédica, HIBA, IUHI, CONICET, Buenos Aires, Argentina


Pandemics pose a major challenge for public health preparedness, requiring a coordinated international response and the development of solid containment plans. Early and accurate identifica tion of high-risk patients in the course of the current COVID-19 pandemic is vital for planning and making proper use of available resources. The purpose of this study was to identify the key variables that account for worse outcomes to create a predictive model that could be used effectively for triage. Through literature review, 44 variables that could be linked to an unfavorable course of COVID-19 disease were obtained, including clinical, laboratory, and X-ray variables. These were used for a 2-round modified Delphi processing with 14 experts to select a final list of variables with the greatest predictive power for the construction of a scoring system, leading to the creation of a new scoring system: the COVID-19 Severity Index. The analysis of the area under the curve for the COVID-19 Severity Index was 0.94 to predict the need for ICU admission in the following 24 hours against 0.80 for NEWS-2. Additionally, the digital medical record of the Hospital Italiano de Buenos Aires was electronically set for an automatic calculation and constant update of the COVID-19 Severity Index. Specifically designed for the current COVID-19 pandemic, COVID-19 Severity Index could be used as a reliable tool for strategic planning, organization, and administration of resources by easily identifying hospitalized patients with a greater need of intensive care.

Key words: Coronavirus; Critical care; Hospital administration; Early warning score


La pandemia por COVID-19 planteó un desafío para el sistema salud, debido a la gran demanda de pacientes hospitalizados. La identificación temprana de pacientes hospitalizados con riesgo de evo lución desfavorable es vital para asistir en forma oportuna y planificar la demanda de recursos. El propósito de este estudio fue identificar las variables predictivas de mala evolución en pacientes hospitalizados por COVID-19 y crear un modelo predictivo que pueda usarse como herramienta de triage. A través de una revisión narrativa, se obtuvieron 44 variables vinculadas a una evolución desfavorable de la enfermedad COVID-19, incluyendo variables clínicas, de laboratorio y radiográficas. Luego se utilizó un procesamiento por método Delphi modificado de 2 rondas para seleccionar una lista final de variables incluidas en el score llamado COVID-19 Severity Index. Luego se calculó el Área Bajo la Curva (AUC) del score para predecir el pase a terapia intensiva en las próximas 24 horas. El score presentó un AUC de 0,94 frente a 0,80 para NEWS-2. Finalmente se agregó el COVID-19 Severity Index a la historia clínica electrónica de un hospital universitario de alta complejidad. Se programó para que el mismo se actualice de manera automática, facilitando la planificación estratégica, organización y administración de recursos a través de la identificación temprana de pacientes hospitalizados con mayor riesgo de transferencia a la Unidad de Cuidados Intensivos.

Palabras clave: Coronavirus; Cuidados intensivos; Administración hospitalaria; Puntaje de alerta temprana


Current knowledge

• Early Warning Scores that detect early and accurately high-risk patients in the course of the current COVID-19 pandemic are vital for planning and making proper use of available resources.

Article´s contribution to knowledge

• COVID-19 Severity Index is an Early Warning Score that was electronically set for an automatic calculation and constant update in COVID-19 patients. This score was used as a tool for strategic planning, organization, and administration of resources during the actual

Infectious disease outbreaks constitute a serious problem to global health with a major impact on countries’ economies, healthcare systems and resources1. The spread of Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) known as COVID-19, has already gone into pandemic proportions registering, at the moment of this study, a total of 80 million confirmed cases, 1.7 mil lion deaths and 57 million recovered patients across 215 countries in a short elapse of time2. The way in which outbreaks affect countries depends on multiple factors and its impact is difficult to foresee. However, the numbers of infected people and casualties are evidence that despite the attempts to plan in advance, the global healthcare systems remain unprepared3.

The intensity of staffing needed and the sophisticated training required for the care of patients with viral infections during pandemics result in the fact that a relatively small number of patients could easily overwhelm healthcare systems4.

Accurate identification of variables related to worse outcomes is key for triaging and adapting the intensity of care that each patient requires, allowing effective strategic planning and a better administration of human and material resources. Moreover, the need for a sensitive and predic tive model is mandatory to avoid a delayed recognition of severely ill patients or even those at risk of presenting further complications.

During the early phase of COVID-19 pandemic, Liao et al. propose an early warning score based on an adapted version of the National Early Warning Score 2 (NEWS-2) adding age as a variable to reflect emerging evidence of age as an independent risk factor for survival5. In that score, patients were divided into four categories based on the risk of respiratory failure: low, medium, high, and extreme. The score was used to manage the monitoring frequency and to activate a rapid response team.

The NEWS2 is a Early Warning Scores (EWS) that predict deterioration in hospitalized patients6, but are de signed for general hospitalized patients in a non-pandemic scenario. COVID-19 pandemic has now a high proportion of hospitalized patients with a single disease. Therefore, a specific EWS for COVID-19 including laboratory test results, clinical features, and radiological findings7-12, could improve the detection of high-risk patients with the aim of optimizing the management of hospital resources. This tool is mainly relevant in low-income countries where re sources are insufficient, even before the actual pandemic.

Aware of the impact of the current pandemic, CO VID-19 Severity Index was developed as a triage tool based on the NEWS-2 score, that could rapidly and reli ably be used by frontline healthcare personnel to identify high-risk patients.

Materials and methods

A narrative review was conducted to generate a list of pos sible predictors based on clinical signs and symptoms, co morbidities, laboratory, and radiographic findings. After initial identification of predictive variables, they were subjected to expert analysis through a 2-round Delphi process13. The output was a set of potential variables based on expert opinion to be added to the NEWS-2.

The narrative review was conducted in April 2020 using the Ovid MEDLINE and medRxiv database for articles written in English and published until April 2020. The search strategy included terms such as “COVID-19”, “Risk Factors”, “Respira tory Insufficiency” and “Mortality”. Studies were selected on the basis of the following inclusion criteria: population over 18 years of age where signs and symptoms were recorded together with comorbidities, laboratory and radiographic find ings and in which all these parameters were valued against the occurrence of death or disease severity in confirmed COVID-19 patients.

To select the final predictive variables a Delphi process was carried on. First experts from both resource-rich and resource-limited settings were recruited, including profes sionals from Argentina, Chile, and Canada. Participants are involved in the care of critically ill adults and active in medical research areas including critical care, infectious diseases, and pneumology on a daily basis. The first contact was by email to communicate the objective of the study and extend an invita tion to participate in both rounds of the Delphi process. The target sample size of expert contributors was 14.

The first round of the Delphi process had a 7-day elapse; it was initiated on April 15th of 2020 and completed by April 22nd of 2020. The questionnaire form was distributed among expert participants who were asked to evaluate the variables gathered from the literature review before mentioned and suggested as potential predictors of worse outcome in COVID-19 patients.

When the participant answered “yes” to the fact that a given variable was valuable as a predictor, they were prompted to evaluate 3 domains: predictive potential, measurement reliability, level of training, and/or resources required to mea sure and collect the variable13. Each domain had 4 possible answers: high (3), moderate (2), minimal (1), or not applicable (0). Finally, there was an option for the participant to make comments regarding each variable. Moreover, they were encouraged to add new variables to the suggested settings.

At the stage of analysis, each answer was given a number between 3 and 0 (high [3], moderate [2], minimal [1], not appli cable [0]) based on the strength of the response. The numbers for each domain were tabulated to calculate a weighted effect (WE) to help to determine the selection threshold. The WE was calculated by following the formula exemplified below, which doubles the weight of the value for predictive potential, adds the value for measurement reliability, and subtracts the value for the level of resources and/or training required.

Weighted Effect = Predictive Potential x2 + Reliability - Resources or Training

eg.: Weighted EffectAsthma = Moderate (2) x2 + Moderate (2) - Minimal (1)

Weighted EffectAsthma = 5

WE was calculated for each variable valued by each expert opinion. The sum of the WE’s for a given variable was ranked for a further selection of those with the greatest value achieved whereas WE of variables considered as medical records, were analyzed separately from clinical features.

Afterward, a threshold was chosen based on the desired number of predictors. Then variables in which WE scored above the threshold, were included in a final set of predictor variables. Those variables that were below the threshold, were carefully reviewed by the research team and discharged or included in Round 2 for re-evaluation depending on the value obtained. Any additional variables proposed by participants were evaluated in Round 2 when they were considered clini cally distinct from the variables already assessed in Round 1.

The 2nd Round was conducted in a 7-day elapse of time, from 29 April of 2020 to 7 May of 2020. It consisted of re-evaluating selected variables from Round 1 as well as evaluat ing newly suggested variables. Participants were provided the responses from Round 1, and a threshold procedure similar to the one used in the initial round was used in the second round. Each round demanded 7 days for completion given to each expert consulted who was sent a reminder one day before the deadline. The final set of variables was obtained, and results were analyzed as follows.

The questionnaire was developed using the secure web-based application Google Forms. Participant’s own responses from Round 1 were available in Round 2 along with a mean response from the other participants.

This project has been approved by the Ethics Committee for Research Protocols at Hospital Italiano de Buenos Aires. Voluntary completion of the questionnaire implied consent and the participants’ responses were received and analyzed anonymously.


After analysis, ten articles fulfilled the final inclusion cri teria and were, therefore, considered. Sixty-four relevant variables predictors analyzed in these studies were sum marized to generate a master list (Supplementary mate rial 1) created using Microsoft Excel to keep track of each predictor variable and the frequency of repetition in other studies as a presumed indicator of its predictive potential and relative commonality. Each variable was organized into the following categories: 1) patient’s characteristics; 2) signs and symptoms; 3) scores; 4) laboratory findings; 5) chest x-ray findings; 6) comorbidities. All variables were presented as either binary or continuous variables, depending on how it was presented in the original study.

In the first round of the Delphi process, there was a high level of agreement on the following variables alerting to a worse outcome: age, male gender, dyspnea, d-dimer > 1 μg/ml, lymphopenia, “Sequential Organ Failure Assess ment” (SOFA) score, bilateral compromise in chest x-ray and comorbidities such as chronic heart failure, diabetes with end-organ damage and hypertension. The thresholds for age and lymphopenia were discussed in Round 2.

Additionally, there was a high level of agreement to not include: pregnancy, plasma albumin, pro-B-type natriuretic peptide, lactate dehydrogenase, and other comorbidities such as chronic obstructive pulmonary disease (COPD), asthma, chronic renal disease, solid tumor, tuberculosis, active smoking and diabetes without end-organ damage.

Since there was a moderate level of agreement re garding thrombocytopenia, C reactive protein, creatinine serum levels, chest x-ray findings other than bilateral compromise, those variables were discussed in Round 2.

Experts also proposed 2 new variables; findings in pulmonary ultrasound and reticulonodular interstitial pat tern in the chest x-ray. These 2 new variables were not considered due to interobserver variability and the need for highly trained professionals to perform and to interpret those studies.

Due to the lack of a specific cut-off value for age and lymphopenia, possible thresholds were proposed to participants. After Round 2, cut-off values for age were defined as follows: low risk for age < 60; moderate risk for ages between 60 and 65; and high risk for age > 65. Regarding lymphopenia, thresholds were: >1000 mm3, low risk; between 500 and 1000 mm3, moderate risk; and less than 500 mm3, high risk.

Variables from Round 1 that did not reach the WE threshold value for immediate consideration, were reas sessed. Low platelet count, C-reactive protein, lactate dehydrogenase, and serum creatinine were reconsidered to select the one with a higher predictive value of the worse outcome. Consensus on platelet count <100 000 mm3 was the variable with higher potential. The list of all potential variables proposed to the Delphi process is available for reference in Supplementary material 2. The final set variables selected from the Delphi process are exposed in Figure 1.

Fig. 1 Delphi process results. A: The trends of the sum of responses to each domain for all 16 variables evaluated in Round 1. B: The trends of the sum of responses to each domain for all 12 antecedents evaluated in Round 1 

Following the analysis and the application of the modi fied Delphi process, a final set of selected variables was combined with a modified NEWS-2 score to generate the COVID-19 Severity Index. The modifications to NEWS-2 Score are described as follows. In the case that the patient needs supplemental oxygen, he will receive 3 score-points instead of 2. The addition of 1 and 2 score-points related to low blood pressure was eliminated. Low temperature only added 1 score-point if it was less than 35.6 °C instead of 36,1 °C, while 1 point-score was added if the temperature was 38 °C or higher. COVID-19 Severity Index score is exposed in Figure 2. Patients were divided into four risk categories based on their score (Fig. 3).

Fig. 2 COVID-19 Severity Index 

Fig. 3 COVID-19 Severity Index risk chart 

The prediction capacity of this score was studied to evaluate its predictive potential of ICU transfer in 24 and 48-hours elapse of time. A group of 220 patients with confirmed COVID-19 was evaluated; 19 of which were unexpectedly transferred to ICU; and 17 of which were transferred to ICU during the first 3 days, one on the 5th day and another on the 6th day of hospitalization.

A comparison between COVID-19 Severity Index, NEWS score adapted by Liao et al.5, and NEWS-2 score was made. All three scores were measured on the first, second, and third day after the hospital admission of the patients.

For those patients who were initially admitted into general wards and were later transferred to the ICU, the score was retrospectively applied for the 72, 48 and 24 hours before the ICU admission, to identify whether they were parameters that could predict the need for more intensive monitoring.

A comparative analysis of the area under the curve (AUC) for the different scores evidenced a better capacity of the COVID-19 Severity Index to predict the need for ICU admis sion. When applied in the 24 hours before ICU admission, the AU-ROC for the score COVID-19 Severity Index was 0.94 vs. 0.88 for the modified NEWS score developed by Liao et al., and 0.80 for NEWS-2 (Figure 4). When applied in the 48 hours before ICU admission, the AU-ROC for COVID-19 Severity Index was 0.88, for the modified NEWS was 0.84, and 0.62 for NEWS-2.

Fig. 4 Area Under the Curve of COVID Severity Index: 0.93, NEWS COVID: 0.87, proposed by Liao et al. 2 and NEWS-2: 0.8; to predict unexpected ICU transfer in the next 24 hours 


In this study, an EWS was designed to predict progression towards critical illness among COVID-19 infected patients during hospitalization.

Although NEWS-2 score is the mainly used EWS, there are few published studies on its use in the specific context of COVID-1914. A paper published during the early phase of the COVID-19 pandemic offered an EWS based on an adapted version of the NEWS-2 score in which age > 65 years was added to reflect emerging evidence of age as an independent risk factor for survival5.

The complexity of COVID-19 and the multiple variables involved in its course evidenced the need to search for a more specific score that could be used in this single dis ease to better discern among patients at risk of presenting severe infection7-12. The development and use of a simple tool built on the basis of signs and symptoms with moder ate to strong predictive potential, could ideally facilitate the triage process and expedite the care for hospitalized adult patients with COVID-19.

Due to the current lack of evidence, the effort to carry out careful research with a methodologically solid process was paramount. The narrative review al lowed the research team to balance information from peer-reviewed articles as well as urgent data reported in preprints.

Additionally, the Delphi method allowed us to merge clinical expertise with theoretical reasoning. Delphi process was chosen with the objective of achieving consensus among a panel of experts on a defined issue, using an iteration of a questionnaire and aggregating the answers to provide feedback to the participants after each completed round. This method, as a way of gener ating consensus, is widely applied in diverse fields such as program planning and resource assessment, even in the healthcare sector15. One of the advantages of using this method is to facilitate the online consensus-building which significantly enabled the participation of experts in the matter from various locations worldwide in the current pandemic scenario13.

The 10 variables added to the modified NEWS-2 score required for calculation of the risk of developing a critical illness are usually available at Hospital admission. CO VID-19 Severity Index is a dynamic tool designed to be actualized with the clinical changes of the patient, with the aim of detecting clinical deterioration within 24 to 48 hours prior to ICU transfer.

Additionally, the digital medical record of the Hos pital Italiano de Buenos Aires was electronically set for an automatic calculation and constant update of the COVID-19 Severity Index as soon as the latest laboratory results and vital signs were recorded. This provided real-time information for deciding the most suitable area of care for each patient16. Likewise, since it is a relatively simple score, it can also be calculated manually.

Even though COVID-19 Severity Index has a short scale validation, it was designed by experts’ opinions. Opinions may be impacted by experts’ training, exposure, and expertise. The application of the present score is being carried out in patients at the Hospital Italiano de Buenos Aires to define the intensity of nursing monitor ing required.

The COVID-19 Severity Index was Specifically de signed for the current COVID-19 pandemic. This score may serve as a reliable tool for strategic planning, organization and administration of resources by easily distinguishing hospitalized patients with a higher risk of ICU transfer.

Final disclosures: In a scientific letter previously presented to Medicina Intensiva 17 a simplified summary of this work was presented, with a stating of the variables included in the score, without mention of the elements that led to its construction. Here we present a detailed description and precise information of the concrete methodology used for the design of the score, includ ing a narrative review (summarized in the master table in Supplementary Material 1) where 44 variables were analyzed. We also describe how the modified Delphi process was applied and how the weighted effect of each of these variables was calculated based on the experts’ opinions (reflected in Fig. 1). In addition, we include here the graphical comparison of the AUROC of the designed model in order to compare it to other early warning sys tems. These factors were not analyzed in the scientific letter17. The research team considers that the methodol ogy applied in the design of this early warning system is as relevant as the variables included in it. The design and process through which the Early Warning System was built is novel and allowed us, in just 3 months, to identify potentially critical patients. It should also be mentioned that our study was initially released in a medRxiv18. Its dissemination was a way of contributing to the scientific community by sharing this tool that proved so useful to us in our institution.


1. Brown RB. Public health lessons learned from biases in coronavirus mortality overestimation. Disaster Med Public Health Prep 2020; 14: 364-71. [ Links ]

2. Coronavirus disease (COVID-19) - World Health Orga nization. In: In: ; accessed October 2020. [ Links ]

3. Maves RC, Jamros CM, Smith AG. Intensive care unit preparedness during pandemics and other biological threats. Crit Care Clin 2019; 35: 609-18. [ Links ]

4. Blumenthal D, Fowler EJ, Abrams M, et al. Covid-19 - Implications for the Health Care System. N Engl J Med 2020; 383: 1483-88. [ Links ]

5. Liao X, Wang B, Kang Y. Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China. Intensive Care Medicine 2020; 46: 357-60. [ Links ]

6. Smith MEB, Chiovaro JC, O’Neil M, et al. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc 2014; 11: 1454-65. [ Links ]

7. Jain V, Yuan J-M. Predictive symptoms and comorbidities for severe COVID-19 and intensive care unit admission: a systematic review and meta-analysis. Int J Public Health 2020; 65: 533-46. [ Links ]

8. Schalekamp S, Huisman M, van Dijk RA, et al. Model-based Prediction of Critical Illness in Hospitalized Pa tients with COVID-19. Radiology 2020; 202723. [ Links ]

9. Guan W-J, Ni Z-Y, Hu Y, et al. Clinical Characteristics of Coronavirus Disease 2019 in China. N Engl J Med 2020; 382: 1708-20. [ Links ]

10. Huang C, Wang Y, Li X, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet 2020; 395: 497-06. [ Links ]

11. Li K, Wu J, Wu F, et al. The Clinical and Chest CT Fea tures Associated With Severe and Critical COVID-19 Pneumonia. Invest Radiol 2020; 55: 327-31. [ Links ]

12. Wang D, Hu B, Hu C, et al. Clinical Characteristics of 138 Hospitalized Patients With 2019 Novel Coronavirus-Infected Pneumonia in Wuhan, China. JAMA 2020; 323: 1061-69. [ Links ]

13. Fung JST, Akech S, Kissoon N, et al. Determining predic tors of sepsis at triage among children under 5 years of age in resource-limited settings: A modified Delphi process. PLoS One 2019; 14: e0211274. [ Links ]

14. NEWS (or NEWS2) score when assessing possible COV ID-19 patients in primary care? - CEBM. In: In: score-when-assessing-patients-with-possible-covid-19-in-primary-care : accessed June 2020. [ Links ]

15. Hsu C-C, Sandford BA. The Delphi Technique. Online Research Methods in Urban and Planning Studies 2012; 173-92. [ Links ]

16. Finlay GD, Duncan Finlay G, Rothman MJ, et al. Measuring the modified early warning score and the Rothman Index: Advantages of utilizing the electronic medical record in an early warning system. Journal of Hospital Medicine 2014; 9: 116-19. [ Links ]

17. Huespe I, Carboni Bisso I, Di Stefano S, et al. COVID-19 Severity Index: A predictive score for hospitalized pa tients. Med Intensiva. Epub ahead of print 29 December 2020. DOI: 10.1016/j.medin.2020.12.001. [ Links ]

18. Huespe I, Bisso IC, Gemelli NA, et al. COVID-19 Severity Index: A predictive score for hospitalized patients. me dRxiv 2020; 2020.08.12.20166579. [ Links ]

Suplementary material 1

Supplementary material 2

List of all potential variables proposed to the Delphi process

Received: December 29, 2020; Accepted: April 08, 2021

*Postal address: Iván A. Huespe, Hospital Italiano de Buenos Aires, Pres. Tte. Gral. Juan Domingo Perón 4190, 1199 Buenos Aires, Argentina e-mail:

Creative Commons License This is an open-access article distributed under the terms of the Creative Commons Attribution License