Materials and methods

A narrative review was conducted to generate a list of pos sible predictors based on clinical signs and symptoms, co morbidities, laboratory, and radiographic findings. After initial identification of predictive variables, they were subjected to expert analysis through a 2-round Delphi process¹³. The output was a set of potential variables based on expert opinion to be added to the NEWS-2.

The narrative review was conducted in April 2020 using the Ovid MEDLINE and medRxiv database for articles written in English and published until April 2020. The search strategy included terms such as “COVID-19”, “Risk Factors”, “Respira tory Insufficiency” and “Mortality”. Studies were selected on the basis of the following inclusion criteria: population over 18 years of age where signs and symptoms were recorded together with comorbidities, laboratory and radiographic find ings and in which all these parameters were valued against the occurrence of death or disease severity in confirmed COVID-19 patients.

To select the final predictive variables a Delphi process was carried on. First experts from both resource-rich and resource-limited settings were recruited, including profes sionals from Argentina, Chile, and Canada. Participants are involved in the care of critically ill adults and active in medical research areas including critical care, infectious diseases, and pneumology on a daily basis. The first contact was by email to communicate the objective of the study and extend an invita tion to participate in both rounds of the Delphi process. The target sample size of expert contributors was 14.

The first round of the Delphi process had a 7-day elapse; it was initiated on April 15th of 2020 and completed by April 22nd of 2020. The questionnaire form was distributed among expert participants who were asked to evaluate the variables gathered from the literature review before mentioned and suggested as potential predictors of worse outcome in COVID-19 patients.

When the participant answered “yes” to the fact that a given variable was valuable as a predictor, they were prompted to evaluate 3 domains: predictive potential, measurement reliability, level of training, and/or resources required to mea sure and collect the variable13. Each domain had 4 possible answers: high (3), moderate (2), minimal (1), or not applicable (0). Finally, there was an option for the participant to make comments regarding each variable. Moreover, they were encouraged to add new variables to the suggested settings.

At the stage of analysis, each answer was given a number between 3 and 0 (high [3], moderate [2], minimal [1], not appli cable [0]) based on the strength of the response. The numbers for each domain were tabulated to calculate a weighted effect (WE) to help to determine the selection threshold. The WE was calculated by following the formula exemplified below, which doubles the weight of the value for predictive potential, adds the value for measurement reliability, and subtracts the value for the level of resources and/or training required.

Weighted Effect = Predictive Potential x2 + Reliability - Resources or Training

eg.: Weighted Effect_Asthma = Moderate (2) x2 + Moderate (2) - Minimal (1)

Weighted Effect_Asthma = 5

WE was calculated for each variable valued by each expert opinion. The sum of the WE’s for a given variable was ranked for a further selection of those with the greatest value achieved whereas WE of variables considered as medical records, were analyzed separately from clinical features.

Afterward, a threshold was chosen based on the desired number of predictors. Then variables in which WE scored above the threshold, were included in a final set of predictor variables. Those variables that were below the threshold, were carefully reviewed by the research team and discharged or included in Round 2 for re-evaluation depending on the value obtained. Any additional variables proposed by participants were evaluated in Round 2 when they were considered clini cally distinct from the variables already assessed in Round 1.

The 2nd Round was conducted in a 7-day elapse of time, from 29 April of 2020 to 7 May of 2020. It consisted of re-evaluating selected variables from Round 1 as well as evaluat ing newly suggested variables. Participants were provided the responses from Round 1, and a threshold procedure similar to the one used in the initial round was used in the second round. Each round demanded 7 days for completion given to each expert consulted who was sent a reminder one day before the deadline. The final set of variables was obtained, and results were analyzed as follows.

The questionnaire was developed using the secure web-based application Google Forms. Participant’s own responses from Round 1 were available in Round 2 along with a mean response from the other participants.

This project has been approved by the Ethics Committee for Research Protocols at Hospital Italiano de Buenos Aires. Voluntary completion of the questionnaire implied consent and the participants’ responses were received and analyzed anonymously.

Results

After analysis, ten articles fulfilled the final inclusion cri teria and were, therefore, considered. Sixty-four relevant variables predictors analyzed in these studies were sum marized to generate a master list (Supplementary mate rial 1) created using Microsoft Excel to keep track of each predictor variable and the frequency of repetition in other studies as a presumed indicator of its predictive potential and relative commonality. Each variable was organized into the following categories: 1) patient’s characteristics; 2) signs and symptoms; 3) scores; 4) laboratory findings; 5) chest x-ray findings; 6) comorbidities. All variables were presented as either binary or continuous variables, depending on how it was presented in the original study.

In the first round of the Delphi process, there was a high level of agreement on the following variables alerting to a worse outcome: age, male gender, dyspnea, d-dimer > 1 μg/ml, lymphopenia, “Sequential Organ Failure Assess ment” (SOFA) score, bilateral compromise in chest x-ray and comorbidities such as chronic heart failure, diabetes with end-organ damage and hypertension. The thresholds for age and lymphopenia were discussed in Round 2.

Additionally, there was a high level of agreement to not include: pregnancy, plasma albumin, pro-B-type natriuretic peptide, lactate dehydrogenase, and other comorbidities such as chronic obstructive pulmonary disease (COPD), asthma, chronic renal disease, solid tumor, tuberculosis, active smoking and diabetes without end-organ damage.

Since there was a moderate level of agreement re garding thrombocytopenia, C reactive protein, creatinine serum levels, chest x-ray findings other than bilateral compromise, those variables were discussed in Round 2.

Experts also proposed 2 new variables; findings in pulmonary ultrasound and reticulonodular interstitial pat tern in the chest x-ray. These 2 new variables were not considered due to interobserver variability and the need for highly trained professionals to perform and to interpret those studies.

Due to the lack of a specific cut-off value for age and lymphopenia, possible thresholds were proposed to participants. After Round 2, cut-off values for age were defined as follows: low risk for age < 60; moderate risk for ages between 60 and 65; and high risk for age > 65. Regarding lymphopenia, thresholds were: >1000 mm³, low risk; between 500 and 1000 mm³, moderate risk; and less than 500 mm³, high risk.

Variables from Round 1 that did not reach the WE threshold value for immediate consideration, were reas sessed. Low platelet count, C-reactive protein, lactate dehydrogenase, and serum creatinine were reconsidered to select the one with a higher predictive value of the worse outcome. Consensus on platelet count <100 000 mm³ was the variable with higher potential. The list of all potential variables proposed to the Delphi process is available for reference in Supplementary material 2. The final set variables selected from the Delphi process are exposed in Figure 1.

Fig. 1 Delphi process results. A: The trends of the sum of responses to each domain for all 16 variables evaluated in Round 1. B: The trends of the sum of responses to each domain for all 12 antecedents evaluated in Round 1 

Following the analysis and the application of the modi fied Delphi process, a final set of selected variables was combined with a modified NEWS-2 score to generate the COVID-19 Severity Index. The modifications to NEWS-2 Score are described as follows. In the case that the patient needs supplemental oxygen, he will receive 3 score-points instead of 2. The addition of 1 and 2 score-points related to low blood pressure was eliminated. Low temperature only added 1 score-point if it was less than 35.6 °C instead of 36,1 °C, while 1 point-score was added if the temperature was 38 °C or higher. COVID-19 Severity Index score is exposed in Figure 2. Patients were divided into four risk categories based on their score (Fig. 3).

Fig. 2 COVID-19 Severity Index 

Fig. 3 COVID-19 Severity Index risk chart 

The prediction capacity of this score was studied to evaluate its predictive potential of ICU transfer in 24 and 48-hours elapse of time. A group of 220 patients with confirmed COVID-19 was evaluated; 19 of which were unexpectedly transferred to ICU; and 17 of which were transferred to ICU during the first 3 days, one on the 5th day and another on the 6th day of hospitalization.

A comparison between COVID-19 Severity Index, NEWS score adapted by Liao et al.⁵, and NEWS-2 score was made. All three scores were measured on the first, second, and third day after the hospital admission of the patients.

For those patients who were initially admitted into general wards and were later transferred to the ICU, the score was retrospectively applied for the 72, 48 and 24 hours before the ICU admission, to identify whether they were parameters that could predict the need for more intensive monitoring.

A comparative analysis of the area under the curve (AUC) for the different scores evidenced a better capacity of the COVID-19 Severity Index to predict the need for ICU admis sion. When applied in the 24 hours before ICU admission, the AU-ROC for the score COVID-19 Severity Index was 0.94 vs. 0.88 for the modified NEWS score developed by Liao et al., and 0.80 for NEWS-2 (Figure 4). When applied in the 48 hours before ICU admission, the AU-ROC for COVID-19 Severity Index was 0.88, for the modified NEWS was 0.84, and 0.62 for NEWS-2.

Fig. 4 Area Under the Curve of COVID Severity Index: 0.93, NEWS COVID: 0.87, proposed by Liao et al. 2 and NEWS-2: 0.8; to predict unexpected ICU transfer in the next 24 hours 

Discussion

In this study, an EWS was designed to predict progression towards critical illness among COVID-19 infected patients during hospitalization.

Although NEWS-2 score is the mainly used EWS, there are few published studies on its use in the specific context of COVID-19¹⁴. A paper published during the early phase of the COVID-19 pandemic offered an EWS based on an adapted version of the NEWS-2 score in which age > 65 years was added to reflect emerging evidence of age as an independent risk factor for survival⁵.

The complexity of COVID-19 and the multiple variables involved in its course evidenced the need to search for a more specific score that could be used in this single dis ease to better discern among patients at risk of presenting severe infection^7-12. The development and use of a simple tool built on the basis of signs and symptoms with moder ate to strong predictive potential, could ideally facilitate the triage process and expedite the care for hospitalized adult patients with COVID-19.

Due to the current lack of evidence, the effort to carry out careful research with a methodologically solid process was paramount. The narrative review al lowed the research team to balance information from peer-reviewed articles as well as urgent data reported in preprints.

Additionally, the Delphi method allowed us to merge clinical expertise with theoretical reasoning. Delphi process was chosen with the objective of achieving consensus among a panel of experts on a defined issue, using an iteration of a questionnaire and aggregating the answers to provide feedback to the participants after each completed round. This method, as a way of gener ating consensus, is widely applied in diverse fields such as program planning and resource assessment, even in the healthcare sector¹⁵. One of the advantages of using this method is to facilitate the online consensus-building which significantly enabled the participation of experts in the matter from various locations worldwide in the current pandemic scenario¹³.

The 10 variables added to the modified NEWS-2 score required for calculation of the risk of developing a critical illness are usually available at Hospital admission. CO VID-19 Severity Index is a dynamic tool designed to be actualized with the clinical changes of the patient, with the aim of detecting clinical deterioration within 24 to 48 hours prior to ICU transfer.

Additionally, the digital medical record of the Hos pital Italiano de Buenos Aires was electronically set for an automatic calculation and constant update of the COVID-19 Severity Index as soon as the latest laboratory results and vital signs were recorded. This provided real-time information for deciding the most suitable area of care for each patient¹⁶. Likewise, since it is a relatively simple score, it can also be calculated manually.

Even though COVID-19 Severity Index has a short scale validation, it was designed by experts’ opinions. Opinions may be impacted by experts’ training, exposure, and expertise. The application of the present score is being carried out in patients at the Hospital Italiano de Buenos Aires to define the intensity of nursing monitor ing required.

The COVID-19 Severity Index was Specifically de signed for the current COVID-19 pandemic. This score may serve as a reliable tool for strategic planning, organization and administration of resources by easily distinguishing hospitalized patients with a higher risk of ICU transfer.

Final disclosures: In a scientific letter previously presented to Medicina Intensiva ¹⁷ a simplified summary of this work was presented, with a stating of the variables included in the score, without mention of the elements that led to its construction. Here we present a detailed description and precise information of the concrete methodology used for the design of the score, includ ing a narrative review (summarized in the master table in Supplementary Material 1) where 44 variables were analyzed. We also describe how the modified Delphi process was applied and how the weighted effect of each of these variables was calculated based on the experts’ opinions (reflected in Fig. 1). In addition, we include here the graphical comparison of the AUROC of the designed model in order to compare it to other early warning sys tems. These factors were not analyzed in the scientific letter¹⁷. The research team considers that the methodol ogy applied in the design of this early warning system is as relevant as the variables included in it. The design and process through which the Early Warning System was built is novel and allowed us, in just 3 months, to identify potentially critical patients. It should also be mentioned that our study was initially released in a medRxiv¹⁸. Its dissemination was a way of contributing to the scientific community by sharing this tool that proved so useful to us in our institution.