SciELO - Scientific Electronic Library Online

 
vol.77 número4Relación entre nivel de instrucción y exceso de peso en pacientes de consulta externa: Estudio transversalEtiología y resistencia a antimicrobianos de la infección no complicada del tracto urinario índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

  • No hay articulos citadosCitado por SciELO

Links relacionados

  • No hay articulos similaresSimilares en SciELO

Compartir


Medicina (Buenos Aires)

versión impresa ISSN 0025-7680versión On-line ISSN 1669-9106

Medicina (B. Aires) vol.77 no.4 Ciudad Autónoma de Buenos Aires ago. 2017

 

ORIGINAL ARTICLE

Validation of age, creatinine and ejection fraction (ACEF) and Cockcroft-Gault ACEF scores in elective cardiac surgery

 

Raúl A. Borracci, Miguel Rubio, Julio Baldi (Jr), Carlos A. Ingino, José L. Barisani

Departamento de Cirugía Cardíaca, Hospital de Clínicas, Facultad de Medicina, Universidad de Buenos Aires, Buenos Aires, Argentina

Postal address: Dr. Raúl A. Borracci, La Pampa 3030, 1428 Buenos Aires, Argentina
e-mail: raborracci@gmail.com

Received: 21-XI-2016
Accepted: 22-III-2017


Abstract

The objective of this study was to evaluate the efficacy of age, creatinine and ejection fraction (ACEF) score and the modified ACEFCG model, incorporating creatinine clearance, to predict immediate operative mortality risk of patients undergoing elective cardiac surgery. A retrospective analysis was performed of prospectively collected data between 2012 and 2015, from a series of 1190 adult patients who underwent elective cardiac surgery. Operative risk mortality was assessed with ACEF, ACEFCG and EuroSCORE II. Overall mortality rate was 4.0% (48 cases), while mean mortality rates predicted by ACEF, ACEFCG, and EuroSCORE II were 2.3% (p = 0.014), 6.4% (p = 0.010) and 2.5% (p = 0.038), respectively. Overall observed/predicted mortality ratio was 1.8 for ACEF score, 0.6 for ACEFCG score and 1.6 for EuroSCORE II. The ACEF score demonstrated an adequate overall performance for the low- and intermediate-risk groups, but underestimated mortality for the high risk group. The ACEFCG score discriminatory power systematically improved the area under the ROC curve (AUC) obtained with the ACEF score; however, EuroSCORE II showed the best AUC. Overall accuracy was 56.1% for the ACEF score, 51.2% for the ACEFCG score and 75.9% for EuroSCORE II. For clinical use, the ACEF score seems to be adequate to predict mortality in low- and intermediate-risk patients. Though the ACEFCG score had a better discriminatory power and calibration, it tended to overestimate the expected risk. Since ideally, a simpler risk stratification score should be desirable for bedside clinical use, the ACEF model reasonably met the expected performance in our population.

Key words: Risk stratification; Operative risk; Elective cardiac surgery.

Resumen

Validación de los puntajes de riesgo por edad, creatinina y fracción de eyección (ACEF) y Cockcroft-Gault ACEF en cirugía cardíaca electiva.

El objetivo fue evaluar la eficacia de la escala de riesgo de edad, creatinina y fracción de eyección (ACEF) y también ACEFCG, que incorpora la depuración de creatinina, para predecir el riesgo de mortalidad operatoria inmediata tras una cirugía cardiaca electiva. Se realizó un análisis retrospectivo de datos recolectados prospectivamente entre 2012 y 2015, de 1190 adultos sometidos a cirugía cardíaca electiva. El riesgo de mortalidad operatoria se evaluó con ACEF, ACEFCG y EuroSCORE II. La tasa de mortalidad global fue 4.0% (48 casos), mientras que las tasas de mortalidad predichas por ACEF, ACEFCG y EuroSCORE II fueron 2.3% (p = 0.014), 6.4% (p = 0.010) y 2.5% (p = 0.038), respectivamente. La razón mortalidad observada/esperada fue 1.8 para el ACEF, 0.6 para el ACEFCG y 1.6 para el EuroSCORE II. La puntuación de ACEF demostró un desempeño adecuado para los grupos de riesgo bajo y medio, pero subestimó la mortalidad del grupo de alto riesgo. La discriminación del ACEFCG mejoró sistemáticamente el área ROC del ACEF; sin embargo, el EuroSCORE II mostró la mejor área ROC. La precisión global fue 56.1% para el ACEF, 51.2% para el ACEFCG y 75.9% para el EuroSCORE II. Para uso clínico, el modelo ACEF parece ser adecuado para predecir la mortalidad en pacientes de riesgo bajo y medio. Aunque el puntaje de ACEFCG tuvo un mejor poder discriminatorio y calibración, tendió a sobrestimar el riesgo esperado. Considerando que sería ideal contar con un método de estratificación de riesgo más simple para uso clínico al lado de la cama, el modelo ACEF tuvo un desempeño razonable en nuestra población.

Palabras clave: Estratificación del riesgo; Riesgo operatorio; Cirugía cardíaca electiva.


 

The mortality risk stratification model including age, creatinine, and left ventricular ejection fraction (ACEF) was first described a few years ago to simplify mortality risk prediction in elective cardiac surgery1. Different investigations assessing the ACEF score performance demonstrated a discriminatory power ranging between 0.63 and 0.81 of the area under the Receiver Operating Characteristic (ROC) curve, according to center and type of surgery2-5. Further research showed that the ACEF model had lower performance than the EuroSCORE II in non-elective surgery6. Improvements in ACEF score accuracy were also obtained for specific risk groups, such as surgical ventricular reconstruction candidates7, and a modified ACEF score incorporating creatinine clearance has been recently assessed in patients undergoing percutaneous coronary interventions, but not in cardiac surgery8. Furthermore, most reports came from the same population in which the risk score was developed and validated, while validation in a different external population was rarely performed4, 9-10. The aim of the present study was to evaluate the efficacy of the original ACEF score and the modified ACEF model incorporating creatinine clearance to predict immediate operative mortality risk of patients undergoing elective cardiac surgery in a local population of Buenos Aires.

Materials and methods

A retrospective analysis was performed of prospectively collected data (ambispective design) over a 4-year period (2012-2015) from a consecutive series of 1190 adult patients who underwent elective cardiac surgery at the Buenos Aires University Hospital and its associated clinics. This series represented the social and demographic structure of Buenos Aires urban and suburban districts where patients came from.
All types of on-pump and off-pump elective cardiac surgery were included, except procedures associated to active valve endocarditis, post-infarction ventricular septal defect or free wall rupture, ischemic mitral regurgitation associated to acute myocardial infarction, urgent coronary surgery secondary to percutaneous coronary angioplasty complication, acute aortic dissection, transplantation and trans-catheter aortic valve implantation. Baseline and operative data were prospectively collected in a clinical registry, which included the variables needed to estimate the operative risk mortality based on ACEF and EuroSCORE II models11, with the latter used for comparison with ACEF model performance. Additional information on other risk factors, comorbidities, major operative complications and in-hospital operative mortality was added in the computerized database. Baseline preoperative clinical variables and in-hospital outcome data were recorded and analyzed with the approval of the local Institutional Review Board which waived the need for a written informed.
The ACEF score was calculated as originally described by Ranucci et al.1, using the formula age/left ventricular ejection fraction +1 point for serum creatinine > 2 mg/dl. The modified ACEF score (ACEFCG) was calculated with the method suggested by Garg et al.2, using the formula age/ejection fraction +1 point for every 10 ml/min reduction in creatinine clearance estimated with the Cockcroft-Gault (CG)13 equation below 60 ml/min per 1.73 m2 (up to a maximum of 6 points). Therefore, a creatinine clearance between 50 to 59 ml/min per 1.73 m2, 40 to 49 ml/min per 1.73 m2 and 30 to 39 ml/min per 1.73 m2 would receive 1, 2, and 3 additional points, respectively.
Operative mortality was defined as in-hospital mortality or mortality at 30 days after surgery for patients discharged from the hospital, including deaths occurring in rehabilitation centers, in secondary hospitals, or at home. Follow-up data after discharge were retrieved by personal or telephone contact.
Continuous variables were expressed as mean and standard deviation(SD) or as median and 25-75% percentiles (P25-75%). Kolmogorov-Smirnov (K–S) goodness-of-fit test was used to analyze normal distributions. For analysis purposes, the patient population was divided into three groups based on each score tertiles. The global accuracy of scores was evaluated in terms of calibration and discrimination. Model calibration was assessed by the Hosmer-Lemeshow goodness-of-fit test (HLχ²) and the Spiegelhalter Z-test14-15. ROC curve analysis was used to estimate the performance of the ACEF and Cockcroft-Gault ACEF scores, and the EuroSCORE II to predict 30-day mortality risk. The area under the ROC curve (AUC) with its 95% confidence interval (CI) was calculated. The same analysis was repeated in the overall population and in 3 groups: isolated coronary artery bypass grafting (CABG), non-CABG surgeries, and isolated or combined aortic valve replacement (AVR). The discrimination versus calibration relationship for each score was plotted into an accuracy graph including the AUC and the HLχ² or Spigelhalter Z-tests. The cut-off values were identified at the point where the ROC curve sum of sensitivity and specificity was highest according to the Youden index. Sensitivity, specificity, positive (PPV) and negative predictive values (NPV), and overall accuracy for each risk score cut-off point were calculated. The difference between predicted and observed mortality rates were explored for different scores, using the EuroSCORE II for the risk definition in low (< 2%), intermediate (2-5%), and high (> 5%) risk groups. Operative mortality risk predicted by the ACEF model was estimated from the graphical relationship between the ACEF score and previously published expected mortality1. Calibration was assessed by using the ACEF predicted mortality risk, and not by the calculated ACEF score, since results may vary significantly. Statistical comparison of observed vs. predicted mortality was done with the Z-test for standard error of the difference between means. Observed versus predicted mortality rates were compared with the χ² test. Statistical analysis was done with SPSS Statistics for Windows, Version 17.0. Chicago, SPSS, Inc. A two-tailed p value ≤ 0.05 was considered statistically significant.

Results

The population characteristics and the immediate postoperative adverse events are shown in Table 1. Overall mortality rate was 4.0% (48 observed deaths), while mean (± SD) mortality rates predicted by ACEF, ACEFCG, and EuroSCORE II models were 2.3% (± 1.8%) (p = 0.014), 6.4% (± 12.3%) (p = 0.010) and 2.5% (± 4.1%) (p = 0.038), respectively. Overall observed/predicted mortality ratio was 1.8 for the ACEF score, 0.6 for the ACEFCG score and 1.6 for EuroSCORE II. Predicted mortality based on the 3 models showed a positively skewed distribution; therefore, values were also presented as median and percentiles. In these cases, the median rates of mortality predicted by ACEF, ACEFCG, and EuroSCORE II models were 1.8% (p = 0.002), 1.9% (p = 0.003) and 1.5% (p = 0.0002), respectively. The patient population was divided into low (n = 760), intermediate (n = 326), and high (n = 104) risk groups according to the EuroSCORE II. Observed versus predicted all-cause of mortality, and observed/predicted ratio corresponding to the different risk groups of ACEF, ACEFCG, and EuroSCORE II models at 30 days are shown in Table 2. All 3 models demonstrated an adequate overall performance for the three risk groups; though the EuroSCORE significantly underestimated mortality for the low risk group, and the ACEF score underestimated mortality for the high risk group. For the intermediate risk group, only the ACEFCG score significantly overestimated mortality risk.

Table 1. Baseline characteristics of the population and operative outcomes (n = 1190)

Table 2. Performance of the ACEF and ACEFCG scores and the EuroSCORE II, according to risk groups

Values of the AUC with 95% CI for each risk score and type of surgery are shown in Table 3. The ACEFCG score discriminatory power systematically improved AUC values obtained with the original ACEF score for all types of surgery; however, the EuroSCORE II showed the best AUC values. Table 4 reports the sensitivity, specificity, PPV and NPV for the cutoff values identified for each score in the overall population. The best results were observed for NPV in the 3 models. Overall accuracy was 56.1% for the ACEF score, 51.2% for the ACEFCG score and 75.9% for the EuroSCORE II.

Table 3. Area under the ROC curve for the three scores

Table 4. Cutoff points and relative sensitivity, specificity, and positive and negative predictive values of the three scores in the overall population

A graphical comparison of the 3 models for each type of surgery in terms of discrimination and calibration is shown in Fig. 1. The best balances for 30-day all-cause death were provided in the following order by the EuroSCORE II and ACEFCG models in isolated or combined AVR, and in non-CABG surgeries, respectively, when assessing calibration by the HLχ² statistic. On the contrary, the ACEFCG score had the best calibration accuracy in AVR and in CABG according to the Spiegelhalter Z-test.


Fig. 1
. Predictive accuracy of the ACEF, ACEFCG and EuroSCORE II models for each group of surgeries. The position of the score in the graphs depends on the balance between the score calibration assessed by the χ² value of the Hosmer-Lemeshow statistic (HLχ²) or the Zα=0.025 value of the Spiegelhalter test, and the discriminatory power assessed by the area under the ROC curve (AUC). The accuracy of each score increases from top to bottom and from left to right. Circle size is proportional to the sample size of each group. The dotted line indicates a p-value threshold of 0.05 for 8 degrees-of-freedom in the χ² distribution, or the Zα=0.025 value, respectively for each graph.

Discussion

Ideally, an easier operative mortality predictive model should be advantageous in clinical surgical practice. In this sense, the joint guidelines of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS) have recently included the ACEF score to stratify risk in cardiac surgery16. In the present study, the preoperative risk profile was assessed using the ACEF and ACEFCG scores, and the EuroSCORE II previously validated in Argentina17. The ACEF and ACEFCG risk models based on only 3 predictors had worse accuracy compared with the EuroSCORE II, when applied to all types of elective cardiac surgery in our population. Nevertheless, the ACEFCG score systematically improved accuracy of the original ACEF score. Regarding calibration assessed by HLχ², ACEFCG and EuroSCORE II models had a similar good performance, though this was not confirmed by the Spiegelhalter test. The ACEFCG score was previously used to predict cardiac surgery outcome only in patients with left main coronary artery disease10. In our study, the incorporation of creatinine clearance improved the prognostic ability of the original ACEF model for all types of cardiac surgery. On the other hand, clinical performance estimated by the ratio between observed and predicted mortality rates was fairly adequate for patients at the three risk groups, except with the EuroSCORE II model for the low risk group, the ACEF score for the high risk group, and the ACEFCG for the intermediate risk group. However, the reduced number of patients remaining in the intermediate and high risk groups could reduce the sample size power to detect a significant type II error. The EuroSCORE II is actually designed to predict in-hospital mortality, and the ACEF score in-hospital mortality plus mortality at 30 days after discharge. In our study, mortality was defined as in-hospital plus mortality at 30 days; therefore, there is some imbalance in the prediction analysis. Actually, the authors of the EuroSCORE II admitted that by including 30-day mortality, the death rates had to be increased by about 0.6%11. So, the real mortality risk prediction according to the EuroSCORE II should increase from 2.47% up to about 3%, with an overall observed/predicted mortality ratio equal to 1.31. All three scores demonstrated a very good negative predictive value (near 98%). Conversely, the positive predictive value for all three scores was very poor, not exceeding 10%. Since the models easily identified patients unlikely to die but not patients likely to die, they cannot be used to assess individual mortality risks. Notwithstanding, cutoff values for ROC curves are especially useful in transversal predictive models, usually correlating to a specific diagnosis. Consequently, its value in models discrimination assessment is limited, only showing the predicted mortality where the model is more accurate.
Only three external validations of the ACEF score have been done outside Italy. In Brazil, Mejia et al.4 found a poor discriminatory power with an AUC of 0.63. The ACEF score performance in a French population undergoing isolated or combined aortic valve replacement showed an AUC of 0.669. Finally, Chung et al.10 evaluated long-term outcome predictions using ACEF and ACEFCG scores in Asian patients with left main coronary artery disease undergoing CABG surgery. They found an AUC of 0.76 and 0.65 for all-cause death and cardiovascular mortality, respectively, with the ACEF score, whereas the accuracy with the ACEFCG score was 0.73 and 0.64 for each type of mortality. Although in our study, all-cause death was considered, cardiovascular origin was prevalent in presenting 30-day mortality; therefore, AUCs with ACEF models may be considered equivalent to those observed by Chung.
Notwithstanding the ACEFCG score showed an overall better performance than the ACEF score, the first model overestimated mortality in high-risk patients. In a recent research, Ranucci et al.18 found that the ACEF score overestimated mortality in very high-risk cardiac surgical patients (AUC 0.52).
Recently, Chang et al.19 found that the ACEF score satisfactorily predicted risk of acute kidney injury after mitral valve repair (AUC 0.76). They categorized renal failure severity by an increase in serum creatinine ≥ 0.3 mg/dl within 48 hours after surgery, or an increase in creatinine ≥ 1.5 times the baseline value within 7 days. In the present study, we did not categorize postoperative kidney injury by creatinine level assessment. Nevertheless, de novo dialysis in our population was three times lower than that found by Chang, though it was associated with a high mortality rate. In future research, the ACEFCG score should be considered a better candidate to screen patients at risk of renal failure after cardiac surgery.
A disadvantage of ACEF and ACEFCG scores is that predicted operative mortality risk is not directly calculated from the three variables, but must be extrapolated from an univariate association plot published by the authors1. The fast exponential growth of predicted mortality risk estimated from this plot may justify the overprediction of the ACEFCG score in higher-risk patients. Since the original chart was based on ACEF performance, a new tailored estimation for the ACEFCG score could be more accurate. Other limitation is that in the ACEFCG score, the age is included twice: age and creatinine clearance formula; hence, a co-linearity mistake may explain the miscalibration of this score.
The first limitation of the present study is its retrospective design, and that the sample represents only a portion of patients undergoing major cardiac surgery annually in Argentina. The low performance of the ACEF score in our sample, in terms of accuracy and calibration, may be explained by the fact that internal risk models, developed and validated in the same population, may perform better than externally developed models. Differences may be also due to different proportions of low- or high-risk patients enrolled in the present series, compared with the original series of ACEF score development. Nonetheless, mortality underestimation observed in high-risk patients may be interpreted as an inadequate behavior of the model, as poor performance of surgeons, or both. Except for the ACEF score in patients undergoing CABG, all the scores demonstrated a good calibration profile assessed by the HLχ² test, and for every type of surgery. Nevertheless, limitations of the HLχ² test are well-known and include its dependence on arbitrary groupings of patients, poor power in small data sets, as well as the fact that it only results in a p-value20, 21. The resulting calibration patterns were different when using the HLχ² or Spiegelhalter Z-tests, and miscalibration was usually found with the second method for all the scores and types of surgery, except for the ACEFCG score in patients undergoing AVR. While it is true that the HLχ² test is widely accepted to assess calibration, differences found in the present study with an alternative method should be considered when evaluating score performance. Finally, significant differences between the scores, in terms of accuracy, calibration and clinical performance, may be due to the effect of a small sample size.
In conclusion, the two ACEF models demonstrated variable efficacy in predicting mortality risk, as evaluated in terms of accuracy, calibration, or clinical performance. For clinical use, the ACEF score seems to be adequate to predict mortality in low- and intermediate-risk patients. Though the ACEFCG score had a better discriminatory power and calibration, it tended to overestimate the expected risk. In comparison, the EuroSCORE II showed better overall accuracy than both ACEF models. Nonetheless, although the original ACEF model had a worse discriminatory capacity and calibration than the other scores, an acceptable clinical performance may be expected in predicting operative mortality in the group of low- and intermediate-risk patients, particularly in those undergoing elective non-CABG surgery and combined or isolated AVR. Since ideally, a simpler risk stratification score should be desirable for bedside clinical use, the ACEF model reasonably met the expected performance in our population.

Conflict of interests: None declared

References

1. Ranucci M, Castelvecchio S, Menicanti L, Frigiola A, Pelissero G. Risk of assessing mortality risk in elective cardiac operations. Age, creatinine, ejection fraction and the law of parsimony. Circulation 2009; 119: 3053-61.         [ Links ]

2. Ranucci M, Castelvecchio S, Conte M, et al. The easier, the better: age, creatinine, ejection fraction score for operative mortality risk stratification in a series of 29,659 patients undergoing elective cardiac surgery. J Thorac Cardiovasc Surg 2011; 142: 581-6.         [ Links ]

3. Barili F, Pacini D, Capo A, et al. Reliability of new scores in predicting perioperative mortality after isolated aortic valve surgery: a comparison with the society of thoracic surgeons score and logistic EuroSCORE. Ann Thorac Surg 2013; 95: 1539-44.         [ Links ]

4. Mejía OA, Matrangolo BL, Titinger DP, et al. Age, Creatinine and Ejection Fraction Score in Brazil: Comparison with InsCor and the EuroSCORE. Arq Bras Cardiol 2015; 105: 450-6.         [ Links ]

5. Barili F, Pacini D, Grossi C, Di Bartolomeo R, Alamanni F, Parolari A. Reliability of new scores in predicting perioperative mortality after mitral valve surgery. J Thorac Cardiovasc Surg 2014; 147: 1008-12.         [ Links ]

6. Barili F, Pacini D, Rosato F, et al. In-hospital mortality risk assessment in elective and non-elective cardiac surgery: a comparison between EuroSCORE II and age, creatinine, ejection fraction score. Eur J Cardiothorac Surg 2014; 46: 44-8.         [ Links ]

7. Castelvecchio S, Menicanti L, Ranucci M; Surgical and Clinical Outcome Research (SCORE) Group. Development and validation of a risk score for predicting operative mortality in heart failure patients undergoing surgical ventricular reconstruction. Eur J Cardiothorac Surg 2015; 47: e199-205.

8. Capodanno D, Marcantoni C, Ministeri M, et al. Incorporating glomerular filtration rate or creatinine clearance by the modification of diet in renal disease equation or the Cockcroft-Gault equations to improve the global accuracy of the Age, Creatinine, Ejection Fraction [ACEF] score in patients undergoing percutaneous coronary intervention. Int J Cardiol 2013; 168: 396-402.         [ Links ]

9. Laurent M, Fournet M, Feit B, et al. Simple bedside clinical evaluation versus established scores in the estimation of operative risk in valve replacement for severe aortic stenosis. Arch Cardiovasc Dis 2013; 106: 651-60.         [ Links ]

10. Chung WJ, Chen CY, Lee FY, et al. Validation of scoring systems that predict outcomes in patients with coronary artery disease undergoing coronary artery bypass grafting surgery. Medicine (Baltimore) 2015; 94: e927.         [ Links ]

11. Nashef SA, Roques F, Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg 2012; 41: 734-44.         [ Links ]

12. Garg S, Sarno G, Garcia-Garcia HM, et al. A new tool for the risk stratification of patients with complex coronary artery disease: the Clinical SYNTAX Score. ARTS-II Investigators. Circ Cardiovasc Interv 2010; 3: 317-26.         [ Links ]

13. Cockcroft DW, Gault MH. Prediction of creatinine clearance from serum creatinine. Nephron 1976; 16: 31-41.         [ Links ]

14. Redelmeier DA, Bloch DA, Hickam DH. Assessing predictive accuracy: how to compare Brier scores. J Clin Epidemiol 1991; 44: 1141-46.         [ Links ]

15. Spiegelhalter DJ. Probabilistic prediction in patient management and clinical trials. Stat Med 1986; 5: 421-33.         [ Links ]

16. Windecker S, Kolh P, Alfonso F, et al. Guidelines on myocardial revascularization: The Task Force on Myocardial Revascularization of the European Society of Cardiology (ESC) and the European Association for Cardio-Thoracic Surgery (EACTS). Developed with the special contribution of the European Association of Percutaneous Cardiovascular Interventions (EAPCI). Eur J Cardiothorac Surg 2014; 46: 517-92.         [ Links ]

17. Borracci RA, Rubio M, Celano L, Ingino CA, Allende NG, Ahuad Guerrero RA. Prospective validation of EuroSCORE II in patients undergoing cardiac surgery in Argentinean centres. Interact Cardiovasc Thorac Surg 2014; 18: 539-43.         [ Links ]

18. Ranucci M, Di Dedda U, Castelvecchio S, La Rovere MT, Menicanti L; Surgical and Clinical Outcome Research (SCORE) Group. In search of the ideal risk-scoring system for very high-risk cardiac surgical patients: a two-stage approach. J Cardiothorac Surg 2016; 11: 13.         [ Links ]

19. Chang CH, Lee CC, Chen SW, et al. Predicting acute kidney injury following mitral valve repair. Int J Med Sci 2016; 13: 19-24.         [ Links ]

20. Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York: Springer; 2010, pp 270-9.         [ Links ]

21. Peek N, Arts DG, Bosman RJ, van der Voort PH, de Keizer NF. External validation of prognostic models for critically ill patients required substantial sample sizes. J Clin Epidemiol 2007; 60: 491-501.         [ Links ]

Creative Commons License Todo el contenido de esta revista, excepto dónde está identificado, está bajo una Licencia Creative Commons