Individualized risk stratification for optimal management of acute coronary syndrome (ACS) helps to identify high-risk patients who could benefit most from invasive strategies and prevents complications from unnecessary treatment in low-risk patients, in addition to helping with counseling on prognosis. Accordingly, clinical practice guidelines recommend that risk assessment should be initiated soon after admission, and that the Global Registry of Acute Coronary Events (GRACE) risk score should be preferred to all other risk scores because of its superior accuracy.1–3 The GRACE risk score (version 1.0) was developed from a large multinational prospective patient registry and validated in several studies for the prediction of in-hospital and 6-month mortality rates across a wide range of ACS.4–23 However, it was introduced in the 2000s and its predictive accuracy may not be adequate for current clinical practice, as use of evidence-based therapies has now increased.1–3 An update (version 2.0) was therefore released recently.24 The new risk score (GRACE 2.0) was presented as a more accurate tool than GRACE 1.0, with prediction of mortality over the longer term (1 and 3 years).24 It was derived from the GRACE registry with over 32000 patients enrolled between 2002 and 2007, from 14 countries in Europe, North and South America and Australia. However, this substantial geographic variation has led to the need for validation in different countries. Accordingly, the purpose of the present study was to validate GRACE 2.0 in Turkey.

The baseline characteristics of the study population are presented in Table 2, together with those of the original cohort used to develop GRACE 2.0. Among 396 patients, 270 (68%) had NSTEMI and 126 (32%) had UA. There were eight (2%) in-hospital deaths, all from cardiovascular cause. By one year after discharge, 48 (12.4%) patients had died; of these, 41 (10.4%) had NSTEMI and 7 (1.8%) had UA.

Table 3 presents the analysis of the model's discrimination and calibration performance, as outlined below.

In-hospital

In terms of discrimination performance, the AUC for predicting in-hospital mortality was <0.70 for each risk category as well as for the overall cohort, denoting inadequate discrimination (Figure 1).

Figure 1.

Receiver operating characteristic curves for in-hospital and 1-year mortality. AUC: area under the curve; CI: confidence interval; SE: standard error.

(0.17MB).

Calibration performance was not reported because it would be misleading when >20% of the cells of the two-way table have expected frequencies of <5, as in this case.28

The present study is the first to assess the validity of the GRACE 2.0 risk predictor in a contemporary cohort of patients with NSTE-ACS. The results for in-hospital mortality were inconclusive. Because in-hospital death was observed in a small number of patients (8/396), application of the Hosmer-Lemeshow test was not appropriate. The same factor could also be responsible for the inadequate discrimination for hospital mortality. At one year, calibration performance was good for both the overall cohort and the subgroups classified according to their initial risk category or final management option (Figure 1 and Table 3). On the other hand, although the discrimination performance was good when applied to the overall cohort, different results were observed for the subgroups; it was only accurate for low-risk patients, whereas a tendency to underestimate mortality risk among intermediate- to high-risk patients was observed at one year. This underestimation by GRACE 2.0 might be partly related to differences in the underlying reasons for categorizing patients as high risk (e.g. a 90-year-old patient with left ventricular ejection fraction of 15% vs. a young patient in cardiac arrest). It is also clear that not all variables that might have an impact on prognosis are necessarily included in GRACE 2.0, such as adherence to guideline-directed medical therapy, which in the present study was comparable to previous European and US studies (Supplementary Table).3,29–31 Therefore, it should be emphasized that risk prediction tools can support but not replace sound clinical judgment. There are other potential explanations for the different observations, especially for the subgroups. The reason for inadequate discrimination performance at one year in patients who underwent CABG might be the low number of deaths observed (only 4/54). In other patients, the good discrimination performance obtained may be related to the appropriateness of proceeding with PCI (e.g. to opt for revascularization of ischemia-producing lesions with prognostic significance, such as in the proximal left anterior descending artery, or for medical management in non-ischemic lesions).

In contrast to the previous version of the GRACE score, the new version provides better accuracy and predicts mortality directly. In other words, GRACE 2.0 improves scores because predicted mortality has a non-linear relationship with score.24 Thus, in the present study, the outputs were given in terms of probability of dying as a ‘percentage’ instead of a ‘score’, and the model was designated the ‘GRACE 2.0 risk predictor’ instead of ‘GRACE 2.0 risk score’.

Because every population has its unique characteristics and the present study took place in a different patient population, the baseline characteristics of both the present study's cohort and the original GRACE 2.0 cohort are provided in Table 2. The GRACE 2.0 cohort had the following characteristics more frequently than that of the present study: advanced age, ST-segment deviation, smoking, and peripheral arterial disease. In contrast, patients presenting with angina, diabetes and/or elevated troponin were more common in the present study. These differences make direct comparison difficult, highlight the importance of conducting validation studies in different patient populations, and may be a potential explanation for the discrepancy observed between the results of this validation study and the GRACE 2.0 study. It should be emphasized that the different results obtained from the present study do not imply that GRACE 2.0 is an inaccurate risk predictor; they only mean that the risk predictor was applied in a different patient population.

Several studies have been conducted regarding validation of the previous risk score (GRACE 1.0) in different populations, in different countries and with different types of ACS and different follow-up periods (in-hospital, six months, and/or one year).4,5,7–23 After an extensive analysis of the literature, Khalill et al. concluded that GRACE 1.0 was better than other risk scores, simpler to use, and a powerful predictor of adverse outcomes in patients with ACS.6 However, application of GRACE 1.0 may nowadays be limited. First, most of these validation studies10–12,14 were conducted before 2002, and since then, management of ACS has changed considerably. Second, some more contemporary studies4,13,16,17,20–23 only reported in-hospital mortality. Finally, some studies12,15,22 did not report calibration performance. The latter is important because, as well as having a good discrimination performance, a model is expected to have an adequate level of calibration.26–28 The two are complementary, as discrimination reflects the risk predictor's ability to individually distinguish patients who will experience the outcome (e.g. death, in this case) from patients who will not, whereas calibration reflects the risk predictor's ability to predict death in a given population on the basis of a collective estimate rather than an individual one. The above-mentioned shortcomings were not identified in four studies7,8,18,19 which supported the accuracy of GRACE 1.0 for the prediction of six-month mortality.

To date, there have been two studies on external validation of the new GRACE 2.0: the FAST-MI 2005 registry and a study by Fujii et al.32,33 FAST-MI included patients with NSTEMI and ST-elevation myocardial infarction (STEMI) (i.e. UA was excluded), and was the basis for the prediction of 3-year mortality in the development of GRACE 2.0. In this registry, AUCs of 0.83 and 0.82 for the overall cohort were reported, indicating excellent discrimination at one and three years, respectively. However, it should be noted that the May-Hosmer goodness-of-fit p-value for one-year mortality was less than 0.001. Despite unfavorable calibration performance at one year, the p-value for three-year mortality was 0.60, indicating improved calibration performance over time. In the other study, Fujii et al. enrolled patients with STEMI (i.e. both UA and NSTEMI were excluded). They reported excellent discrimination performance with AUCs of 0.95 and 0.92 for in-hospital and one-year mortality, respectively, but did not report calibration performance.

The proportion of patients who underwent CAG in the present study was 25% (15/60) and 66% (99/149) for high-risk and intermediate-risk patients, respectively. In contrast, 92% (164/179) of low-risk patients underwent CAG. It should be noted that the ages of high-risk patients were ≥85 years in 21 patients, 80–84 in 16, 70–79 in 21, and <70 in only two patients, which could partially explain the reluctance of clinicians to proceed with an invasive procedure in these patient groups. However, the justification for the higher rate of CAG among low-risk patients even than among intermediate-risk patients was not apparent. One reason for this discrepancy may be the phenomenon known as the ‘treatment-risk paradox’ (i.e. most interventions were performed in low-risk rather than high-risk patients).12,34 The treatment-risk paradox is a common issue in current practice, and was seen even in the original GRACE registry.34 The source of the paradox is not known,35 but it might be partly explained by the fact that it is difficult for the clinician to weigh potential benefits against potential hazards. Additional considerations may include defensive medicine (practiced to protect against possible litigation), financial incentives, framing effects, blind obedience, inefficient processes and communication obstacles, and system errors (including problems with policies and procedures).36

This was a single-center study. The sample size was relatively small; however, it was adequate for performing AUC analysis and assessing goodness-of-fit in terms of accurate statistical applications. GRACE 2.0 is the established model not only for NSTE-ACS but also STEMI. Thus, it may be inappropriate to extrapolate the results to those with STEMI. The results may also differ in different healthcare settings and different geographic locations. In addition, we did not calculate the probability of death/myocardial infarction at one year, or death or death/myocardial infarction at three years. It was difficult for us to monitor patients for myocardial infarction, in contrast to the precise determination of death. We focused on the probability of death at one year, because the data for the 1-year death model is more recent and more abundant.24 Finally, because ‘elevated troponin’ is a variable in the risk calculator, the type of assay used (i.e. highly sensitive vs. conventional assay) might have had an impact on the calculation of risk. The latter was used for both the present cohort and the GRACE 2.0 cohort. The precise role of highly sensitive assays in primary risk assessment remains unknown at present.37

In a contemporary cohort of NSTE-ACS patients, the GRACE 2.0 score was valid for 1-year mortality assessment. Its value for in-hospital mortality requires validation in a larger cohort.

None declared.

The authors have no conflicts of interest to declare.