We read with great interest the Letter to the Editor by Paiva et al. We appreciate their interest and their constructive comments on our article recently published in the Journal in which we develop and validate a simple risk stratification score (ProACS) for patients with acute coronary syndromes from a large nationwide registry.1 In our paper, the ProACS risk score showed a significantly lower discriminative power compared to the GRACE score in the external validation cohort, but slightly better compared to the development and internal validation cohorts. The Canada Acute Coronary Syndrome (C-ACS) risk score,5 developed with the same principles and objectives, yields similar discriminative power to our own score but is still considered adequate. Paiva et al. performed an additional external validation (highly recommended for any prediction model) in an independent contemporary cohort of 1000 consecutive myocardial infarction (MI) patients (43.5% with ST-elevation MI [STEMI]), fairly similar to the external validation cohort from our paper.2,3 Their results showed that 98% of in-hospital deaths were accurately classified in the intermediate- or high-risk groups; however, the ProACS score's discriminative power was significantly lower than that of the GRACE score (and slightly lower than our results), which might compromise the accuracy of risk stratification. They also performed a risk reclassification study, which showed that ProACS is better at identifying low-risk patients, particularly in the non-ST-elevation MI (NSTEMI) cohort. GRACE is superior in identifying high-risk patients. We obviously agree with the authors’ concluding remark that it remains to be determined whether the simplicity of this new score is offset by its inferior prognostic power compared to the gold standard GRACE risk score.

Our group also performed an external validation in patients from a single center, although with some different baseline characteristics, particularly a predominance of STEMI (62%) and with more cardiovascular risk factors.4 Compared to GRACE, we also obtained a lower discriminative power, with an area under the curve (AUC) of 0.769 for in-hospital mortality, similar to that reported for C-ACS, and slightly superior in STEMI (0.77 vs. 0.74), albeit non-significant.5 Also in an earlier study from our group in a STEMI cohort, we showed that a simple (reduced) score (not yet the ProACS score) led to significant improvement when reclassification was analyzed, particularly in patients without events, as was also demonstrated by Paiva et al.2,6

From a statistical point of view, dissatisfaction with AUC analysis has prompted proposals for new statistical metrics based on risk categories and reclassification.7 Reclassification analysis with the use of net reclassification improvement (NRI) and integrated discrimination improvement (IDI) are particularly useful because they can help to clarify and quantify the degree of correct reclassification of predicted probabilities. However, for some authors NRI has many of the same problems as the AUC.8,9 It is sometimes difficult to be sure if the measure is clinically meaningful due to lack of experience with the index. Some authors recommend the use of category-free (or continuous) NRI, avoiding predefined risk categories, but this can also mislead investigators by overstating the incremental value of an additional biomarker. Furthermore, without proper attention to model fit, NRI can mislead researchers and it is recommended to use bootstrap methods for estimating the variance of NRI and constructing confidence intervals. For those reasons, some caution is advised when interpreting reclassification analysis. Also, in the presence of a fairly robust risk score, such as GRACE, the quantitative improvement in model performance is expected to be small or even negative.

Finally, from a clinical point of view, the fact that ProACS's predicted ability is lower than GRACE does not hinder its application, because an AUC ≥0.75 means that it is still valid. The ProACS risk score better identifies those who do not have events. This is clinically important, because identification of these “truly low-risk patients” may enable better patient selection, avoiding unnecessary interventions that can increase costs as well as the risk of intervention-related adverse events, and may help in the selection of patients for early discharge. As we stated in our paper, risk stratification is a dynamic process that requires risk recalculation after admission. ProACS can be used at the first medical contact, when it is important to decide whether to refer the patient directly to a tertiary hospital, and due to its simplicity, even healthcare professionals without advanced medical or cardiological training (in a pre-hospital setting or in emergency department triage) can use this simple score. However, when full clinical and laboratory data are available, clinicians should calculate the GRACE score, because it provides more accurate risk stratification, which is crucial to patient management decisions.