Abstract
QRISK algorithms use data from millions of people to help clinicians identify individuals at high risk of cardiovascular disease (CVD). Here, we derive and externally validate a new algorithm, which we have named QR4, that incorporates novel risk factors to estimate 10-year CVD risk separately for men and women. Health data from 9.98 million and 6.79 million adults from the United Kingdom were used for derivation and validation of the algorithm, respectively. Cause-specific Cox models were used to develop models to predict CVD risk, and the performance of QR4 was compared with version 3 of QRISK, Systematic Coronary Risk Evaluation 2 (SCORE2) and atherosclerotic cardiovascular disease (ASCVD) risk scores. We identified seven novel risk factors in models for both men and women (brain cancer, lung cancer, Down syndrome, blood cancer, chronic obstructive pulmonary disease, oral cancer and learning disability) and two additional novel risk factors in women (pre-eclampsia and postnatal depression). On external validation, QR4 had a higher C statistic than QRISK3 in both women (0.835 (95% confidence interval (CI), 0.833–0.837) and 0.831 (95% CI, 0.829–0.832) for QR4 and QRISK3, respectively) and men (0.814 (95% CI, 0.812–0.816) and 0.812 (95% CI, 0.810–0.814) for QR4 and QRISK3, respectively). QR4 was also more accurate than the ASCVD and SCORE2 risk scores in both men and women. The QR4 risk score identifies new risk groups and provides superior CVD risk prediction in the United Kingdom compared with other international scoring systems for CVD risk.
Similar content being viewed by others
Main
CVD is the leading cause of death globally and was responsible for an estimated 17.9 million deaths in 2019 (ref. 1). International guidelines from the World Health Organization2, United States3, Europe4 and the United Kingdom5 all recommend the use of CVD risk prediction tools to target those at high risk for interventions to reduce risk. Consequently, the effectiveness of public health policies relies on risk prediction tools that identify all the important risk groups in the population, with validated risk estimates across the full range of population characteristics. The United States recommends the ASCVD score, which is based on the Pooled Cohort Equations and was developed using 20,338 non-Hispanic and 1,647 African American individuals3,6. European guidelines recommend SCORE2, which was developed using data from 677,684 (refs. 4,7) and SCORE-OP4 developed using data from 28,503 participants from Norway7, and the United Kingdom recommends QRISK3, which was developed using a large, diverse community population of 7.9 million people5,8.
Recent research has highlighted conditions associated with increased CVD risk that are not captured by any of the three most widely used CVD equations globally: ASCVD (ref. 3), QRISK3 (ref. 8) and SCORE2 (ref. 4,7). These include chronic obstructive pulmonary disease (COPD)9, learning disability10, Down syndrome11,12, cancer13 and reproductive health conditions14. If these conditions are independently associated with increased CVD risk, then current CVD risk scores will underestimate risk in these groups, and people with these diagnoses may not be offered the opportunity for beneficial interventions to improve survival. Equally, if risks are overestimated, then individuals may receive unnecessary interventions15. Furthermore, more accurate CVD risk tools are useful for identifying those at higher CVD risk for recruitment into clinical trials, especially for primary prevention.
We sought to derive a new population-based CVD risk score, QR4, to include novel risk factors and account for competing risks, and to externally evaluate its performance against three widely used CVD risk scores (that is, ASCVD, QRISK3 and SCORE2), in large and diverse populations of over 16 million people drawn from across the United Kingdom. We used two established electronic records research databases (QResearch and Clinical Practice Research Datalink (CPRD) GOLD), both of which contain anonymized data collected during routine National Health Service (NHS) clinical care.
Results
Study population
There were 9,976,306 people aged 18–84 years in the QResearch English derivation cohort, 3,246,602 in the QResearch English validation cohort and 3,542,007 in the CPRD validation cohort from the other three UK nations (that is, Scotland, Wales and Northern Ireland). Extended Data Table 1 shows the flow of patients and the relevant exclusions. Extended Data Table 2 shows the new predictors under consideration.
The baseline characteristics of each cohort and the completeness of the recording of data for predictors with missing data are shown in Table 1. The cohorts were broadly similar, except that both English cohorts contained more complete data for ethnicity, smoking, cholesterol and body mass index (BMI) than the CPRD validation cohort from the other three UK nations. Supplementary Table 4 shows the characteristics of participants with complete versus missing data in the QResearch derivation cohort: those with complete data tended to be older, and more likely to be female and to have clinical conditions.
There were 202,424 incident CVD cases (based on primary outcome definition) from 49.1 million person-years in the derivation cohort. Extended Data Table 1 shows the types of CVD events in each cohort for each of the three outcome definitions.
The crude CVD incidence rates for the primary CVD outcome by age, sex, ethnicity and calendar year in the English derivation cohort and CPRD validation cohort are shown in Extended Data Table 3. CVD rates using linked data were higher in the English cohort, which was largely explained by the additional data linkage to hospital and mortality data. Extended Data Fig. 1 shows both CVD incidence rates and non-CVD mortality rates by calendar year and month for the whole study period. CVD rates per 1,000 person-years were lower in 2020, the first year of the COVID-19 pandemic, when the overall rate was 4.03 (95% CI, 3.97–4.08) but returned to pre-pandemic levels in 2021 (4.31; 95% CI, 4.25–4.37). Non-CVD mortality rates increased from 3.45 (95% CI, 3.40–3.50) in 2019 to 3.84 (95% CI, 3.79–3.89) in 2020 and remained elevated in 2021.
Factors associated with increased risk of CVD
The adjusted hazard ratios for CVD incidence in the final cause-specific models in men and women (evaluated at the mean age of 39 years for variables with age interactions) are shown in Fig. 1. Extended Data Fig. 2 shows the adjusted hazard ratios for the fractional polynomial terms for CVD risk for continuous variables and the predictor variables with significant age interactions for both men and women. Supplementary Figs. 1 and 2 show the corresponding results for non-CVD death.
There were seven new CVD predictors in men and women (brain cancer, lung cancer, Down syndrome, blood cancer, COPD, oral cancer and learning disability) and two additional predictors in women (pre-eclampsia and postnatal depression).
We found no association between the following variables and CVD risk in men or women: asthma, hyperthyroidism, hypothyroidism, antiphospholipid antibody syndrome, benign intracranial hypertension, HIV or AIDS, and the remaining cancers. In women, there were no associations with in vitro fertilization, endometriosis, polycystic ovarian syndrome, gestational diabetes, miscarriage, termination or placental abruption. No violations of the proportional hazard assumptions were detected graphically. The values for the heuristic shrinkage16 were all very close to one (0.99), indicating no evidence of overfitting.
New CVD predictors in women
The adjusted hazard ratios (95% CI) for the nine new independent predictors of CVD risk in women (evaluated at the mean age of 39 years for variables with age interactions) are as follows: brain cancer, 4.52 (2.49–8.21); lung cancer, 3.50 (1.31–9.38); Down syndrome, 3.18 (2.40–4.22); blood cancer, 2.13 (1.71–2.67); COPD, 1.85 (1.50–2.29); oral cancer, 1.55 (1.27–1.89); learning disability, 1.45 (1.29–1.64); pre-eclampsia, 1.56 (1.36–1.78); and postnatal depression, 1.18 (1.11–1.26).
The adjusted hazard ratios for several of these predictors were higher at younger ages (for example, under 35 years), except for lung cancer in women, for which adjusted hazard ratios were highest for those around age 40 years and then declined gradually with increasing age (Extended Data Fig. 2). The adjusted hazard ratios (95% CI) at age 69 were as follows: brain cancer, 2.18 (1.29–3.71); lung cancer, 1.97 (1.64–2.37); blood cancer, 1.39 (1.28–1.50); COPD, 1.38 (1.32–1.44); and pre-eclampsia, 1.12 (1.01–1.24).
The magnitude and direction for many of the adjusted hazard ratios for the competing outcome of non-CVD death in women were similar to those for CVD except for large adjusted hazard ratios (evaluated at age 39 years) for Down syndrome (18.32; 95% CI, 16.24–20.66), lung cancer (49.94; 95% CI, 40.61–61.43) and brain cancer (33.35; 95% CI, 26.17–42.49). The adjusted hazard ratios for non-CVD death for family history of coronary heart disease, pre-eclampsia and migraine were significantly less than one (Supplementary Fig. 1).
New CVD predictors in men
The adjusted hazard ratios for the seven new independent predictors of CVD risk in men (evaluated at age 39 years are shown in Fig. 1), and the adjusted hazard ratios (95% CI) for these predictors are as follows: brain cancer, 5.45 (3.49–8.50); Down syndrome, 2.35 (1.84–2.99); blood cancer, 2.06 (1.78–2.39); lung cancer, 1.66 (1.45–1.92); oral cancer, 1.49 (1.30–1.70); COPD, 1.37 (1.32–1.41); and learning disability, 1.17 (1.07–1.29). The adjusted hazard ratios in men for brain cancer and blood cancer declined with age; for example, at age 69 years, the adjusted hazard ratio (95% CI) was 2.12 (1.25–3.61) for brain cancer and 1.23 (1.15–1.31) for blood cancer.
The adjusted hazard ratios for the additional models were similar to our final main models in men and women (Supplementary Figs. 3–7). Model A includes the original QRISK3 predictor variables but without competing risks. Model B is similar to our final model, but the follow-up time ended on 29 February 2020, before the COVID-19 pandemic. Model C shows that the adjusted hazard ratios for CVD risk were similar across periods of time after diagnosis with one of the four cancers (except for oral cancer in women), although the adjusted hazard ratios for non-CVD deaths varied with the highest values for more recently diagnosed cancers.
Predicted risks
The way in which each of the new risk factors affects the predicted 10-year CVD risk for specific individuals is shown in Fig. 2. In this illustration, which is presented for both men and women across ages 18 to 84 years, CVD risk was compared between individuals with a new risk factor and reference individuals with no adverse clinical indicators (a cholesterol/HDL ratio of 4.0, an SBP of 125 mm Hg and a BMI of 25 kg m−2). These risk calculations show the impact of the new risk predictors, which mainly resulted in increased predicted risks compared with the reference individuals at younger ages and decreased predicted risks at older ages as competing risks become more pronounced. Using a reference group of individuals with various conventional risk factors (light smokers with a cholesterol/HDL ratio of 6.0, an SBP of 170 mm Hg and a BMI of 35 kg m−2), a similar pattern was observed, albeit with higher overall predicted risks (Supplementary Fig. 8).
Discrimination
The performance statistics (C statistic, calibration slope and calibration intercept) for QR4 and QRISK3 for the validation cohorts in England, Scotland, Wales and Northern Ireland are shown in Table 2. The C statistic for QR4 was marginally higher than that for QRISK3 in both validation cohorts. For example, the C statistics were 0.835 (95% CI, 0.833–0.837) and 0.831 (95% CI, 0.829–0.832) for QR4 and QRISK3, respectively, in women in the devolved administrations (Scotland, Wales and Northern Ireland). The corresponding values in women in England were 0.864 (95% CI, 0.862–0.866) for QR4 and 0.862 (95% CI, 0.860–0.864) for QRISK3. The C statistic values were generally higher in England than in the other three nations, although all values remained within an excellent range (>0.8). The results for men were similar, though the values were slightly lower.
The overall discrimination results and discrimination results delineated by ethnic group for QR4, SCORE and ASCVD are shown in Extended Data Table 4; these results were restricted to those aged 40 years and older in the validation cohort in England. For women, overall discrimination was highest with QR4 (0.781; 95% CI, 0.778–0.784), followed by ASCVD (0.767; 95% CI, 0.764–0.770) and SCORE2 (0.767; 95% CI, 0.764–0.770). There was a similar pattern for men.
The C statistics, calibration slopes and calibration intercepts (overall and by ethnic group) for QRISK3 and QR4 in men and women aged 18–84 years in the English validation cohort are shown in Extended Data Table 5. These results show that discrimination varied by ethnic group in England: the C statistic for QR4 was highest for Chinese men (0.923; 95% CI, 0.906–0.939) and lowest for Caribbean men (0.825; 95% CI, 0.801–0.841).
The definitions of the CVD outcomes used for sensitivity analyses are shown in Supplementary Table 1. Supplementary Table 2 shows the performance statistics for QR4, SCORE2 and ASCVD for each outcome measure for each of the four UK nations among those aged 40 years and older. The C statistic values for all scores (QR4, SCORE2 and ASCVD) were highest for the tertiary outcome measure, and for all outcome measures discrimination values were higher for QR4 than those for SCORE2 and ASCVD, which yielded values similar to one another.
Decision curve analysis
The decision curves in Fig. 3 indicate a slightly larger net benefit with QR4 compared with QRISK3 and Model A, but differences in net benefit are more marked in the devolved administrations than in England.
The decision analysis curves for QR4, SCORE2 and ASCVD for the primary outcome in England are shown in Extended Data Fig. 3. Supplementary Figs. 9 and 10 show corresponding results for the secondary and tertiary CVD outcomes.
Calibration
QR4 was well-calibrated in England, showing a close correspondence between predicted and observed risks, whereas QRISK3 overpredicted risk in the higher centiles of predicted risk (Fig. 4). Table 2 shows the calibration slope and intercept values for QRISK3 and QR4 by country. There was a degree of miscalibration for QRISK3 and QR4 in each of the devolved administrations (Supplementary Fig. 11) on the basis of general practitioner (GP) data only.
The calibration results for ASCVD and SCORE2 in the English validation cohort, which are based on our primary outcome definition, are shown in Extended Data Fig. 4. Supplementary Table 2 and Supplementary Figs. 12 and 13 show the corresponding results for the secondary and tertiary outcomes. Overall, there was a degree of overprediction for ASCVD and a degree of underprediction for SCORE2, which were improved when comparisons were made with the more specific secondary and tertiary outcomes.
Reclassification
The characteristics of the 84,700 (2.6%) participants in the English validation cohort reclassified using QR4 instead of QRISK3 at the 10% risk threshold are shown in Extended Data Table 6. Of the 3,554 people reclassified from low risk to high risk using QR4, 1,168 (32.9%) had COPD, 57 (1.6%) had a learning disability, 72 (2.0%) had Down syndrome, 72 (2.0%) had a history of pre-eclampsia, 125 (3.5%) had a history of postnatal depression, 90 (2.5%) had oral cancer, 54 (1.5%) had brain cancer, 92 (2.6%) had lung cancer and 322 (9.1%) had blood cancer. Those reclassified as high risk using QR4 tended to be younger (mean age of 52.4 years) than the 81,146 people reclassified as low risk (mean age of 60.5 years). Supplementary Table 3 shows the corresponding analyses for the 4,068 participants reclassified as high risk using QR4 compared with Model A as well as the 12,791 participants reclassified as low risk; the pattern was similar, although the total number of participants that were reclassified was much smaller (16,859, 0.52%).
Discussion
We have developed and externally validated a new CVD risk score, QR4, that incorporates nine novel predictors with good face validity and clinical utility to predict 10-year risk of CVD in a diverse population of men and women. These new predictors (in both men and women) are learning disability, Down syndrome, COPD, lung cancer, oral cancer, blood cancer and brain cancer, and pre-eclampsia and postnatal depression in women only. The performance of QR4 was more accurate than other widely used CVD risk scores, namely, ASCVD, QRISK3 and SCORE2. QR4 is likely to result in clinically important changes in risk, leading to different CVD risk reduction advice or interventions, particularly for those with the new predictors, which might lead to interventions at an earlier age, as in the examples given. Furthermore, QR4 accounts for the competing risk of non-CVD death, thereby reducing overprediction of risk, especially among the more elderly populations17. Last, we have used and published the SNOMED-CT clinical code groups used to derive our model, facilitating reuse for further research and international comparisons.
Widely available CVD risk equations have been used for many millions of CVD health checks worldwide and are supported by international guidelines2,3,4,5. However, it is important that guidance is based on the best algorithms available because this will materially affect which patients are offered risk-reducing interventions. Failure to adequately assess CVD risk and offer appropriate risk-reducing interventions across all patient groups could further disadvantage vulnerable patients, particularly cancer survivors and those with significant comorbidities such as COPD, Down syndrome, a learning disability or a history of postnatal depression or pre-eclampsia. Although the underlying conditions themselves may not be modifiable, the identification of high-risk people in these groups can lead to targeted interventions to reduce CVD risk.
Our findings regarding QR4 for cancer are particularly striking and confirm associations with CVD risk for four cancers (that is, blood, brain, lung and oral)13 despite accounting for reduced life expectancy using a competing risk analysis. The increased risk of CVD for cancer survivors particularly at younger ages needs to be considered in the context of the prognosis of the cancer itself because it would be inappropriate to prescribe therapies that lower CVD risk for those with a very poor prognosis. Although only 15% of people with lung cancer survive more than 5 years, 90% of people with blood cancers18 and 55% of people with oral cancers now survive more than 5 years19, and hence, targeted prevention has a potential clinical net benefit. The use of QR4 in clinical practice will need careful consideration and discussion in patients with cancer and will need to account for patient preferences. There are also opportunities for further research to more finely characterize the association between cancer treatment(s) and subsequent CVD. However, longitudinal data on cancer treatments (such as radiotherapy and chemotherapy) are only just becoming available for this type of research data in the United Kingdom. These data are not yet routinely linked to primary care data for clinical use, so at present, they could not be used to implement more personalized risk prediction.
The lack of an association between asthma and CVD risk is interesting, especially given the preconception that inhaled corticosteroids may increase the risk of CVD. By contrast, the 1.4-fold to 1.9-fold increased risk of CVD associated with COPD is consistent with the 2-fold increased risk of CVD reported among US patients hospitalized with COPD9 and is clinically very important. COPD is now one of the top three most deadly diseases worldwide, resulting in an estimated 3 million deaths annually, 90% of which occur in low-income and middle-income countries20. It is striking that this association was strongest in women with COPD, and there are two important implications of this finding. First, clinicians need to actively consider COPD as a diagnosis and to confirm it with spirometry, especially in women who are often neglected in this regard21,22. Second, therapies that reduce CVD risk should be prescribed; these therapies include optimizing inhaled therapies, as this has now been demonstrated to reduce mortality23,24.
The increased risk of CVD associated with pre-eclampsia declined with age, but the 54% increase that we observed at a mean age of 39 years is consistent with other research25,26 and may reflect damage to the maternal cardiovascular system26. This highlights an important opportunity to systematically target CVD prevention14. The twofold to threefold increased risk of CVD among those with Down syndrome is consistent with the results of the limited analyses available11 and may reflect premature aging and adverse cardiometabolic profiles. This underscores US recommendations for continued CVD research and surveillance in people with Down syndrome, especially given improved life expectancy12. Incorporation of postnatal depression and learning disabilities into QR4 will help to operationalize policy initiatives to ensure parity of esteem with physical health for these patients.
Our study reports robust discrimination for ASCVD and SCORE2. Although there was a degree of miscalibration with ASCVD and SCORE2 with the main outcome, which used a broader definition of CVD, this improved with endpoint definitions aligned with those for which ASCVD and SCORE2 were developed. Any residual miscalibration compared with the original studies may relate to a combination of different study populations (which might have different underlying CVD rates), different cohort selection criteria (for example, the inclusion of statin users in the SCORE2 studies), use of a different study period, and use of recalibration measures in SCORE2 (including the differential application of multipliers by age and sex, which are yet to be validated). This suggests that CVD risk equations may be transportable to other geographical settings if recalibrated.
The strengths and limitations of this study are similar to those for other well-established risk prediction tools. The strengths include size, duration of follow up, representativeness, lack of selection, recall and respondent bias, and no evidence of overfitting8. The inclusion of more granular information on predictors is a strength in that the predictions for individual patients are likely to better reflect their individual risk, although this needs to be balanced against the increased complexity of the algorithm with regard to its implementation. However, this is mitigated in settings for which electronic health records are available because most relevant information is already available at the point of care and can be automatically populated27. Although we report improved discrimination for QR4 compared with QRISK3, the absolute values of the improvement in the C statistics were small. The C statistic is a familiar but limited measure that does not effectively balance misclassification errors28. It needs to be interpreted in the context of other relevant measures, including decision analysis, reclassification, calibration and clinical utility28. Our study has strong face validity because it was conducted in a setting where most patients are managed, and hence, QR4 could be implemented in similar clinical settings, subject to local validation or recalibration. Last, our results are unlikely to have been affected by the COVID-19 pandemic in 2020 and 2021, as the risk factors were predominantly recorded prior to the pandemic. The CVD incidence rates were temporarily affected in 2020 but have since returned to pre-pandemic levels, and Model C showed very similar results to our main model.
One limitation is the lack of formal adjudication of CVD diagnoses. However, the use of linked hospital and mortality data ensures a clear ascertainment of CVD outcomes in the English cohorts. The miscalibration for QR4 in the other three UK nations (Scotland, Wales and Northern Ireland) reflects the lack of linked hospital data and Office for National Statistics mortality outcome data for these nations in the CPRD validation cohort, as this would have resulted in an underestimation of CVD outcomes. Although there is potential for bias because of missing data, our data are substantially more complete than previous studies8, with the mitigation of residual biases through multiple imputations using recommended approaches29. We expect that in clinical practice, any missing data will be collected from the patient or their caregiver during a consultation with the clinician, so missing data for the implementation of QR4 are unlikely to be a substantial issue. Although our validation covers a fully external population, further research should validate QR4 in different countries with different CVD rates. This could be addressed by further validation using different datasets with appropriate data linkages.
In conclusion, these results demonstrate the strength of QR4 in the general UK population and its superior performance compared with three other widely used international CVD risk scores. QR4 enables more accurate CVD risk estimation, which should lead to significant improvements in health outcomes, especially for those with COPD, Down syndrome and a learning disability, cancer survivors and women with pre-eclampsia or postnatal depression.
Methods
Study design and data sources
We undertook community-based cohort studies using two large electronic medical records databases, QResearch and CPRD GOLD. We randomly allocated three-quarters of QResearch practices in England to the derivation cohort and the remainder to an internal English validation dataset. Both QResearch and CPRD GOLD are based on anonymized medical records data collected during the course of clinical care. QResearch is based on a commercial computer system known as Egton Medical Information Systems (EMIS), whereas CPRD GOLD is based on a different commercial system known as Vision. We used CPRD GOLD practices from three UK nations (Scotland, Wales and Northern Ireland) to create a second fully external geographically distinct validation cohort.
We included adults aged 18–84 years between 1 January 2010 and 31 December 2021. The cohort entry date was the latest of the following: 18th birthday, date of registration with the practice plus 1 year or 1 January 2010. We excluded participants with pre-existing CVD, those prescribed statins and (for QResearch) those with a missing Townsend deprivation score because they often represent temporary or incompletely registered patients with substantial missing data30. We followed participants up until the earliest date of CVD diagnosis, death, deregistration with the practice or the study end date.
Outcome definitions
Our primary outcome for model derivation and validation in QResearch was an incident diagnosis of CVD (fatal or nonfatal myocardial infarction, ischemic heart disease, ischemic, hemorrhagic or unspecified stroke or transient ischemic attack) identified from the GP record or linked mortality and hospital records using published clinical codes31. Our primary outcome for model validation in CPRD GOLD was based on the same diagnoses but recorded solely from the GP data because linked data for deaths and hospital admissions were not available for Scotland, Wales and Northern Ireland.
We had two additional outcomes for the validation comparisons between QR4, SCORE2 and ASCVD. Our secondary outcome, aligned with ASCVD, included nonfatal myocardial infarction or coronary heart disease-related death and fatal or nonfatal stroke. Our tertiary outcome was similar to our secondary outcome but also included fatal congestive cardiac failure, hypertension and cardiac arrhythmias in order to align with the SCORE2 outcome definition4,7. For more details of the definitions of the primary and two additional outcomes, including the SNOMED-CT and ICD-10 codes used, see Supplementary Table 1. We compared the performance of all three algorithms (QR4, SCORE2 and ASCVD) using all three outcome definitions only in England because of the availability of linked cause of death data, which were not available for the devolved administrations.
Predictor variables
We included established risk factors from ASCVD3, QRISK3 (ref. 8) or SCORE2 (ref. 4) and new candidate variables highlighted in the literature (see Extended Data Table 2, which includes more details of the definitions of each predictor considered)9,10,11,12,13,32. Cholesterol ratio was defined as total serum cholesterol/HDL. Ethnicity was self-reported.
Model development
We used cause-specific Cox models to estimate the 10-year risk of CVD, accounting for non-CVD death as a competing risk for men and women separately using the biological sex recorded on the electronic health record33. This involved fitting two separate Cox models: one for CVD diagnoses and CVD deaths and one for non-CVD deaths with time from cohort entry as the underlying function34. We used fractional polynomials35 to model nonlinear risk relationships with continuous variables. We used multiple imputation with chained equations to replace missing values for ethnicity, BMI, SBP, total cholesterol, HDL and smoking status36. For binary variables, we coded them as present if there was a recorded diagnosis in the GP medical record and otherwise coded them as absent. We carried out five separate imputations for men and women in the derivation dataset. We included all predictor variables in the imputation model, along with age interaction terms, the Nelson–Aalen estimator of the CVD baseline cumulative hazard, the CVD outcome indicator, the baseline cumulative hazard and the outcome indicator for non-CVD death29. We combined results from Cox models using Rubin’s rules37. We included variables from existing QRISK3 models8 and retained additional variables with an adjusted hazard ratio of <0.90 or >1.10 (for binary variables) and statistical significance at the 0.01 level. We included significant interactions with age in the final model. We assessed model optimism by calculating heuristic shrinkage16. We combined estimates from the two cause-specific models to derive risk equations for the predicted risk of CVD at 10 years, accounting for competing events in men and women34.
We developed three additional models following peer review: Model A included the original QRISK3 parameters but did not account for competing risks; Model B was similar to our final model, but the follow-up time ended on 29 February 2020, before the COVID-19 pandemic; and Model C included time since cancer diagnosis as a predictor variable.
Model evaluation
We also carried out multiple imputations, with five separate imputations for men and women in each validation cohort. We applied the risk equations to the internal and external validation cohorts and evaluated performance by country (England, Wales, Scotland and Northern Ireland).
We calculated concordance indices, equivalent to the C statistics, and accounted for competing risks33. We assessed model calibration, comparing the mean predicted risks at 10 years with the observed risks accounting for competing risks (cumulative incidence) by hundredths of predicted risk. We generated pseudo-values that accounted for competing risks in order to calculate the calibration slope and intercept at 10 years38.
We compared performance statistics for QR4 with those for ASCVD3,39, QRISK3 (ref. 8) and SCORE2 in England. We used the SCORE2 algorithm for those aged 40–69 years without diabetes4, SCORE2-OP for those 70 years and older without diabetes7 and SCORE2-Diabetes40 for those with diabetes using the authors’ published Stata code from July 2023 (ref. 41). We restricted comparisons between QR4, ASCVD and SCORE2 to people aged 40 years and older3,4,6. We also evaluated performance of QR4, ASCVD and SCORE2 using our secondary and tertiary outcomes.
Decision curve analysis
We used decision curve analysis accounting for competing risks in both validation cohorts, in order to evaluate the net benefit of QR4 compared with that of QRISK3 and Model A and compared these with alternative strategies, such as assuming that all people were treated or nobody was treated42. The strategy with the highest net benefit at any given risk threshold was considered to have the most clinical value43. We also used decision curve analyses in people aged 40 years and older to compare QR4 with ASCVD and SCORE2 in the English validation cohort using all three outcomes.
Reclassification statistics
We classified individuals as ‘high risk’ for CVD if their predicted 10-year risk was ≥10%, which is in line with current UK guidelines5. We compared the predicted risks of QR4 with those of QRISK3 and Model A to determine the percentage and characteristics of people reclassified at this high-risk threshold.
We also applied the predicted risk equations to men and women in the validation group to illustrate how each of the new risk factors affected 10-year CVD risk. In these calculations, men and women aged 18 to 84 were included, and CVD risk was compared between an individual with each of the new risk factors and a comparable reference individual but with no adverse clinical indicators (a cholesterol/HDL ratio of 4.0, an SBP of 125 mm Hg and a BMI of 25 kg m−2). For the example presented, we selected ‘White’ as the reference group, as this group had the largest number of participants.
We used all eligible individuals to develop and validate the models in order to maximize the power and generalizability of the results. We used Stata (version 17) for analyses.
Inclusion and ethics
This study used anonymized data from two electronic health care records databases, and hence, participant consent was not required. The databases cover a diverse population that is representative of the UK population. The QResearch ethics approval was completed by the East Midlands-Derby Research Ethics Committee (reference 18/EM/0400). The CPRD ERAP approval reference is 20_000162.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Data availability
To guarantee the confidentiality of personal and health information, only the authors had access to the data during the study in accordance with the relevant license agreements. The QResearch data are on the QResearch website (https://www.qresearch.org), and the CPRD data are on the CPRD website (https://www.cprd.com).
Code availability
Clinical codes are published under a creative commons license at https://www.qresearch.org/data/qcode-group-library, with accompanying details in Supplementary Table 1. Software implementing the QR4 algorithm will be made available for research under an academic license by Oxford University Innovations (enquiries@innovation.ox.ac.uk). The QRISK3 algorithm is available at https://qrisk.org/src.php, and the ASCVD code is available at https://econpapers.repec.org/software/bocbocode/s459162.htm. SCORE2 algorithms are available at https://www.phpc.cam.ac.uk (ps.reception@medschl.cam.ac.uk).
References
Cardiovascular diseases (CVDs). World Health Organization www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds) (2021).
Roth, G. A., Mensah, G. A. & Fuster, V. The global burden of cardiovascular diseases and risks: a compass for global action. J. Am. Coll. Cardiol. 76, 2980–2981 (2020).
Goff, D. C. Jr et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J. Am. Coll. Cardiol. 63, 2935–2959 (2014).
SCORE2 Working Group & ESC Cardiovascular risk collaboration. SCORE2 risk prediction algorithms: new models to estimate 10-year risk of cardiovascular disease in Europe. Eur. Heart J. 42, 2439–2454 (2021).
NICE Guideline [NG238]. Cardiovascular Disease: Risk Assessment and Reduction, Including Lipid Modification (National Institute for Clinical Excellence, 2023); www.nice.org.uk/guidance/ng238
Karmali, K. N., Goff, D. C. Jr, Ning, H. & Lloyd-Jones, D. M. A systematic examination of the 2013 ACC/AHA pooled cohort risk assessment tool for atherosclerotic cardiovascular disease. J. Am. Coll. Cardiol. 64, 959–968 (2014).
SCORE2-OP working group and ESC Cardiovascular risk collaboration. SCORE2-OP risk prediction algorithms: estimating incident cardiovascular event risk in older persons in four geographical risk regions. Eur. Heart J. 42, 2455–2467 (2021).
Hippisley-Cox, J., Coupland, C. & Brindle, P. Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ 357, j2099 (2017).
Sidney, S. et al. COPD and incident cardiovascular disease hospitalizations and mortality: Kaiser Permanente Medical Care Program. Chest 128, 2068–2075 (2005).
Wang, H. et al. Association of intellectual disability with overall and type-specific cardiovascular diseases: a population-based cohort study in Denmark. BMC Med. 21, 41 (2023).
Sobey, C. G. et al. Risk of major cardiovascular events in people with Down syndrome. PLoS ONE 10, e0137093 (2015).
Dimopoulos, K. et al. Cardiovascular complications of Down syndrome: scoping review and expert consensus. Circulation 147, 425–441 (2023).
Strongman, H. et al. Cancer history as a predictor in cardiovascular risk scores: a primary care cohort study. Br. J. Gen. Pract. 73, e34–e42 (2022).
Mehta, L. S. et al. Call to action: maternal health and saving mothers: a policy statement from the American Heart Association. Circulation 144, e251–e269 (2021).
DeFilippis, A. P. et al. An analysis of calibration and discrimination among multiple cardiovascular risk scores in a modern multiethnic cohort. Ann. Intern. Med. 162, 266–275 (2015).
Van Houwelingen, J. C. & Le Cessie, S. Predictive value of statistical models. Stat. Med. 9, 1303–1325 (1990).
Puddu, P. E., Amaduzzi, P. L. & Ricci, B. Coronary heart disease incidence and competing risks: an important issue. J. Geriatr. Cardiol. 14, 425–429 (2017).
Facts and information about blood cancer. Blood cancer UK http://bloodcancer.org.uk/news/blood-cancer-facts (2022).
Survival for mouth and oropharyngeal cancer. Cancer Research UK www.cancerresearchuk.org/about-cancer/mouth-cancer/survival (2023).
Chronic obstructive pulmonary disease (COPD). World Health Organization www.who.int/news-room/fact-sheets/detail/chronic-obstructive-pulmonary-disease-(copd) (2023).
Chapman, K. R., Tashkin, D. P. & Pye, D. J. Gender bias in the diagnosis of COPD. Chest 119, 1691–1695 (2001).
Jenkins, C. R. et al. Improving the management of COPD in women. Chest 151, 686–696 (2017).
Rabe, K. F. et al. Triple inhaled therapy at two glucocorticoid doses in moderate-to-very-severe COPD. N. Engl. J. Med. 383, 35–48 (2020).
Global strategy for prevention, diagnosis and management of COPD: 2023 report (Global Initiative for Chronic Obstructive Lung Disease, 2023); https://goldcopd.org/2023-gold-report-2/
Crump, C. et al. Adverse pregnancy outcomes and long term risk of ischemic heart disease in mothers: national cohort and co-sibling study. BMJ 380, e072112 (2023).
Magee, L. A., Nicolaides, K. H. & von Dadelszen, P. Preeclampsia. N. Engl. J. Med. 386, 1817–1832 (2022).
Majeed, A. Sources, uses, strengths and limitations of data collected in primary care in England. Health Stat. Q. 21, 5–14 (2004).
Pencina, M. J. & D’Agostino, R. B. Sr Evaluating discrimination of risk prediction models: the C statistic. JAMA 314, 1063–1064 (2015).
Bonneville, E. F., Resche-Rigon, M., Schetelig, J., Putter, H. & de Wreede, L. C. Multiple imputation for cause-specific Cox models: assessing methods for estimation and prediction. Stat. Methods Med. Res. 31, 1860–1880 (2022).
Townsend, P. Deprivation. J. Soc. Policy 16, 125–146 (1987).
Hippisley-Cox, J. Open Code Lists 2023 (QResearch, 2023); www.qresearch.org/data/qcode-group-library
Rawshani, A. et al. Risk factors, mortality, and cardiovascular outcomes in patients with type 2 diabetes. N. Engl. J. Med. 379, 633–644 (2018).
Wolbers, M., Koller, M. T., Witteman, J. C. M. & Steyerberg, E. W. Prognostic models with competing risks: methods and application to coronary risk prediction. Epidemiology 20, 555–561 (2009).
Kalbfleisch, J. D. & Prentice, R. L. The Statistical Analysis of Failure Time Data (John Wiley & Sons, Inc., 2002).
Royston, P., Ambler, G. & Sauerbrei, W. The use of fractional polynomials to model continuous risk variables in epidemiology. Int. J. Epidemiol. 28, 964–974 (1999).
White, I. R. & Royston, P. Imputing missing covariate values for the Cox model. Stat. Med. 28, 1982–1998 (2009).
Rubin, D. B. Multiple Imputation for Nonresponse in Surveys (John Wiley & Sons, 1987).
van Geloven, N. et al. Validation of prediction models in the presence of competing risks: a guide through modern methods. BMJ 377, e069249 (2022).
Linden A. ASCVD: Stata Module for Computing ACC/AHA 10-Year Risk for an Initial Hard Atherosclerotic Cardiovascular Disease (ASCVD) Event. Statistical Software Components S459162 (Boston College Department of Economics, 2023).
SCORE2-Diabetes Working Group and the ESC Cardiovascular Risk Collaboration. SCORE2-Diabetes: 10-year cardiovascular risk estimation in type 2 diabetes in Europe. Eur. Heart J. 44, 2544–2556 (2023).
Stata file to calculate score2 risk. University of Cambridge Cardiovascular Epidemiology Unit https://www.phpc.cam.ac.uk (2023).
Vickers, A. J., Cronin, A. M., Elkin, E. B. & Gonen, M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med. Inf. Decis. Mak. 8, 53 (2008).
Vickers, A. J., Van Calster, B. & Steyerberg, E. W. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 352, i6 (2016).
Acknowledgements
We acknowledge the contribution of EMIS, which contributes to QResearch and EMIS Health, and the Universities of Nottingham and Oxford for expertise in establishing, developing or supporting the QResearch database. This project involved data derived from patient-level information collected by the NHS. The Hospital Episode Statistics data and civil registration mortality data were used after obtaining permission from NHS England, which retains the copyright to that data. NHS England bears no responsibility for the analysis or interpretation of the data. We are grateful to Drs G. Schiff and A. Singh for their feedback on an earlier draft of this manuscript. There was no external funding. This QResearch database was supported by funds from the John Fell Oxford University Press Research Fund.
Author information
Authors and Affiliations
Contributions
J.H.-C. initiated the study, developed the research question, undertook the literature review, performed data extraction and manipulation, conducted primary data analysis and wrote the first draft of the paper. C.A.C.C. contributed to the refinement of the research question and the design, analysis, interpretation and drafting of the paper. M.B. and R.E.K.R. contributed to the refinement of the research question, data interpretation and the drafting of the paper. K.M.C., A.S. and P.B. contributed to data interpretation and clinical interpretation. J.H.-C. performed the final editing of the paper. All authors commented critically on drafts of the manuscript and approved the final submission.
Corresponding author
Ethics declarations
Competing interests
All authors have completed the International Committee of Medical Journal Editors uniform disclosure form at https://www.icmje.org/coi_disclosure.pdf. J.H.-C. reports grants from the National Institute for Health Research, John Fell Oxford University Press Research Fund, Cancer Research UK (C5255/A18085), Wellcome Institutional Strategic Support Fund (204826/Z/16/Z) and other research councils during the conduct of the study. J.H.-C. is an unpaid director of QResearch, a not-for-profit organization that is a partnership between the University of Oxford and EMIS Health, which supplied the QResearch database used in this work. Until 9 August 2023, J.H.-C. had a 50% shareholding in ClinRisk and co-owned it with her husband, who was an executive director. On 9 August 2023, 100% of the share capital was donated to Endeavour Health Care Charitable Trust, and the company was renamed Endeavour Predict. J.H.-C. is an unpaid consultant to Endeavour Predict, and her husband is a nonexecutive director to cover the transition. The company licenses software both to the private sector and to NHS bodies or bodies that provide services to the NHS (through GP electronic health records providers, pharmacies, hospital providers and other NHS providers). This software implements algorithms (including QRISK3) that were developed from the QResearch database during her time at the University of Nottingham. C.A.C.C. reports receiving personal fees from ClinRisk outside of this work. K.M.C. reports grant funding from the British Heart Foundation and is an academic cofounder, shareholder and the director of Caristo Diagnostics, a University of Oxford cardiac image analysis spin-out company. M.B. received grants paid to her institution from AstraZeneca, Roche, Asthma + Lung UK and Horizon Europe, consulting fees or honoraria paid to her institution from AstraZeneca, Sanofi, GSK and Areteia, and support from Chiesi for attending meetings.
Peer review
Peer review information
Nature Medicine thanks Lisa Pennells and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Michael Basson, in collaboration with the Nature Medicine team.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 CVD (primary outcome) and non-CVD death rates per 1000 by calendar year and month over the full study period in the English derivation cohort in those aged 18–84 years.
The two red spikes on the monthly graph show the non-CVD deaths occurring during the first and second COVID-19 pandemic waves in April 2020 and Jan 2021. Data presented in left panel are mean rates with 95% confidence intervals.
Extended Data Fig. 2 Adjusted hazard ratios for CVD risk for fractional polynomial terms for age and BMI and age interactions in the derivation cohort.
Fractional polynomial terms for adjusted hazard ratios for CVD risk for age, BMI and age interactions in the derivation cohort.
Extended Data Fig. 3 Decision curves for QR4, ASCVD and SCORE2 in people aged 40+ in the England validation cohort using the primary CVD outcome definition.
The greatest net benefit is observed for QR4 followed by ASCVD followed by SCORE2.
Extended Data Fig. 4 Predicted and observed 10-year CVD risks for ASCVD and SCORE2 in the English validation cohort in people aged 40+ using the primary CVD outcome.
Predicted and observed 10-year CVD risks for ASCVD and SCORE2 in the cohorts from England in people aged 40+ using the primary CVD outcome definition.
Supplementary information
Supplementary Information
Supplementary Figs. 1–13 and Tables 1–4.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Hippisley-Cox, J., Coupland, C.A.C., Bafadhel, M. et al. Development and validation of a new algorithm for improved cardiovascular risk prediction. Nat Med (2024). https://doi.org/10.1038/s41591-024-02905-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41591-024-02905-y