Arising from: Katz E M and Kattan M W (2005) Nat Clin Pract Oncol 2: 482–483

We thank Dr Katz and Dr Kattan1 for their support of our efforts to improve tumor marker prognostic study reporting, and we would like to address the concerns they raised about points 16 and 17 of the REMARK guidelines2. Katz and Kattan state that the guidelines “appear to assume that the new marker’s prognostic value will be inferred from its coefficient in a multivariate regression model”. We fully agree that when the body of evidence for a new marker develops to the point that one wishes to definitively evaluate its clinical utility, a coefficient in a multivariate regression model is seriously lacking in interpretability. Previous authors, including Kattan3 and Pepe4 have clearly described the limitations of hazard ratios and odds ratios for assessing the performance of markers. It was never our intention to suggest that the REMARK guidelines provide metrics for judging the clinical utility of a marker or to suggest that they outline a strategy for validation of clinical utility of a marker. The request in our point 17 is only an initial step to address the question “does the new marker enhance our ability to predict?” In many published reports, the authors do not provide this information. The reality is that for the majority of tumor markers the body of evidence consists of multiple retrospective studies carried out with often with no written protocol for design or analysis, no pre-defined hypotheses, and no eligibility criteria. The result may be a collection of studies for which the patients are very clinically heterogeneous and may have received an assortment of treatments (including no treatment). Whether one were to estimate an adjusted hazard or odds ratio from a multivariate model or a measure of improvement in predictive accuracy as suggested in the Viewpoint by Katz and Kattan, interpretability for purposes of assessing clinical utility would be limited.

The intention of the REMARK guidelines is to encourage more complete reporting so that the differences across studies could be more fully appreciated and so that there would be a greater understanding of the context in which the conclusions might apply. For these purposes, we feel that reporting of a hazard ratio or odds ratio along with a confidence interval adequately addresses the very basic question of whether there is statistical evidence for an association between the new marker and the outcome, with or without adjustments for standard clinical factors. Of course, this assumes that there has not been extensive model fitting and subset analyses that would inflate the chances of spuriously significant findings. If one were to be in the fortunate, but rare, position of analyzing a study that had been carefully designed to answer specific questions on a clearly defined patient population using specified analysis methods, it would be our preference, as we understand it would be Katz and Kattan’s preference, to provide a measure of a marker’s prognostic value that is more interpretable than a hazard ratio or odds ratio. We were careful in point #16 of the REMARK guidelines to provide hazard ratio as only an example, and we intentionally refer more vaguely to “estimated effects” in point #17 to allow greater flexibility depending on the type of study.

One of the greatest challenges in producing the REMARK guidelines was striking an appropriate balance between broad applicability and detailed recommendations of reporting. Even broad recommendations for specific design or analysis strategies will require more investigations and discussions about advantages and disadvantages of the large number of strategies available. We are currently preparing an explanatory document that will expand upon numerous issues such as those raised by Katz and Kattan. Our expectations from the start were that whatever recommendations were published would generate substantial discussion. Such discussion can serve a very valuable educational role and we heartily welcome it.