Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Correspondence
  • Published:

Reply to ‘Inflated performance measures in enhancer–promoter interaction-prediction methods’

Nature Genetics volume 51, pages 1198–1200 (2019)Cite this article

Subjects

First, let us elaborate on how enhancer–promoter interaction data can violate the IID assumption. The unit of measurement here is an enhancer–promoter pair, upon which many genomic features are observed. Dependence arises from (i) promoters interacting with more than one enhancer, (ii) enhancers interacting with more than one promoter and (iii) genomic features, such as broad histone modification peaks, that overlap more than one enhancer–promoter pair, primarily because of proximal clusters of regulatory elements. In the published TargetFinder training data, promiscuous promoters interacting with many enhancers are the major contributor to dependence between pairs. Regarding the identically distributed assumption, we note that genomic loci have different types and levels of genomic features, as do different cell lines. Consequently, enhancer–promoter pairs that are independent (for example, on different chromosomes) may not be identically distributed. Thus, both parts of the IID assumption may be violated, and data more consistent with one part of the assumption may be less consistent with the other.

Violations of the IID assumption are important for several reasons. At the most fundamental level, the mathematics underlying statistical models—including parameter estimators, hypothesis tests and optimality of cross-validation for performance evaluation—depends on the IID assumption. Furthermore, when evaluating a machine-learning model by using standard k-fold CV, in which samples are randomly shuffled before repeated splitting into training and test sets, biological correlation causes information to cross the training and test divide. This process can inflate performance metrics and obscure over-fitting, thus resulting in models that may not generalize to new contexts. These problems are serious when the primary aim of the machine learning is to provide a tool for novel predictions (for example, in silico Hi-C). When the goal is to learn something about relationships between variables and to shed light on mechanisms underlying patterns in the data, as in our TargetFinder paper, the importance of over-fitting and inflated performance depends on whether these problems obscure the biological signal.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Corrected performance across and within cell types.
Fig. 2: Corrected performance within cell types only.

References

  1. Cao, F. & Fullwood, M. J. Nat. Genet. https://doi.org/10.1038/s41588-019-0434-7 (2019).

  2. Xi, W. & Beer, M. A. PLoS Comput. Biol. 14, e1006625 (2018).

    Article  Google Scholar 

  3. Whalen, S., Truty, R. M. & Pollard, K. S. Nat. Genet. 48, 488–496 (2016).

    Article  CAS  Google Scholar 

  4. Dundar, M., Krishnapuram, B., Bi, J. & Rao, R.B. in Proc. 20th International Joint Conference on Artificial Intelligence 756–761 (IJCAI, 2007).

  5. Kwasnieski, J. C., Fiore, C., Chaudhari, H. G. & Cohen, B. A. Genome Res. 24, 1595–1602 (2014).

    Article  CAS  Google Scholar 

  6. Cao, Q. et al. Nat. Genet. 49, 1428–1436 (2017).

    Article  CAS  Google Scholar 

  7. Kelley, D. R. et al. Genome Res. 28, 739–750 (2018).

    Article  CAS  Google Scholar 

  8. Hait, T. A., Amar, D., Shamir, R. & Elkon, R. Genome Biol. 19, 56 (2018).

    Article  Google Scholar 

  9. Schreiber, J., Libbrecht, M., Bilmes, J. & Noble, W.S. Preprint at bioRxiv https://doi.org/10.1101/103614 (2018).

  10. Chicco, D. BioData Min. 10, 35 (2017).

    Article  Google Scholar 

  11. Arthur, R. K., An, N., Khan, S. & McNerney, M. E. Nucleic Acids Res. 45, 6350–6361 (2017).

    Article  CAS  Google Scholar 

  12. Weintraub, A. S. et al. Cell 171, 1573–1588.e28 (2017).

    Article  CAS  Google Scholar 

  13. Tan, J. Y. et al. Cell Rep. 18, 2280–2288 (2017).

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Gladstone Institutes, San Francisco, CA, USA

    Sean Whalen & Katherine S. Pollard

  2. Department of Epidemiology and Biostatistics, Institute for Human Genetics, Quantitative Biology Institute, and Institute for Computational Health Sciences, University of California, San Francisco, San Francisco, CA, USA

    Katherine S. Pollard

  3. Chan-Zuckerberg Biohub, San Francisco, CA, USA

    Katherine S. Pollard

Authors
  1. Sean Whalen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Katherine S. Pollard
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katherine S. Pollard.

Ethics declarations

Competing interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Whalen, S., Pollard, K.S. Reply to ‘Inflated performance measures in enhancer–promoter interaction-prediction methods’. Nat Genet 51, 1198–1200 (2019). https://doi.org/10.1038/s41588-019-0473-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41588-019-0473-0

This article is cited by

Search

Advanced search

Quick links

Nature Briefing AI and Robotics

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing: AI and Robotics