Data-Driven Mathematical Approach for Removing Rare Features in Zero-Inflated Datasets
01:41 Share

Link to bioRxiv paper: http://biorxiv.org/cgi/content/short/2023.03.11.532198v1?rss=1

Authors: Ortiz-Velez, A. N., Kelley, S. T.

Abstract: Sparse feature tables, in which many features are present in very few samples, are common in big biological data (e.g., metagenomics, transcriptomics). Ignoring the problem of zero-inflation can result in biased statistical estimates and decrease power in downstream analyses. Zeros are also a particular issue for compositional data analysis using log-ratios since the log of zero is undefined. Researchers typically deal with zero-inflated data by removing low frequency features, but the thresholds for removal differ markedly between studies with little or no justification. Here, we present CurvCut, a data-driven mathematical approach to zero-inflated feature removal based on curvature analysis of a 'ball rolling down a hill', where the hill is a histogram of feature distribution. These histograms typically contain a point of regime change, a discontinuity with a sharp change in the characteristics of the distribution, that can be used as a cutoff point for low frequency feature removal that considers the data-specific nature of the feature distribution. Our results show that CurvCut works well across a variety of biological data types, including ones with both right- and left-skewed feature distributions, and rapidly generates clear visual results allowing researchers to select data-appropriate cutoffs for feature removal.

Copy rights belong to original authors. Visit the link for more info

Podcast created by Paper Player, LLC