High dimensional discriminant analysis using sparse covariance estimator

Abstract

High dimensional classification has drawn massive attention due to its increasing application in genetic diagnosis, image or speech recognition and financial analysis. Traditional methods such as Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA), which are optimal Bayes classifiers under normality assumption, sometimes fail in high dimensional space where the number of variables is considerably greater than the sample size, and thus it is impossible to obtain a good estimation of the covariance matrix by using the conventional empirical estimator. An alternative approach is Naive Bayes which instead assumes all features are independent. Although independence is a critical assumption, it surprisingly does work well in many practical cases. Inspired by the success of Naive Bayes, we aim to find a balance between Naive Bayes and LDA. Hence, it is reasonable to assume only few correlations between features exist in high dimension so that we can take advantage of the sparsity and get a better covariance estimator. The main contribution of this thesis is that we improved the conventional LDA under the sparsity assumption by replacing the empirical covariance estimator with a sparse one. We also review various classification methods specific for high dimensional space. We compared our approach with some of these methods available in R with both simulation and two real data sets and the result showed that our method outperformed many baselines.