Large Sample Covariance Matrices And High Dimensional Data Analysis

Author : Jianfeng Yao
Publisher : Cambridge University Press
ISBN 13 : 9781107065178
Total Pages : 0 pages
Book Rating : 4.78/5 ( download)

DOWNLOAD NOW!

Download or read book Large Sample Covariance Matrices and High-Dimensional Data Analysis written by Jianfeng Yao and published by Cambridge University Press. This book was released on 2015-03-26 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: High-dimensional data appear in many fields, and their analysis has become increasingly important in modern statistics. However, it has long been observed that several well-known methods in multivariate analysis become inefficient, or even misleading, when the data dimension p is larger than, say, several tens. A seminal example is the well-known inefficiency of Hotelling's T2-test in such cases. This example shows that classical large sample limits may no longer hold for high-dimensional data; statisticians must seek new limiting theorems in these instances. Thus, the theory of random matrices (RMT) serves as a much-needed and welcome alternative framework. Based on the authors' own research, this book provides a first-hand introduction to new high-dimensional statistical methods derived from RMT. The book begins with a detailed introduction to useful tools from RMT, and then presents a series of high-dimensional problems with solutions provided by RMT methods.

Author : Aygul Zagidullina
Publisher : Springer Nature
ISBN 13 : 3030800652
Total Pages : 123 pages
Book Rating : 4.59/5 ( download)

DOWNLOAD NOW!

Download or read book High-Dimensional Covariance Matrix Estimation written by Aygul Zagidullina and published by Springer Nature. This book was released on 2021-10-29 with total page 123 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book presents covariance matrix estimation and related aspects of random matrix theory. It focuses on the sample covariance matrix estimator and provides a holistic description of its properties under two asymptotic regimes: the traditional one, and the high-dimensional regime that better fits the big data context. It draws attention to the deficiencies of standard statistical tools when used in the high-dimensional setting, and introduces the basic concepts and major results related to spectral statistics and random matrix theory under high-dimensional asymptotics in an understandable and reader-friendly way. The aim of this book is to inspire applied statisticians, econometricians, and machine learning practitioners who analyze high-dimensional data to apply the recent developments in their work.

Author : Mohsen Pourahmadi
Publisher : John Wiley & Sons
ISBN 13 : 1118034295
Total Pages : 204 pages
Book Rating : 4.93/5 ( download)

DOWNLOAD NOW!

Download or read book High-Dimensional Covariance Estimation written by Mohsen Pourahmadi and published by John Wiley & Sons. This book was released on 2013-06-24 with total page 204 pages. Available in PDF, EPUB and Kindle. Book excerpt: Methods for estimating sparse and large covariance matrices Covariance and correlation matrices play fundamental roles in every aspect of the analysis of multivariate data collected from a variety of fields including business and economics, health care, engineering, and environmental and physical sciences. High-Dimensional Covariance Estimation provides accessible and comprehensive coverage of the classical and modern approaches for estimating covariance matrices as well as their applications to the rapidly developing areas lying at the intersection of statistics and machine learning. Recently, the classical sample covariance methodologies have been modified and improved upon to meet the needs of statisticians and researchers dealing with large correlated datasets. High-Dimensional Covariance Estimation focuses on the methodologies based on shrinkage, thresholding, and penalized likelihood with applications to Gaussian graphical models, prediction, and mean-variance portfolio management. The book relies heavily on regression-based ideas and interpretations to connect and unify many existing methods and algorithms for the task. High-Dimensional Covariance Estimation features chapters on: Data, Sparsity, and Regularization Regularizing the Eigenstructure Banding, Tapering, and Thresholding Covariance Matrices Sparse Gaussian Graphical Models Multivariate Regression The book is an ideal resource for researchers in statistics, mathematics, business and economics, computer sciences, and engineering, as well as a useful text or supplement for graduate-level courses in multivariate analysis, covariance estimation, statistical learning, and high-dimensional data analysis.

Author : Arup Bose
Publisher : CRC Press
ISBN 13 : 1351398156
Total Pages : 272 pages
Book Rating : 4.52/5 ( download)

DOWNLOAD NOW!

Download or read book Large Covariance and Autocovariance Matrices written by Arup Bose and published by CRC Press. This book was released on 2018-07-03 with total page 272 pages. Available in PDF, EPUB and Kindle. Book excerpt: Large Covariance and Autocovariance Matrices brings together a collection of recent results on sample covariance and autocovariance matrices in high-dimensional models and novel ideas on how to use them for statistical inference in one or more high-dimensional time series models. The prerequisites include knowledge of elementary multivariate analysis, basic time series analysis and basic results in stochastic convergence. Part I is on different methods of estimation of large covariance matrices and auto-covariance matrices and properties of these estimators. Part II covers the relevant material on random matrix theory and non-commutative probability. Part III provides results on limit spectra and asymptotic normality of traces of symmetric matrix polynomial functions of sample auto-covariance matrices in high-dimensional linear time series models. These are used to develop graphical and significance tests for different hypotheses involving one or more independent high-dimensional linear time series. The book should be of interest to people in econometrics and statistics (large covariance matrices and high-dimensional time series), mathematics (random matrices and free probability) and computer science (wireless communication). Parts of it can be used in post-graduate courses on high-dimensional statistical inference, high-dimensional random matrices and high-dimensional time series models. It should be particularly attractive to researchers developing statistical methods in high-dimensional time series models. Arup Bose is a professor at the Indian Statistical Institute, Kolkata, India. He is a distinguished researcher in mathematical statistics and has been working in high-dimensional random matrices for the last fifteen years. He has been editor of Sankhyā for several years and has been on the editorial board of several other journals. He is a Fellow of the Institute of Mathematical Statistics, USA and all three national science academies of India, as well as the recipient of the S.S. Bhatnagar Award and the C.R. Rao Award. His first book Patterned Random Matrices was also published by Chapman & Hall. He has a forthcoming graduate text U-statistics, M-estimates and Resampling (with Snigdhansu Chatterjee) to be published by Hindustan Book Agency. Monika Bhattacharjee is a post-doctoral fellow at the Informatics Institute, University of Florida. After graduating from St. Xavier's College, Kolkata, she obtained her master’s in 2012 and PhD in 2016 from the Indian Statistical Institute. Her thesis in high-dimensional covariance and auto-covariance matrices, written under the supervision of Dr. Bose, has received high acclaim.

Author : Yilei Wu
Publisher :
ISBN 13 :
Total Pages : 169 pages
Book Rating : 4.73/5 ( download)

DOWNLOAD NOW!

Download or read book High-dimensional Discriminant Analysis and Covariance Matrix Estimation written by Yilei Wu and published by . This book was released on 2017 with total page 169 pages. Available in PDF, EPUB and Kindle. Book excerpt: Statistical analysis in high-dimensional settings, where the data dimension p is close to or larger than the sample size n, has been an intriguing area of research. Applications include gene expression data analysis, financial economics, text mining, and many others. Estimating large covariance matrices is an essential part of high-dimensional data analysis because of the ubiquity of covariance matrices in statistical procedures. The estimation is also a challenging part, since the sample covariance matrix is no longer an accurate estimator of the population covariance matrix in high dimensions. In this thesis, a series of matrix structures, that facilitate the covariance matrix estimation, are studied. Firstly, we develop a set of innovative quadratic discriminant rules by applying the compound symmetry structure. For each class, we construct an estimator, by pooling the diagonal elements as well as the off-diagonal elements of the sample covariance matrix, and substitute the estimator for the covariance matrix in the normal quadratic discriminant rule. Furthermore, we develop a more general rule to deal with nonnormal data by incorporating an additional data transformation. Theoretically, as long as the population covariance matrices loosely conform to the compound symmetry structure, our specialized quadratic discriminant rules enjoy low asymptotic classification error. Computationally, they are easy to implement and do not require large-scale mathematical programming. Then, we generalize the compound symmetry structure by considering the assumption that the population covariance matrix (or equivalently its inverse, the precision matrix) can be decomposed into a diagonal component and a low-rank component. The rank of the low-rank component governs to what extent the decomposition can simplify the covariance/precision matrix and reduce the number of unknown parameters. In the estimation, this rank can either be pre-selected to be small or controlled by a penalty function. Under moderate conditions on the population covariance/precision matrix itself and on the penalty function, we prove some consistency results for our estimator. A blockwise coordinate descent algorithm, which iteratively updates the diagonal component and the low-rank component, is then proposed to obtain the estimator in practice. In the end, we consider jointly estimating large covariance matrices of multiple categories. In addition to the aforementioned diagonal and low-rank matrix decomposition, it is further assumed that there is some common matrix structure shared across the categories. We assume that the population precision matrix of category k can be decomposed into a diagonal matrix D, a shared low-rank matrix L, and a category-specific low-rank matrix Lk. The assumption can be understood under the framework of factor models -- some latent factors affect all categories alike while others are specific to only one of these categories. We propose a method that jointly estimates the precision matrices (therefore, the covariance matrices) -- D and L are estimated with the entire dataset whereas Lk is estimated solely with the data of category k. An AIC-type penalty is applied to encourage the decomposition, especially the shared component. Under certain conditions on the population covariance matrices, some consistency results are developed for the estimators. The performances in finite dimensions are shown through numerical experiments. Using simulated data, we demonstrate certain advantages of our methods over existing ones, in terms of classification error for the discriminant rules and Kullback--Leibler loss for the covariance matrix estimators. The proposed methods are also applied to real life datasets, including microarray data, stock return data and text data, to perform tasks, such as distinguishing normal from diseased tissues, portfolio selection and classifying webpages.

Author : Zhidong Bai
Publisher : Springer Science & Business Media
ISBN 13 : 1441906614
Total Pages : 560 pages
Book Rating : 4.18/5 ( download)

DOWNLOAD NOW!

Download or read book Spectral Analysis of Large Dimensional Random Matrices written by Zhidong Bai and published by Springer Science & Business Media. This book was released on 2009-12-10 with total page 560 pages. Available in PDF, EPUB and Kindle. Book excerpt: The aim of the book is to introduce basic concepts, main results, and widely applied mathematical tools in the spectral analysis of large dimensional random matrices. The core of the book focuses on results established under moment conditions on random variables using probabilistic methods, and is thus easily applicable to statistics and other areas of science. The book introduces fundamental results, most of them investigated by the authors, such as the semicircular law of Wigner matrices, the Marcenko-Pastur law, the limiting spectral distribution of the multivariate F matrix, limits of extreme eigenvalues, spectrum separation theorems, convergence rates of empirical distributions, central limit theorems of linear spectral statistics, and the partial solution of the famous circular law. While deriving the main results, the book simultaneously emphasizes the ideas and methodologies of the fundamental mathematical tools, among them being: truncation techniques, matrix identities, moment convergence theorems, and the Stieltjes transform. Its treatment is especially fitting to the needs of mathematics and statistics graduate students and beginning researchers, having a basic knowledge of matrix theory and an understanding of probability theory at the graduate level, who desire to learn the concepts and tools in solving problems in this area. It can also serve as a detailed handbook on results of large dimensional random matrices for practical users. This second edition includes two additional chapters, one on the authors' results on the limiting behavior of eigenvectors of sample covariance matrices, another on applications to wireless communications and finance. While attempting to bring this edition up-to-date on recent work, it also provides summaries of other areas which are typically considered part of the general field of random matrix theory.

Author : Roman Vershynin
Publisher : Cambridge University Press
ISBN 13 : 1108415199
Total Pages : 299 pages
Book Rating : 4.94/5 ( download)

DOWNLOAD NOW!

Download or read book High-Dimensional Probability written by Roman Vershynin and published by Cambridge University Press. This book was released on 2018-09-27 with total page 299 pages. Available in PDF, EPUB and Kindle. Book excerpt: An integrated package of powerful probabilistic tools and key applications in modern mathematical data science.

Author : S. Ejaz Ahmed
Publisher : Springer
ISBN 13 : 3319415735
Total Pages : 390 pages
Book Rating : 4.34/5 ( download)

DOWNLOAD NOW!

Download or read book Big and Complex Data Analysis written by S. Ejaz Ahmed and published by Springer. This book was released on 2017-03-21 with total page 390 pages. Available in PDF, EPUB and Kindle. Book excerpt: This volume conveys some of the surprises, puzzles and success stories in high-dimensional and complex data analysis and related fields. Its peer-reviewed contributions showcase recent advances in variable selection, estimation and prediction strategies for a host of useful models, as well as essential new developments in the field. The continued and rapid advancement of modern technology now allows scientists to collect data of increasingly unprecedented size and complexity. Examples include epigenomic data, genomic data, proteomic data, high-resolution image data, high-frequency financial data, functional and longitudinal data, and network data. Simultaneous variable selection and estimation is one of the key statistical problems involved in analyzing such big and complex data. The purpose of this book is to stimulate research and foster interaction between researchers in the area of high-dimensional data analysis. More concretely, its goals are to: 1) highlight and expand the breadth of existing methods in big data and high-dimensional data analysis and their potential for the advancement of both the mathematical and statistical sciences; 2) identify important directions for future research in the theory of regularization methods, in algorithmic development, and in methodologies for different application areas; and 3) facilitate collaboration between theoretical and subject-specific researchers.

Author : Sumeet Dua
Publisher : CRC Press
ISBN 13 : 0849328012
Total Pages : 351 pages
Book Rating : 4.15/5 ( download)

DOWNLOAD NOW!

Download or read book Data Mining for Bioinformatics written by Sumeet Dua and published by CRC Press. This book was released on 2012-11-06 with total page 351 pages. Available in PDF, EPUB and Kindle. Book excerpt: Covering theory, algorithms, and methodologies, as well as data mining technologies, Data Mining for Bioinformatics provides a comprehensive discussion of data-intensive computations used in data mining with applications in bioinformatics. It supplies a broad, yet in-depth, overview of the application domains of data mining for bioinformatics to help readers from both biology and computer science backgrounds gain an enhanced understanding of this cross-disciplinary field. The book offers authoritative coverage of data mining techniques, technologies, and frameworks used for storing, analyzing, and extracting knowledge from large databases in the bioinformatics domains, including genomics and proteomics. It begins by describing the evolution of bioinformatics and highlighting the challenges that can be addressed using data mining techniques. Introducing the various data mining techniques that can be employed in biological databases, the text is organized into four sections: Supplies a complete overview of the evolution of the field and its intersection with computational learning Describes the role of data mining in analyzing large biological databases—explaining the breath of the various feature selection and feature extraction techniques that data mining has to offer Focuses on concepts of unsupervised learning using clustering techniques and its application to large biological data Covers supervised learning using classification techniques most commonly used in bioinformatics—addressing the need for validation and benchmarking of inferences derived using either clustering or classification The book describes the various biological databases prominently referred to in bioinformatics and includes a detailed list of the applications of advanced clustering algorithms used in bioinformatics. Highlighting the challenges encountered during the application of classification on biological databases, it considers systems of both single and ensemble classifiers and shares effort-saving tips for model selection and performance estimation strategies.

Author : Robert C. Qiu
Publisher : John Wiley & Sons
ISBN 13 : 1118716809
Total Pages : 630 pages
Book Rating : 4.09/5 ( download)

DOWNLOAD NOW!

Download or read book Smart Grid using Big Data Analytics written by Robert C. Qiu and published by John Wiley & Sons. This book was released on 2017-01-23 with total page 630 pages. Available in PDF, EPUB and Kindle. Book excerpt: This book is aimed at students in communications and signal processing who want to extend their skills in the energy area. It describes power systems and why these backgrounds are so useful to smart grid, wireless communications being very different to traditional wireline communications.