High Dimensional Data

Stream classification methods classify a continuous stream of data as new labelled samples arrive. They often also have to deal with concept drift. This paper focuses on seasonal drift in stream classification, which can be found in many real-world application data sources. Traditional approaches of stream classification consider seasonal drift by including seasonal dummy/indicator variables or building separate models for each season. But these approaches have strong limitations in high-dimensional classification problems, or with complex seasonal patterns. This paper explores how to best handle seasonal drift in the specific context of news article categorization (or classification/tagging), where seasonal drift is overwhelmingly the main type of drift present in the data, and for which the data are high-dimensional. We introduce a novel classifier named Seasonal Averaged One-Dependence Estimators (SAODE), which extends the AODE classifier to handle seasonal drift by including tim...

Manifold learning has been successfully used for finding dominant factors (low-dimensional manifold) in a high-dimensional data set. However, most existing manifold learning algorithms only consider one manifold based on one dissimilarity matrix. For utilizing multiple manifolds, a key question is how different pieces of information can be integrated when multiple measurements are available. Amari proposed α-integration for stochastic model integration, which is a generalized averaging method that includes as a special case arithmetic, geometric, and harmonic averages. In this paper, we propose a new generalized manifold integration algorithm equipped with α-integration, manifold α -integration (MAI). Interestingly, MAI can be shown to be a generalization of other integration methods (that may or may not use manifolds) like kernel fusion or mixture of random walk. Our experimental results also confirm that integration of multiple sources of information on individual manifolds is superior to the use of individual manifolds separately, in tasks including classification and sensorimotor integration.

Contemporary biological technologies produce extremely high-dimensional data sets from which to design classifiers, with 20,000 or more potential features being common place. In addition, sample sizes tend to be small. In such settings, feature selection is an inevitable part of classifier design. Heretofore, there have been a number of comparative studies for feature selection, but they have either considered settings with much smaller dimensionality than those occurring in current bioinformatics applications or constrained their study to a few real data sets. This study compares some basic feature-selection methods in settings involving thousands of features, using both model-based synthetic data and real data. It defines distribution models involving different numbers of markers (useful features) versus non-markers (useless features) and different kinds of relations among the features. Under this framework, it evaluates the performances of feature-selection algorithms for different distribution models and classifiers. Both classification error and the number of discovered markers are computed. Although the results clearly show that none of the considered feature-selection methods performs best across all scenarios, there are some general trends relative to sample size and relations among the features. For instance, the classifier-independent univariate filter methods have similar trends. Filter methods such as the t-test have better or similar performance with wrapper methods for harder problems. This improved performance is usually accompanied with significant peaking. Wrapper methods have better performance when the sample size is sufficiently large. ReliefF, the classifier-independent multivariate filter method, has worse performance than univariate filter methods in most cases; however, ReliefF-based wrapper methods show performance similar to their t-test-based counterparts.

This paper focuses on feature selection for problems dealing with high-dimensional data. We discuss the benefits of adopting a regularized approach with L 1 or L 1–L 2 penalties in two different applications—microarray data analysis in computational biology and object detection in computer vision. We describe general algorithmic aspects as well as architecture issues specific to the two domains. The very promising results obtained show how the proposed approach can be useful in quite different fields of application.

Subspace clustering is an extension of traditional clustering that seeks to find clusters in different subspaces within a dataset. Often in high dimensional data, many dimensions are irrelevant and can mask existing clusters in noisy data. Feature selection removes irrelevant and redundant dimensions by analyzing the entire dataset. Subspace clustering algorithms localize the search for relevant dimensions allowing them to find clusters that exist in multiple, possibly overlapping subspaces. There are two major ...

Dimension reduction and variable selection are two types of effective methods that deal with high-dimensional data. In particular, variable selection techniques are of wide-spread use and essentially consist of individual selection methods and interval selection methods. Given the fact that the vibrational spectra have continuous features of spectral bands, interval selection instead of individual spectral wavelength point selection allows for more stable models and easier interpretation. Numerous methods have been suggested for interval selection recently. Therefore, this paper is devoted to a selective review on interval selection methods with partial least squares (PLS) as the calibration model. We described the algorithms in the five classes: classic methods, penalty-based, sampling-based, correlation-based, and projection-based methods. Finally, we compared and discussed the performances of a subset of these methods on three real-world spectroscopic datasets.

Data compression is the most important step in many signal processing and pattern recogni- tion applications. We come across very high dimensional data in such applications. Before processing of large-dimensional datasets, we need to reduce the dimensions to have lesser storage space and reduced computational complexities while retaining the maximum information. Principal Component Analysis (PCA) is one such technique that

This article introduces the sparse group fused lasso (SGFL) as a statistical framework for segmenting sparse regression models with multivariate time series. To compute solutions of the SGFL, a nonsmooth and nonseparable convex program, we develop a hybrid optimization method that is fast, requires no tuning parameter selection, and is guaranteed to converge to a global minimizer. In numerical experiments, the hybrid method compares favorably to state-of-the-art techniques with respect to computation time and numerical accuracy; benefits are particularly substantial in high dimension. The method's statistical performance is satisfactory in recovering nonzero regression coefficients and excellent in change point detection. An application to air quality data is presented. The hybrid method is implemented in the R package sparseGFL available on the author's Github page.

GAMLSS is a general framework for fitting regression type models where the distribution of the response variable does not have to belong to the exponential family and includes highly skew and kurtotic continuous and discrete distribution. GAMLSS allows all the parameters of the distribution of the response variable to be modelled as linear/non-linear or smooth functions of the explanatory variables.

This PhD thesis explains the recent issue concerning the resolution of high-dimensional problems. We
present methods designed to solve them, and their applications for feature selection problems, in the
data mining eld.
In the rst part of this thesis, we introduce the stakes of solving high-dimensional problems. We mainly
investigate line search methods, because we consider them to be particularly suitable for solving such
problems. Then, we present the methods we developed, based on this principle : CUS, EUS and EM323.
We emphasize, in particular, the very high convergence speed of CUS and EUS, and their simplicity of
implementation. The EM323 method is based on an hybridization between EUS and a one-dimensional
optimization algorithm developed by F. Glover : the 3-2-3 algorithm.We show that the results of EM323
are more accurate, especially for non-separable problems, which are the weakness of line search based
methods.
In the second part, we focus on data mining problems, and especially those concerning microarray data
analysis. The objectives are to classify data and to predict the behavior of new samples. A collaboration
with the Tenon Hospital in Paris allows us to analyze their private breast cancer data. To this end,
we develop an exact method, called -test, enhanced by a method designed to automatically select the
optimal number of variables. In a second time, we develop an heuristic, named ABEUS, based on the
optimization of the DLDA classier performances. The results obtained from publicly available data
show that our methods manage to select very small subsets of variables, which is an important criterion
to avoid overtting.

We study the asymptotic properties of the Adaptive LASSO (adaLASSO) in sparse, high-dimensional, linear time-series models. The adaLASSO is a one-step implementation of the family of folded concave penalized least-squares. We assume that both the number of covariates in the model and the number of candidate variables can increase with the sample size (polynomially or geometrically). In other words, we let the number of candidate variables to be larger than the number of observations. We show the adaLASSO consistently chooses the relevant variables as the number of observations increases (model selection consistency) and has the oracle property, even when the errors are non-Gaussian and conditionally heteroskedastic. This allows the adaLASSO to be applied to a myriad of applications in empirical finance and macroeconomics. A simulation study shows that the method performs well in very general settings with t-distributed and heteroskedastic errors as well with highly correlated regressors. Finally, we consider an application to forecast monthly US inflation with many predictors. The model estimated by the adaLASSO delivers superior forecasts than traditional benchmark competitors such as autoregressive and factor models.

Hypothesis tests in models whose dimension far exceeds the sample size can be formulated much like the classical studentized tests only after the initial bias of estimation is removed successfully. The theory of debiased estimators can be developed in the context of quantile regression models for a fixed quantile value. However, it is frequently desirable to formulate tests based on the quantile regression process, as this leads to more robust tests and more stable confidence sets. Additionally, inference in quantile regression requires estimation of the so called sparsity function, which depends on the unknown density of the error. In this paper we consider a debiasing approach for the uniform testing problem. We develop high-dimensional regression rank scores and show how to use them to estimate the sparsity function, as well as how to adapt them for inference involving the quantile regression process. Furthermore, we develop a Kolmogorov-Smirnov test in a location-shift high-dimensional models and confidence sets that are uniformly valid for many quantile values. The main technical result are the development of a Bahadur representation of the debiasing estimator that is uniform over a range of quantiles and uniform convergence of the quantile process to the Brownian bridge process, which are of independent interest. Simulation studies illustrate finite sample properties of our procedure.

This paper develops robust confidence intervals in high-dimensional and left-censored regression. Type-I censored regression models are extremely common in practice, where a competing event makes the variable of interest unobservable. However, techniques developed for entirely observed data do not directly apply to the censored observations. In this paper, we develop smoothed estimating equations that augment the de-biasing method, such that the resulting estimator is adaptive to censoring and is more robust to the mis-specification of the error distribution. We propose a unified class of robust estimators, including Mallow's, Schweppe's and Hill-Ryan's one-step estima-tor. In the ultra-high-dimensional setting, where the dimensionality can grow exponentially with the sample size, we show that as long as the preliminary estimator converges faster than n −1/4 , the one-step estimator inherits asymp-totic distribution of fully iterated version. Moreover, we show that the size of the residuals of the Bahadur representation matches those of the simple linear models, s 3/4 (log(p ∨ n)) 3/4 /n 1/4 – that is, the effects of censoring asymptoti-cally disappear. Simulation studies demonstrate that our method is adaptive to the censoring level and asymmetry in the error distribution, and does not lose efficiency when the errors are from symmetric distributions. Finally, we apply the developed method to a real data set from the MAQC-II repository that is related to the HIV-1 study.

Cluster analysis divides data into groups (clusters) for the purposes of summarization or improved understanding. For example, cluster analysis has been used to group related documents for browsing, to find genes and proteins that have similar functionality, or as a means of data compression. While clustering has a long history and a large number of clustering techniques have been developed