LassoNet: A Neural Network with Feature Sparsity

Abstract: Much work has been done recently to make neural networks more interpretable, and one obvious approach is to arrange for the network to use only a subset of the available features. In linear models, Lasso (or $\ell_1$-regularized) regression assigns zero weights to the most irrelevant or redundant features, and is widely used in data science. However the Lasso only applies to linear models. Here we introduce LassoNet, a neural network framework with global feature selection. Our approach enforces a hierarchy: specifically a feature can participate in a hidden unit only if its linear representative is active. Unlike other approaches to feature selection for neural nets, our method uses a modified objective function with constraints, and so integrates feature selection with the parameter learning directly. As a result, it delivers an entire regularization path of solutions with a range of feature sparsity. On systematic experiments, LassoNet significantly outperforms state-of-the-art methods for feature selection and regression. The LassoNet method uses projected proximal gradient descent, and generalizes directly to deep networks. It can be implemented by adding just a few lines of code to a standard neural network.

probabilitystatistics theory

Audience: researchers in the topic

( video )

Probability & Statistics (IST-CEMAT, FC-CEAUL, ULisbon)

Series comments: To receive the series announcements, which include the
Zoom access password*, please register in
math.tecnico.ulisboa.pt/seminars/pe/index.php?action=subscribe#subscribe
*the last announcement for a seminar is sent 2 hours before the seminar.

ProbStat@IST video channel: portal.educast.fccn.pt/videos?c=6781