# On the Expected Complexity of Maxout Networks

@article{Tseran2021OnTE, title={On the Expected Complexity of Maxout Networks}, author={Hanna Tseran and Guido Mont{\'u}far}, journal={ArXiv}, year={2021}, volume={abs/2107.00379} }

Learning with neural networks relies on the complexity of the representable functions, but more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as a complexity measure, recent works have shown that the practical complexity of deep ReLU networks is often far from the theoretical maximum. In this work we show that this phenomenon also occurs in networks with maxout (multi-argument) activation functions and… Expand

#### Figures and Tables from this paper

#### References

SHOWING 1-10 OF 49 REFERENCES

Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

- Computer Science, Mathematics
- ICLR
- 2020

Instead of just counting the number of the linear regions, this paper studies their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. Expand

Complexity of Linear Regions in Deep Networks

- Mathematics, Computer Science
- ICML
- 2019

The theory suggests that, even after training, the number of linear regions is far below exponential, an intuition that matches the empirical observations and concludes that the practical expressivity of neural networks is likely far below that of the theoretical maximum, and this gap can be quantified. Expand

On the Number of Linear Regions of Deep Neural Networks

- Computer Science, Mathematics
- NIPS
- 2014

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep… Expand

Bounding and Counting Linear Regions of Deep Neural Networks

- Computer Science, Mathematics
- ICML
- 2018

The results indicate that a deep rectifier network can only have more linear regions than every shallow counterpart with same number of neurons if that number exceeds the dimension of the input. Expand

Deep ReLU Networks Have Surprisingly Few Activation Patterns

- Computer Science, Mathematics
- NeurIPS
- 2019

This work shows empirically that the average number of activation patterns for ReLU networks at initialization is bounded by the total number of neurons raised to the input dimension, and suggests that realizing the full expressivity of deep networks may not be possible in practice, at least with current methods. Expand

Maxout Networks

- Computer Science, Mathematics
- ICML
- 2013

A simple new model called maxout is defined designed to both facilitate optimization by dropout and improve the accuracy of dropout's fast approximate model averaging technique. Expand

A General Computational Framework to Measure the Expressiveness of Complex Networks Using a Tighter Upper Bound of Linear Regions

- Computer Science, Mathematics
- ArXiv
- 2020

This work proposes ageneral computational approach to compute a tight upper bound of regions number for theoretically any network structures (e.g. DNN with all kind of skip connec-tions and residual structures). Expand

Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks

- Computer Science, Mathematics
- ICML
- 2021

Tight bounds on the smallest eigenvalue of NTK matrices for deep ReLU nets are provided, both in the limiting case of infinite widths and for finite widths. Expand

Understanding the difficulty of training deep feedforward neural networks

- Computer Science, Mathematics
- AISTATS
- 2010

The objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks, to better understand these recent relative successes and help design better algorithms in the future. Expand

A Framework for the Construction of Upper Bounds on the Number of Affine Linear Regions of ReLU Feed-Forward Neural Networks

- Computer Science, Mathematics
- IEEE Transactions on Information Theory
- 2019

By using explicit formulas for a Jordan-like decomposition of the involved matrices, the framework to derive upper bounds on the number of regions that feed-forward neural networks with ReLU activation functions are affine linear on is presented. Expand