My Graduate Years
While I worked on many projects during my graduate studies, my research focused on initialising noisy neural networks. I co-authored multiple papers on this topic, but I described and expanded on that work in my thesis. I give a 30 minute summary of my thesis in the following video.
My thesis - Initialisation of noise-regularised neural networks
The key conclusions of my thesis were:
- one must take regularisation strength into account when initialising neural networks that use noise regularisation;
- using noise-regularisation limits the depth to which one can train neural networks;
- initialisation values much larger than ours lead to poor performing networks;
- no initialisation outperforms ours; thus, our suggested initialisation is a safe default.
Critical initialisation for deep signal propagation in noisy rectifier neural networks
We used mean-field theory to predict how the output of ReLU neural networks that use dropout and Gaussian noise would change due to initialisation. As a result, we found a boundary on the maximum depth that these networks could be trained using those predictions. Finally, we derived an initialisation that should allow one to train networks to the depth boundary.
If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
To validate results from our previous work more thoroughly, we brought the idea of randomised control trials into this paper. This also allowed us to test many different initialisation strategies and compare their performance to our own. We trained over 12 000 neural networks and found that no other initialisation consistently outperformed ours.
