Report From the Edinburgh Deep Learning Workshop
I spent yesterday at the very interesting Second Edinburgh Deep Learning Workshop. Edinburgh is a short train ride from Stirling so these events are very convenient.
I particularly enjoyed Rich Caruana's talk about reproducing deep network functionality in shallow networks (i,e, standard 1 hidden layer MLPs). He has a paper on the subject
here. The basic premise is based on an idea called model compression, which was originally used to train a simple neural network to mimic the behaviour of an ensemble of many different classification techniques. By training the network to mimic the behaviour of the ensemble, it is possible to gain the performance benefit of that ensemble without the cost of making and combining a great many classifications. In this work, Rich and his team take a large deep network that has learned to perform a classification task very well and use it to generate training data for a simple MLP. They found that the simple MLP was able to perform as well as the deep network once training was complete. For me (and, I'm sure, many others) this is a very interesting result. I'll certainly be adding it to the content I cover in the analytics course on my Big Data MSc.
The event was sponsored by the Scottish Informatics & Computing Science Alliance (
SICSA), which is a fantastic way of bringing together the staff and students in computing across the Scottish universities. Stirling students benefit from access to some very good events because of our membership of SICSA.
