AutoML-Zero: Evolving Machine Learning Algorithms from Scratch

Machine learning research has advanced in multiple aspects, including model structures and learning methods. Theeffort to automate such research, known asAutoML, has also made significant progress. However, this progresshas largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designedlayers as building blocks—or similarly restrictive search spaces. Our goal is to show thatAutoMLcan go further:it is possible today to automatically discover complete machine learning algorithms just using basic mathematicaloperations as building blocks. We demonstrate this by introducing a novel framework that significantly reduceshuman bias through a generic search space. Despite the vastness of this space, evolutionary search can stilldiscover two-layer neural networks trained by backpropagation. These simple neural networks can then besurpassed by evolving directly on tasks of interest,e.g. CIFAR-10 variants, where modern techniques emerge inthe top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolutionadapts algorithms to different task types:e.g., dropout-like techniques appear when little data is available. Webelieve these preliminary successes in discovering machine learning algorithms from scratch indicate a promisingnew direction for the field.