A curated list of awesome research papers, datasets, and tools for applying machine learning techniques to compilers and program optimisation.
- Papers
- Survey
- Iterative Compilation and Compiler Option Tuning
- Instruction-level Optimisation
- Parallelism Mapping and Task Scheduling
- Languages and Compilation
- Auto-tuning and Design Space Exploration
- Code Size Reduction
- Cost and Performance Models
- Domain-specific Optimisation
- Learning Program Representation
- Enabling ML in Compilers and Systems Optimisation
- Memory/Cache Modeling/Analysis
- Books
- Talks and Tutorials
- Software
- Benchmarks and Datasets
- Conferences
- Journals
- How to Contribute
Machine Learning in Compiler Optimisation - Zheng Wang and Michael O'Boyle, Proceedings of the IEEE, 2018
A survey on compiler autotuning using machine learning - Amir H. Ashouri, William Killian, John Cavazos, Gianluca Palermo, and Cristina Silvano, ACM Computing Surveys (CSUR), 2018
A Taxonomy of ML for Systems Problems - Martin Maas, IEEE Micro, 2020
The Deep Learning Compiler: A Comprehensive Survey - Mingzhen Li, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Lin Gan, Guangwen Yang, Depei Qian, IEEE Transactions on Parallel and Distributed Systems, 2021
SRTuner: Effective Compiler Optimization Customization by Exposing Synergistic Relations - Sunghyun Park, Salar Latifi, Yongjun Park, Armand Behroozi, Byungsoo Jeon, Scott Mahlke. CGO 2022.
Iterative Compilation Optimization Based on Metric Learning and Collaborative Filtering - Hongzhi Liu, Jie Luo, Ying Li, Zhonghai Wu. ACM TACO 2022.
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020 - Ryan Turner, David Eriksson, Michael McCourt, Juha Kiili, Eero Laaksonen, Zhen Xu, Isabelle Guyon. arXiv 2021.
Bliss: auto-tuning complex applications using a pool of diverse lightweight learning models - RB Roy, T Patel, V Gadepally, D Tiwari. PLDI 2021.
Efficient Compiler Autotuning via Bayesian Optimization - Junjie Chen, Ningxin Xu, Peiqi Chen, Hongyu Zhang. ICSE 2021.
Customized Monte Carlo Tree Search for LLVM/Polly's Composable Loop Optimization Transformations - Jaehoon Koo, Prasanna Balaprakash, Michael Kruse, Xingfu Wu, Paul Hovland, Mary Hall. Arxiv.org, 2021.
FuncyTuner: Auto-tuning Scientific Applications With Per-loop Compilation - Tao Wang, Nikhil Jain, David Beckingsale, David Böhme, Frank Mueller, Todd Gamblin. ICPP 2019.
Micomp: Mitigating the compiler phase-ordering problem using optimization sub-sequences and machine learning - Amir H. Ashouri, Andrea Bignoli, Gianluca Palermo, Cristina Silvano, Sameer Kulkarni, and John Cavazos. ACM Transactions on Architecture and Code Optimization (TACO) 2017.
Iterative Schedule Optimization for Parallelization in the Polyhedron Model - Stefan Ganser, Armin Grösslinger, Norbert Siegmund, Sven Apel, and Christian Lengauer. ACM Transactions on Architecture and Code Optimization (TACO), 2017.
Learning to superoptimize programs - Rudy Bunel, Alban Desmaison, M. Pawan Kumar, Philip H.S. Torr, Pushmeet Kohlim. ICLR 2017
Multi-objective Exploration for Practical Optimization Decisions in Binary Translation - Sunghyun Park, Youfeng Wu, Janghaeng Lee, Amir Aupov, and Scott Mahlke. ACM Transactions on Embedded Computing Systems (TECS), 2019.
Learning to schedule straight-line code - J. Eliot B. Moss, Paul E. Utgoff, John Cavazos, Doina Precup, Darko Stefanovic, Carla E. Brodley, and David Scheeff. NeurIPS 1998.
The Droplet Search Algorithm for Kernel Scheduling - Michael Canesche, Vanderson M. Rosario, Edson Borin, Fernando Magno Quintão Pereira. ACM TACO 2024
BaCO: A Fast and Portable Bayesian Compiler Optimization Framework - Erik Hellsten, Artur Souza, Johannes Lenfers, Rubens Lacouture, Olivia Hsu, Adel Ejjeh, Fredrik Kjolstad, Michel Steuwer, Kunle Olukotun, Luigi Nardi. ASPLOS 2024.
Glimpse: Mathematical Embedding of Hardware Specification for Neural Compilation - Byung Hoon Ahn, Sean Kinzer, Hadi Esmaeilzadeh. DAC 2022.
Ansor: Generating High-Performance Tensor Programs for Deep Learning - Lianmin Zheng, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, Jun Yang, Danyang Zhuo, Koushik Sen, Joseph E. Gonzalez, Ion Stoica. OSDI 2020. (slides, presentation)
Bandit-based optimization on graphs with application to library performance tuning - Frédéric De Mesmay, Arpad Rimmel, Yevgen Voronenko, and Markus Püschel. ICML 2009.
Learning Nonlinear Loop Invariants with Gated Continuous Logic Networks - J. Yao, G. Ryan, J. Wong, S. Jana, and R. Gu. PLDI 2020.
(De/Re)-Composition of Data-Parallel Computations via Multi-Dimensional Homomorphisms - Ari Rasch, TOPLAS 2024.
Predicting new workload or CPU performance by analyzing public datasets - Yu Wang, Victor Lee, Gu-Yeon Wei, and David Brooks. ACM Transactions on Architecture and Code Optimization (TACO), 2019.
-
Program Representations for Predictive Compilation: State of Affairs in the Early 20's - Anderson Faustino da Silva, Edson Borin, Fernando Magno Quintao Pereira, Nilton Luiz Queiroz Junior and Otavio Oliveira Napoli. JCL 2022. Code and Data
IR2VEC: LLVM IR Based Scalable Program Embeddings - S. VenkataKeerthy, Rohit Aggarwal, Shalini Jain, Maunendra Sankar Desarkar, Ramakrishna Upadrasta and Y. N. Srikant. TACO 2020.
Learning Semantic Program Embeddings with Graph Interval Neural Network - Yu Wang, Ke Wang, Fengjuan Gao, and Linzhang Wang. OOPSLA 2020.
DeepCoder: Learning to write programs - Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. ICLR 2017.
Meta Large Language Model Compiler: Foundation Models of Compiler Optimization - Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, Hugh Leather. Arxiv 2024.
VESPA: static profiling for binary optimization - Angelica Aparecida Moreira, Guilherme Ottoni, and Fernando Magno Quintao Pereira. OOPSLA 2021. Code and Data
Mapping Computations in Heterogeneous Multicore Systems with Statistical Regression on Program Inputs - Junio Cezar Ribeiro Da Silva, Lorena Leao, Vinicius Petrucci, Abdoulaye Gamatie and Fernando Magno Quintao Pereira. TECS 2021.
Automatic Tuning of Compilers Using Machine Learning - Amir H. Ashouri, Gianluca Palermo, John Cavazos, and Cristina Silvano. Springer 2018.
Software Automatic Tuning - From Concepts to State-of-the-Art Results - K Naono, K Teranishi, J Cavazos, and R Suda. Springer 2010.
- Saman Amarasinghe, Compiler 2.0: Using Machine Learning to Modernize Compiler Technology. LCTES 2020.
- Amir Ashouri, Compiler Autotuning using Machine Learning: A State-of-the-art Review (slides). Polytechnic University of Milan 2018.
- ML-Compiler-Bridge - Library to interface Compilers and ML models for ML-Enabled Compiler Optimizations (paper).
- Supersonic - Automate reinforcement learning architecture design (paper).
- CompilerGym - Reinforcement learning environments for compiler optimizations (paper).
- CodeBert - pre-trained DNN models for programming languages (paper).
- IR2Vec - LLVM IR based program embeddings for machine learning (paper).
- programl - LLVM and XLA IR program representation for machine learning (paper).
- NeuroVectorizer - Using deep reinforcement learning (RL) to predict optimal vectorization compiler pragmas (paper).
- TVM - Open Deep Learning Compiler Stack for cpu, gpu and specialized accelerators (paper; slides).
- clgen - Benchmark generator using LSTMs (paper; slides).
- COBAYN - Compiler Autotuning using BNs (paper).
- OpenTuner - Framework for building domain-specific multi-objective program autotuners (paper; slides)
- ONNX-MLIR - Representation and Reference Lowering of ONNX Models in MLIR Compiler Infrastructure (paper).
- IREE - A retargetable MLIR-based machine learning compiler and runtime toolkit.
- TenSet: A Large-scale Program Performance Dataset for Learned Tensor Compilers - A dataset of tensor program performance records for six commonly used hardware platforms (paper).
- The Alberta Workloads for the SPEC CPU® 2017 Benchmark Suite - Additional workloads for the SPEC CPU2017 Benchmark Suite.
- Project CodeNet - Code samples written in 50+ programming languages, annotated with info, such as code size, memory footprint, CPU run time, and status (acceptance/error types)
- CodeXGLUE - A Machine Learning Benchmark Dataset for Code Understanding and Generation (paper)
- ANGHABENCH - A suite with One Million Compilable C Benchmarks (paper)
- BHive - A Benchmark Suite and Measurement Framework for Validating x86-64 Basic Block Performance Models (paper).
- cBench - 32 C benchmarks with datasets and driver scripts.
- PolyBench - 30 Stencil and Linear-algebra benchmarks with datasets and driver scripts. See also: GPU version, pre-computed datasets (paper).
- DeepDataFlow - 469k LLVM-IR files and 8.6B data-flow analysis labels for classification (paper).
- devmap - 650 OpenCL benchmark features and CPU/GPU classification labels (paper; slides).
ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI
International Symposium on Code Generation and Optimization, CGO
International Parallel and Distributed Processing Symposium, IPDPS
Machine Learning and Programming Languages Workshop, MAPL
International Conference on Learning Representations, ICLR
See Contribution Guidelines. TL;DR: send one of the maintainers a pull request.
