Definition
Systems for ML is the engineering discipline of building the substrate that machine learning runs on: distributed training, inference serving, schedulers, storage, networking, hardware abstractions. It’s where most of the practical compute-efficiency wins of the past few years have come from.
Episodes covering this
Worth reading next
Papers we haven't done a deep dive on yet, but would recommend on this topic.
- Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- AlphaEvolve: A Learning Framework to Discover Novel Algorithms
- SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification