Pico

It is time to turn our attention to modern-day system design, which would be incomplete without a distributed Machine Learning (ML) serving system. ML models are used in several modern-day systems. For serving inferences from these models, there is usually a dedicated system that exposes a microservice upon which all other services rely. In this insight, we will explore the design of one such serving system for the Transformer model - a cornerstone ML model in Generative Artificial Intelligence. This paper was presented at USENIX Operating System Design and Implementation (OSDI) '22 by Seoul National University and offers an excellent introduction to modern ML serving systems. In this insight, we will start by discussing what hardware accelerators are and how they are becoming crucial in modern computing. Then, we will explore GPUs - the most popular hardware accelerators today - in great detail, discussing their specific architectures and specifications. Next, we will perform a de...

In the CliqueMap paper, we explored how Remote Direct Memory Access (RDMA) serves as a backbone for an in-memory key-value store. In this paper, we will explore how RDMA can be leveraged to implement an algorithm for a more sophisticated problem: consensus. Consensus is a non-trivial problem, and achieving it in distributed systems is impossible and can only be done under certain assumptions. The algorithms that achieve consensus are often hard to understand - for example, Paxos. This paper takes the concepts of Paxos and implements them using RDMA to achieve microsecond-level consensus. This is very useful in applications demanding microsecond latency. Presented at USENIX Operating System Design and Implementation (OSDI) '20, this influential paper from VMware Research and EPFL (Swiss) has since received significant attention within the distributed systems community. It was authored by Marcos Aguirela, a leading researcher at VMware. In this insight, we will start with the latenc...