To never miss an article subscribe to my newsletter
No ads. One click unsubscribe.

How the big boys run their infrastructure

2 min read

Ciao grande persona,

somewhat troughout the week the name Borg came to my ears for google running their clusters with that. As it has the same name as my favorite backup tool, I was keen to learn more about it. This weeks papers showcases the design and the thoughts that went into building borg. I kinda wondered if it is still in use (as the paper is from 2015, which is stone age in computer times), but regarding an article by the register (Q2, 2020) it is sill in use.

Also this paper showed me I should learn more about cgroups and chroot, some links regarding this two Linux features are enclosed in the links section.


Abstract:

Google’s Borg system is a cluster manager that runs hundreds of thousands of jobs, from many thousands of different applications, across a number of clusters each with up to tens of thousands of machines.It achieves high utilization by combining admission control, efficient task-packing, over-commitment, and machine sharing with process-level performance isolation. It supports high-availability applications with runtime features that minimize fault-recovery time, and scheduling policies that reduce the probability of correlated failures. Borg simplifies life for its users by offering a declarative job specification language, name service integration, real-time job monitoring, and tools to analyze and simulate system behavior.We present a summary of the Borg system architecture and features, important design decisions, a quantitative analysis of some of its policy decisions, and a qualitative examination of lessons learned from a decade of operational experience with it.

Download Link:

https://storage.googleapis.com/pub-tools-public-publication-data/pdf/43438.pdf


Additional Links:


It would be awesome if you could help growing our little paper community even more by sharing it with your circles (you can also @eu_frey me on Twitter for retweets :D):

simon-frey.com/weeklycspaper

If you have any paper recommendation for me, please do not hesitate to approach me via [email protected] (Please keep the Backend & DevOps topic focus in mind)


With much love,

Simon Frey