In order to try and fix these issues, we built a better observability platform from scratch in around 6 months, using these technologies:
Better built tool box Offline#
There was only a handful of nodes, so one node dying meant the loss of around 1/8th of all Uber’s data!Īdding capacity took too long: the system needed to be taken offline for a week or more, to add capacityĭuring the first week I spent oncall, all I did was delete data from the backend to free up much-needed space and keep the observability stack up and running.
No replicas: if a node died, the company lost data. Not horizontally scalable : it was not possible to add capacity to the system just by adding more machines In 2015 when I started at Uber, the observability platform was a Graphite, Carbon and WhisperDB stack, which had several issues: Why Uber needed a new observability platform With that, it’s over to Martin, who narrates the rest of this article. Learnings from building an observability product and team from scratch Why Uber needed a new observability platformīecoming manager of the observability team I asked Martin how they built M3, and what he learned from leading that effort. In 2018, it processed 600 million data points per second (!!), and this has only grown since. I caught up with Martin, who believes the current production version of M3 is probably still one of the largest in the world, scale-wise. He’s since left Uber to co-found Chronosphere, an efficient observability platform.
Better built tool box code#
Martin Mao headed up M3’s engineering team at the time, from the writing of the first line of code for it, until it was rolled out across the organization. We needed to monitor payments business metrics across hundreds of major cities, and get alerted when any payment method saw regressions on either iOS or Android: and M3 was built to support such use cases. When I worked at Uber, my team – Rider Payments – was among the first to onboard this new and internal platform in 2017. The ride-hailing service’s open-source, large-scale metrics engine called M3, is an impressive piece of software.
0 kommentar(er)
