Link Search Menu Expand Document
Notes

What I wish I had known before scaling Uber to 1000 services

Why micro services

  • Move and release independently
  • Own your uptime (on-call)
  • Use the “best” tool for the job

What are the costs

  • Now you have a distributed system
  • Everything is an RPC
  • What if it breaks?

Less obvious costs

  • Everything is a trade-offs, you are going to give something up
  • You can build around problems
  • Might trade complexity for politics
  • You get to keep your biases

Tech stack

  • Pre-history: PHP (outsourced)
  • Dispatch: Node.JS, moving Go
  • Core Services: Python, moving to Go
  • Maps: Python and Java
  • Data: Python and Java
  • Metrics: Go

Different languages costs

  • Hard to share code
  • Hard to move between teams
  • What I wish I knew: Fragments the culture

RPC

  • HTTP/REST gets complicated
  • JSON needs a schema
  • RPCs are slower than PCs
  • What I wish I knew: Servers are not browsers

How many repos

  • With one repo you can make cross-project changes easier. Not really usable
  • Multiple repos
  • 7000 repositories

Operational

  • What happens when things break? Downstream dependencies; people issues, people fix your service
  • Can other teams release your service?
  • Understand a service in the larger context

Performance

  • Depends on language tools. Tools are all different. Flamegraphs
  • If dashboards are not generated automatically, teams create their own. This could be solved as if you expose your service automatically, to generate the dashboards automatically and homogeneously across all teams
  • Doesn’t matter until it does. You can place SLAs
  • Probably want at least simple perf requirements
  • What I wish I knew: “good” not required, but “known” is

Fanout

  • Go for the slowest thing that you have in your chain
  • Overall latency >= latency of slowest.
  • Looking at p95, p99, p99.9

Tracing

  • Lots of ways to get this. Zipkin, Open-tracing, etc.
  • Best way to understand fanout
  • Probably want sampling
  • What I wish I knew: cross-lang context propagation

Logging

  • Need consistent, structure logging
  • Multiple languages makes this hard
  • Logging floods can amplify problems
  • What I wish I knew: Accounting

Load testing

  • Need to test against production
  • Without breaking metrics
  • Preferably all the time
  • What I wish I knew: All systems need to handle “test” traffic

Failure testing

  • What I wish I knew: People won’t like it

Migrations

  • Old stuff still has to work, you cannot stop the world, people are making changes all the time
  • What happened to immutable?
  • What I wish I knew: mandates are bad. People will be more willing for change if you provide a better system rather than forcing them to change the current one

Open source

  • Build/buy trade-off is hard
  • Commoditisation
  • What I wish I knew: This will make people sad

Politics

Services allow people to play politics. Politics happen when “Company > Team > Self” is out of order.

Trade-offs

Everything is a trade-off Try to make them intentionally