blog

How ChatGPT scales to millions of users

I put together this presentation to explore a question: what has to happen between sending a message and seeing an AI response when millions of people are using the service?

Follow a single request

  • I use the journey of a request to connect networking, regional application services, data systems, and AI compute.
  • The deck builds a mental model of a large-scale AI service. It is a learning presentation, not an internal architecture document from OpenAI.

Different workloads, different constraints

  • I explore why scaling an application service, a database, and AI inference are different problems.
  • The presentation connects regional services and GPU infrastructure instead of treating the system as one giant server.
  • My focus is on how these pieces work together and the questions their boundaries raise.

Keep asking why

  • I find infrastructure easier to understand when I start with the constraint and work towards the design decision.
  • The slides capture my understanding at the time I made them. Architecture and usage figures can change, so treat them as a starting point for further reading.

How ChatGPT works at scale

33 MB · 32 pages

Open PDF in a new tab