ResearchOpenAI

OpenAI scales Habitat for 12x faster ChatGPT reads

Original title: Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)

Plain language

What it is: OpenAI updated its storage platform, called Habitat, to better handle the growing number of ChatGPT users. The change improves how database resources are managed across different services.
How it works: They worked with the Azure team to run multiple services on shared database resources. The system automatically slows down less urgent tasks when needed to prevent slowdowns for regular users.
Where it is available: This applies to OpenAI's storage platform, Habitat, which runs on Azure Cosmos DB and supports ChatGPT and other products across 40 geographic regions.
Example: If a heavy background task like a large data export starts running, the system slows it down first so that regular users do not experience delays.

What it is: OpenAI optimized its online storage platform, Habitat, by scaling its underlying Azure Cosmos DB database. They transitioned from single-tenant containers back to a co-managed multi-tenant design.
How it works: They implemented hot key throttling to prevent noisy neighbors from consuming physical partition throughput. They also used Quality of Service request tiering to prioritize interactive user requests over offline workloads.
Where it is available: The architecture applies to Habitat, OpenAI's storage platform, which connects to Azure Cosmos DB across 40 geographic regions.
Example: Assigning lower priority to offline workloads like bulk inserts or data extracts so they are throttled first during high traffic.

The headline and explanation on this page are based on OpenAI's post. The full details are in the original.

Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)