Architecting for Hyperscale: An In-Depth Analysis of Discord's Billion-Message-Per-Day Infrastructure

I. Executive Summary

Discord operates at an immense scale, processing billions of messages daily and connecting hundreds of millions of users worldwide. This report provides an in-depth technical analysis of the sophisticated and continually evolving architecture that underpins this remarkable capacity. At its core, Discord's strategy relies on a polyglot microservices approach, leveraging languages best suited for specific tasks, alongside a highly scalable and resilient database ecosystem. The migration of its primary message store to ScyllaDB, a C++ based NoSQL database, marks a significant milestone in its data handling capabilities. Real-time communication, the lifeblood of the platform, is predominantly powered by Elixir and the Erlang VM (BEAM), enabling massive concurrency for features like WebSocket gateways and presence updates. The entire infrastructure is built upon Google Cloud Platform (GCP), providing a robust and scalable foundation, further augmented by Cloudflare for global content delivery and security. Critical engineering decisions, including strategic database migrations from MongoDB to Cassandra and subsequently to ScyllaDB, the adoption of Rust for performance-sensitive data services, and the architectural patterns chosen for Elixir, have been pivotal in maintaining high performance, reliability, and scalability as the platform continues to grow. This document will dissect these components, offering a comprehensive view of how Discord sustains its operations at such a demanding scale.

II. Core Technical Architecture

The foundation of Discord's ability to handle billions of daily messages lies in its meticulously designed backend systems, diverse database ecosystem, and sophisticated traffic management. This section examines these core components, detailing the interplay of programming languages, service architectures, and data storage solutions.

A. Backend Systems: Languages, Monolithic vs. Microservice Components, API Design

Discord's backend is characterized by a pragmatic, polyglot approach, selecting technologies best suited for the unique demands of different parts of its system. This has resulted in a hybrid architecture that combines the stability of a central monolith with the flexibility and targeted performance of microservices.

Polyglot Environment:

A diverse set of programming languages forms the backbone of Discord's services, each chosen for its specific strengths in addressing particular challenges:

• Elixir: This functional language, running on the Erlang VM (BEAM), is the cornerstone of Discord's real-time communication infrastructure.¹ It powers the critical WebSocket gateways, message relaying, and presence systems. The BEAM's renowned capabilities in handling massive concurrency (millions of lightweight processes), fault tolerance (isolated processes and supervision trees), and low-latency operations make Elixir an ideal choice for managing persistent connections and broadcasting events to millions of simultaneously connected users.¹ Discord's Elixir stack comprises approximately 20 distinct services, highlighting its modular design for real-time functionalities.¹
• Python: Python has historically powered Discord's main monolithic API.¹ Its rapid development capabilities, extensive libraries, and mature ecosystem likely contributed to its initial adoption for core API functionalities.⁵ While Python remains an actively developed language at Discord ⁵, and is understood to be the primary language for the API monolith, explicit, recent official confirmations detailing its current dominance or any gradual refactoring within the monolith are not extensively covered in the latest engineering discussions.
• Rust: Increasingly, Rust has been adopted for services where performance, memory safety, and low-level control are paramount.¹ It is notably used for building high-performance "data services" that act as an intermediary layer to the ScyllaDB message store, optimizing data access and managing traffic to the database.⁷ Rust is also employed to enhance the performance of certain Elixir components through Rustler (Natively Implemented Functions - NIFs), allowing CPU-intensive tasks to be offloaded from the BEAM.¹
• Go: The Go programming language had a more significant role in Discord's past, for instance, powering the "read states" service which tracked message read/unread status.¹⁰ However, its usage has since diminished. Currently, its primary remaining function appears to be in a media proxy service responsible for image resizing. This service itself is largely a CGO wrapper around underlying C/C++ libraries that perform the actual image manipulation.¹⁰
• JavaScript (React/React Native): As expected, JavaScript, along with frameworks like React and React Native, is the foundation for Discord's client-side applications. This includes the web application, the desktop application (built with Electron), and the mobile applications for iOS and Android.⁶
• C/C++: Beyond its use in image resizing libraries wrapped by Go, C/C++ is also utilized for other performance-critical native modules within Discord's ecosystem, where direct hardware interaction or maximum computational efficiency is required.⁵

Architectural Approach (Monolith and Microservices):

Discord's system architecture is not purely monolithic nor entirely microservices-based; rather, it's a hybrid that has evolved over time. A central Python monolithic API handles many of the core business logic and API functionalities.1 This monolith is complemented by a growing ecosystem of specialized microservices. This architectural duality suggests a pragmatic approach to system development: the monolith likely provides stability for established features, while microservices are introduced for new functionalities, areas requiring extreme scalability, or components benefiting from independent development and deployment cycles.

The most prominent examples of microservices include the numerous Elixir services dedicated to real-time communication (e.g., WebSocket handling, presence updates, voice call signaling) ¹ and the Rust-based data services that mediate access to the ScyllaDB message store.⁷ This separation allows Discord to scale and optimize these critical, high-load components independently of the main API. The adoption of microservices aligns with the need for enhanced scalability, agility in development, fault isolation, and the ability for different teams to work on discrete parts of the system using the most appropriate technologies.¹³

API Design:

Discord's external API interactions are primarily facilitated through two main layers:

1. A HTTPS/REST API for general operations, such as fetching user profiles, server information, or sending messages through traditional HTTP requests.¹⁵
2. A persistent, secure WebSocket-based connection (Gateway API) for sending and subscribing to real-time events.¹⁵ This is crucial for receiving live updates like new messages, presence changes, and typing indicators. Authentication for API access is handled either via bot tokens or OAuth2 bearer tokens for third-party applications and services.¹⁵

B. Database Ecosystem

Discord's data storage strategy has undergone significant evolution, particularly for its massive message store, reflecting the platform's explosive growth and the continuous need for greater performance and scalability.

1. Message Storage Evolution: From MongoDB to Cassandra to ScyllaDB

The journey of Discord's message database is a compelling narrative of adapting to ever-increasing scale:

• Initial Phase (MongoDB): In its early days (2015), Discord utilized a single MongoDB replica set for all its data storage needs, including messages.⁶ This choice facilitated rapid iteration and product development. However, as the platform grew, this solution quickly hit its limits. By November 2015, with around 100 million messages stored, the dataset and its indexes could no longer fit into RAM, leading to unpredictable latencies and performance degradation.⁷
• Second Phase (Apache Cassandra): To address MongoDB's scaling limitations, Discord migrated its message store to Apache Cassandra.⁶ Cassandra was chosen for its proven linear scalability, fault tolerance (data replication and no single point of failure), and its successful adoption by other large-scale companies.¹⁸ Messages were modeled in Cassandra with channel_id as the partition key and message_id (a Snowflake ID, which is a time-sortable unique identifier) as the clustering key, ensuring messages within a channel were stored together and sorted chronologically.⁷ By 2017, Discord was running 12 Cassandra nodes, storing billions of messages. This scaled up to 177 nodes storing trillions of messages by early 2022.⁷ Despite its scalability, Cassandra presented its own set of challenges at Discord's extreme scale. These included:
• Hot Partitions: Certain highly active channels could create hot partitions, overwhelming specific nodes and causing cascading latency issues.⁷
• Unpredictable Latency: Read latencies (p99) could range from 40ms to 125ms, and write latencies (p99) from 5ms to 70ms, which were not ideal for a real-time application.⁷
• Maintenance Overhead: Operations like data compaction (merging SSTables) became increasingly expensive and performance-impacting.⁷
• JVM Garbage Collection (GC) Pauses: Being a Java-based database, Cassandra was susceptible to GC pauses, which could cause significant latency spikes and, in severe cases, require manual node intervention.⁷
• Current Phase (ScyllaDB): Faced with the operational complexities and performance variability of Cassandra at the trillion-message scale, Discord undertook another major migration, moving its primary message store to ScyllaDB.⁶ ScyllaDB is a NoSQL database written in C++, offering Cassandra compatibility at the API level (CQL) but with a fundamentally different internal architecture. The key drivers for this migration were:
• Performance: ScyllaDB's C++ implementation eliminates JVM GC pauses, a major source of latency spikes with Cassandra.⁷
• Architecture: Its shard-per-core architecture promises better resource utilization, stronger workload isolation, and potentially faster repair operations.⁷
• Reduced Latency: Post-migration, Discord reported significant improvements, with p99 read latencies for historical messages dropping to 15ms and p99 message insert latencies stabilizing at a consistent 5ms.⁷
• Efficiency: The migration allowed Discord to reduce the number of database nodes from 177 Cassandra nodes to 72 ScyllaDB nodes, while handling the same (and growing) volume of trillions of messages, indicating better resource efficiency.⁷

This migration was not merely a database swap. It involved significant engineering effort, including the development of intermediary Rust-based "data services" to manage client traffic to ScyllaDB, implement request coalescing, and provide consistent hashing for routing requests related to the same channel to the same service instance, thereby mitigating hot partition issues.⁷ Furthermore, Discord innovated at the storage layer by creating a "super-disk" storage topology on GCP. This custom solution combines fast, ephemeral Local SSDs (configured in a RAID0 array for maximum read speed) with durable Persistent Disks (in a RAID1 configuration with the Local SSD array, with the Persistent Disk set to "write-mostly"). This hybrid approach provides the low read latency of NVMe drives crucial for ScyllaDB's performance, while retaining the data durability and snapshotting capabilities of Persistent Disks.⁷The iterative journey of Discord's message database from MongoDB to Cassandra, and finally to ScyllaDB, underscores a deep commitment to addressing performance bottlenecks as they emerge at new orders of magnitude. Each step was a response to concrete operational challenges, and the latest migration to ScyllaDB involved holistic architectural changes—introducing new data services and custom storage solutions—demonstrating a sophisticated, multi-layered approach to achieving hyperscale data storage.

2. Metadata Storage (User Accounts, Guilds, Roles, Permissions, etc.)

While the evolution of the message store is well-documented, the specific database technology currently underpinning core metadata—such as user accounts, guild (server) information, roles, permissions, and other non-message related data—is not definitively and consistently identified in recent, detailed engineering disclosures.⁶

Historically, when Discord launched, MongoDB was used to store "everything".⁶ A 2019 blog post referred to efforts to reduce load on their "main user database," but did not specify its underlying technology.²⁷ Given Discord's scale of over 200 million monthly active users ²⁸ and the complexity of relationships between users, guilds, channels, and permissions, this metadata store must be robust, scalable, and highly available.

Plausible technologies could include:

• A separate, highly optimized NoSQL cluster (perhaps another ScyllaDB or Cassandra deployment, or even a modernized MongoDB setup if earlier scaling concerns for this type of data were addressed).
• A managed relational database service on GCP, such as Cloud SQL (PostgreSQL or MySQL) or Cloud Spanner, which offer scalability and consistency. However, direct confirmation of Discord's use of these specific services for primary operational metadata is lacking in the provided materials.²⁹
• Specialized graph databases could also be considered for managing the intricate relationships within Discord's social graph, although this is speculative based on the provided information.

Discord does utilize Google BigQuery extensively as its data warehouse for analytics, processing petabytes of data. Data from this warehouse can be exported to operational stores like ScyllaDB to power low-latency application features.²⁵ However, BigQuery serves analytical purposes and is not the primary operational database for real-time metadata transactions.

The absence of explicit, recent disclosures about this critical metadata store is noteworthy. It could imply that the current solution is stable and less prone to the dramatic scaling challenges faced by the message store, or it may be considered a more sensitive component of their architecture. The "main user database" mentioned in 2019 remains a key, yet technologically unspecified, part of their infrastructure.

3. Caching Layers

Distributed caching is an indispensable component for any system operating at Discord's scale, aimed at reducing latency and offloading primary datastores.

• Redis: Historically, Redis was employed to back the real-time message indexing queue. However, it was replaced by Google Cloud Pub/Sub due to issues with message dropping when the indexing queue experienced backlogs, particularly during Elasticsearch node failures.³⁸
• General Caching (Redis/Memcached): While specific, widespread current use of Redis or Memcached for general-purpose caching (e.g., user profiles, guild settings) is not explicitly detailed in recent engineering blogs, architectural best practices for systems of this magnitude strongly suggest their presence.¹¹ Such systems are typically used to cache frequently accessed, relatively static data to improve response times and lessen the load on backend databases. The need to serve data quickly to millions of users makes a robust caching strategy almost a certainty.

4. Search Infrastructure (Elasticsearch)

To enable users to search through the vast history of messages, Discord employs Elasticsearch.

• Core Functionality: Elasticsearch is used to index and search trillions of messages across the platform.³⁸
• Architectural Evolution: The Elasticsearch architecture has evolved from a few large clusters to a more resilient and manageable multi-cluster "cell" architecture. This involves deploying numerous smaller Elasticsearch clusters. This design includes:
• Dedicated DM Cells: To support cross-DM search functionality, a separate Elasticsearch cell is dedicated to user Direct Message (DM) messages, with data sharded by user_id.³⁸
• Dedicated BFG Clusters: For "Big Freaking Guilds" (BFGs)—servers with exceptionally large message volumes (billions of messages)—Discord provisions dedicated Elasticsearch clusters. These clusters utilize indices with multiple primary shards to handle the increased data volume and query load effectively.³⁸
• Initially, messages were sharded by guild or DM.³⁸
• Deployment and Management: Elasticsearch is deployed on Kubernetes and managed using the Elastic Kubernetes Operator (ECK). This approach facilitates automated cluster upgrades and rolling restarts with minimal service impact.³⁸
• Indexing Process: Messages are indexed lazily. A message queue, now backed by Google Cloud Pub/Sub, feeds messages to worker processes. These workers pull chunks of messages and leverage Elasticsearch's bulk-indexing capabilities for efficiency.³⁸

The sophisticated "cell" architecture and specialized clusters for different data types (DMs, BFGs) demonstrate a mature strategy for managing search at an enormous scale, focusing on workload isolation and performance optimization.

5. Real-time Data Queuing (Google Cloud Pub/Sub)

For managing asynchronous data flows, particularly for message indexing, Discord has transitioned to Google Cloud Pub/Sub.

• Reliable Indexing Queue: The message indexing queue, which feeds messages into Elasticsearch, was migrated from Redis to Google Cloud Pub/Sub.³⁸ This change was driven by the need for guaranteed message delivery and the ability to tolerate large backlogs of messages, especially if the Elasticsearch cluster experiences slowdowns or failures. With Redis, backlogs could lead to dropped messages when CPU limits were hit.³⁸
• Custom Message Router: To optimize the bulk indexing process, Discord implemented a custom Pub/Sub message router. This system streams messages from Pub/Sub, groups them by their ultimate destination (the specific Elasticsearch cluster and index), and then utilizes Tokio tasks (from the Rust asynchronous runtime) to collect batches of these grouped messages for efficient bulk indexing operations.³⁸

The adoption of Pub/Sub for the indexing pipeline underscores a prioritization of reliability and durable queuing for critical asynchronous tasks, ensuring that data destined for search is not lost even under system stress.

The following table provides a consolidated overview of some of the core technologies employed in Discord's backend:

Table 1: Core Technologies at Discord

C. Load Balancing and Traffic Management

Distributing incoming user traffic effectively across numerous backend servers is critical for maintaining high availability and performance at Discord's scale. Load balancing is employed extensively to achieve this.¹¹ While Discord's engineering blogs do not always specify the exact Google Cloud Load Balancing products in use ²⁴, their comprehensive adoption of GCP and the nature of their traffic (HTTP/S API calls, persistent WebSocket connections) make it highly probable that they utilize GCP's robust global load balancing solutions.⁴¹

Standard load balancing techniques such as Round Robin may be used, particularly in simpler contexts like bot infrastructure.⁴⁰ More sophisticated strategies are necessary for the main platform. Cloud-native load balancing services, like those offered by GCP (e.g., Global External Application Load Balancer for HTTP/S, and potentially Network Load Balancers or specific configurations for WebSocket traffic), provide features such as global traffic distribution using a single anycast IP address, automatic scaling, health checks, and integration with other cloud services like CDNs and security tools.¹¹

For WebSocket traffic, which forms the basis of Discord's real-time communication via its Gateway API ¹⁶, load balancing requires careful consideration of connection persistence. "Sticky sessions" are often necessary if load balancing at Layer 7 to ensure that a client's WebSocket connection remains tied to the same backend server instance throughout its lifecycle. Alternatively, Layer 4 load balancing can be used, with the application layer (e.g., Elixir services) managing session state and routing. Google Cloud's load balancers support WebSockets ⁴², offering options that Discord can tailor to its architecture. The Elixir-based session management system within Discord might also play a significant role in how WebSocket connections are maintained and potentially re-established across different gateway nodes, working in concert with the load balancing layer.

In microservice architectures, such as the one Discord employs for many of its components, load balancers are fundamental for distributing requests to horizontally scaled instances of each service, ensuring that no single instance becomes a bottleneck.¹³ This allows individual services to scale independently based on demand.

III. Strategies for Massive Scalability

Discord's capacity to support hundreds of millions of users and billions of daily messages hinges on a multi-faceted scalability strategy. This involves architectural choices that allow components to grow independently and handle vast increases in load.

A. Horizontal Scaling of Services

A cornerstone of Discord's scalability is the ability to scale its services horizontally.¹³ This means that as demand increases, more instances of a service can be deployed across additional servers or containers, rather than trying to make a single server more powerful (vertical scaling). Load balancers are then crucial for distributing the incoming traffic or workload evenly among these instances.¹³ This approach is fundamental to cloud-native architectures and allows for elastic scaling in response to real-time demand.

The effectiveness of horizontal scaling depends on services being designed to be stateless or to manage their state in a distributed manner. If a service instance relies heavily on local, non-sharable state, simply adding more instances won't effectively distribute the load for stateful operations. Discord's adoption of microservices and the inherent design of Elixir's process model, where state can be managed by many lightweight, independent processes, are key enablers for effective horizontal scaling. For example, individual Elixir services handling WebSocket connections or specific API functionalities can be scaled out by adding more nodes running these services.

B. Microservices and Decoupling

Discord has embraced a microservices architecture for significant parts of its platform, particularly for real-time functionalities and data handling.¹ This involves breaking down what might otherwise be a large, monolithic application into a collection of smaller, independent services. Each microservice is responsible for a specific business capability and communicates with other services through well-defined APIs, typically over HTTP or gRPC.¹³

The primary benefits of this approach in the context of scalability include:

• Independent Scaling: Different microservices experience different load patterns. For instance, the service handling message sending might be under more strain than a service managing user profile updates. Microservices allow Discord to scale each service independently based on its specific needs, optimizing resource utilization.¹
• Technological Diversity: Teams can choose the most appropriate technology stack (languages, databases) for their specific microservice, as seen with Discord's use of Elixir for real-time services and Rust for data services.¹
• Improved Agility and Decoupling: Development teams can work on different microservices concurrently and deploy them independently, as long as API contracts are maintained. This accelerates development cycles and reduces the risk associated with large, monolithic deployments.¹³
• Fault Isolation: If one microservice fails or experiences performance degradation, the impact can often be contained within that service, preventing a cascading failure across the entire platform. Other services can continue to operate, enhancing overall system resilience.¹³

The existence of a main Python API monolith alongside these microservices ¹ suggests an evolutionary path. It's common for large systems to gradually decompose a monolith by carving out new or performance-critical functionalities as microservices, rather than attempting a risky "big bang" rewrite. This pragmatic approach allows Discord to leverage the stability of existing code while gaining the scalability and flexibility benefits of microservices for targeted areas.

C. Database Scaling Techniques

Scaling the data layer is one of the most significant challenges for platforms like Discord. They employ a variety of techniques:

• Sharding/Partitioning: This is a fundamental strategy for distributing data and query load across multiple database nodes.
• In their Elasticsearch deployment for message search, messages are sharded across different indices. This has evolved into a "cell" architecture with dedicated clusters for DMs (sharded by user_id) and very large guilds (BFGs), allowing for more granular control and performance tuning.³⁸
• For their primary message store, ScyllaDB (and previously Cassandra) inherently uses partitioning. Data is distributed across the cluster based on a partition key. For messages, this is the channel_id, ensuring all messages for a given channel are co-located on the same set of replicas, which is then further sorted by the message_id (clustering key).⁷
• Replication: Both Cassandra and ScyllaDB automatically replicate data across multiple nodes (typically three or more, depending on the replication factor). This provides data redundancy for fault tolerance (if a node fails, data is still available on other replicas) and can also improve read scalability by allowing read requests to be served from multiple replicas.⁷
• Specialized Clusters and Workload Isolation: As seen with Elasticsearch, creating dedicated clusters for different types of data or workloads (like DMs vs. large guilds) is a key strategy.³⁸ This prevents noisy neighbors and allows for independent scaling and tuning of these clusters based on their unique characteristics. This principle likely extends to other parts of their database ecosystem.
• Custom Storage Solutions for Performance: The "super-disk" architecture developed for their ScyllaDB nodes on GCP is a prime example of going beyond standard database scaling techniques.⁷ By combining Local SSDs (for extremely low-latency reads) with Persistent Disks (for durability and snapshots) using Linux software RAID, Discord has engineered a storage solution tailored to the demanding I/O patterns of their message database. This demonstrates a deep understanding of their workload and a willingness to innovate at the infrastructure level to achieve performance goals.
• Optimized Access Layers: The introduction of Rust-based data services to sit in front of ScyllaDB (discussed further in III.E) is also a database scaling strategy. By coalescing requests and managing connections, these services reduce the direct load on the database, allowing it to perform more efficiently and handle a higher volume of unique queries.

Discord's approach to database scaling is not monolithic; it's a sophisticated combination of partitioning strategies within databases, physical distribution of data, replication for availability, workload isolation through specialized clusters, and custom hardware/software configurations. This multi-faceted approach is essential for handling the sheer volume and velocity of data generated on the platform.

The following table illustrates the evolution of Discord's message storage database, a critical component of their scaling journey:

Table 2: Evolution of Discord's Message Storage Database

D. Leveraging Elixir and the BEAM VM for Concurrency

Elixir, built on the Erlang Virtual Machine (BEAM), is a linchpin in Discord's strategy for handling massive concurrency, which is essential for a real-time chat platform with millions of simultaneous users.¹ The BEAM VM was designed from the ground up for building highly available, distributed, and fault-tolerant systems, making it an excellent fit for Discord's requirements.

Key aspects of Elixir/BEAM that Discord leverages include:

• Lightweight, Isolated Processes: Unlike traditional operating system threads, BEAM processes are extremely lightweight in terms of memory footprint and creation time.² This allows Discord to spin up millions of concurrent processes. Each process has its own isolated memory heap and performs its own garbage collection independently, preventing one misbehaving process from affecting others or causing system-wide GC pauses.²
• Massive Parallelism: Discord's architecture famously involves running a dedicated Elixir process for each guild (server) to act as a central routing and state management point for that guild, and another dedicated "session" process for each connected user's client (e.g., desktop app, mobile app).² This model allows for fine-grained state management and communication handling for millions of entities simultaneously.
• Message Passing: BEAM processes communicate by asynchronously sending messages to each other, rather than sharing memory directly.² This avoids the complexities and potential deadlocks associated with shared-memory concurrency (e.g., locks, mutexes) and fits well with a distributed system design.
• Fault Tolerance ("Let It Crash" Philosophy): Erlang and Elixir embrace a "let it crash" philosophy, coupled with supervision trees.² Supervisors are special processes that monitor other processes (workers). If a worker process encounters an unrecoverable error and crashes, its supervisor can detect this and restart it according to predefined strategies. This inherent fault tolerance is crucial for maintaining high availability in a system with so many independent moving parts. A crash in one user's session process, for example, will not bring down the entire WebSocket gateway.
• Hot Code Swapping: The BEAM VM supports hot code swapping, allowing code to be updated in a running system without downtime.² While not explicitly detailed in the snippets as a routine production practice by Discord, this capability is a hallmark of Erlang systems and contributes to high availability.

The choice of Elixir is therefore not merely a language preference but a fundamental architectural decision. The concurrency model provided by BEAM allows Discord to manage state and real-time interactions for its vast user base in a way that is both highly concurrent and resilient to failures, forming the bedrock of features like presence updates, message fan-out, and voice call signaling.

E. Data Services and Request Coalescing (e.g., Rust services for ScyllaDB)

To further optimize database interactions and manage the immense load on their ScyllaDB clusters, particularly in the face of challenges like "hot partitions" (where a single data partition receives a disproportionate amount of traffic), Discord engineered a layer of "data services" written in Rust.⁷

These Rust data services act as an intelligent intermediary between the API monolith (and potentially other backend services) and the ScyllaDB clusters. They expose gRPC endpoints, with roughly one endpoint corresponding to each type of database query.⁷

A critical feature of these data services is request coalescing.⁷ In a high-traffic scenario, many users might simultaneously request the same piece of data (e.g., the latest messages in a very active channel, or information about a popular guild). Without coalescing, each of these user requests could translate into a separate query to the database, potentially overwhelming it. Request coalescing ensures that if multiple identical requests arrive at the data service within a short time window, the service issues only a single query to the database. Once the database responds, the data service distributes this single result to all the waiting requesters. This dramatically reduces the load on the ScyllaDB cluster, conserves database resources, and improves overall response times.

To enhance the effectiveness of request coalescing, Discord also implemented consistent hash-based routing for these data services.⁷ This ensures that requests pertaining to the same entity (e.g., the same channel) are consistently routed to the same instance of the data service. This increases the likelihood that multiple requests for the same data will arrive at the same service instance, maximizing the benefits of coalescing.

The development of these Rust data services illustrates a sophisticated understanding of distributed system performance. It acknowledges that scaling a database is not just about the database software itself but also about managing and optimizing the access patterns to it. This layer was a key innovation that contributed to the successful and performant migration to ScyllaDB, particularly in handling Discord's notoriously bursty and often highly concentrated workloads. The choice of Rust for these services likely reflects the need for high throughput, low latency, and fine-grained control over memory and concurrency in this critical data access path.

IV. Ensuring Message Delivery: Reliability and Low Latency

For a platform centered around communication, ensuring that messages are delivered reliably and with minimal delay is paramount. Discord employs a combination of real-time protocols, asynchronous processing, fault-tolerant service design, and a global content delivery network to achieve this.

A. Real-time Communication Protocols (WebSockets, WebRTC)

Discord's real-time capabilities are built upon two primary communication protocols:

• WebSockets: This protocol is the workhorse for most of Discord's real-time eventing and messaging features.⁴⁶ WebSockets establish a persistent, bidirectional communication channel over a single TCP connection between the client (user's app) and Discord's servers. This persistent connection allows for low-latency, full-duplex data exchange, meaning both the client and server can send data to each other simultaneously without the overhead of establishing new HTTP connections for each interaction. Discord's Gateway API specifically utilizes WebSockets for clients to connect and receive a stream of real-time events, such as new messages, presence updates, channel updates, and role creations.¹⁶ The Elixir-based backend is responsible for managing these millions of concurrent WebSocket connections.¹ The efficiency of Elixir and the BEAM VM in handling such a large number of persistent connections is critical to the scalability of this layer.
• WebRTC (Web Real-Time Communication): When it comes to voice and video communication, Discord leverages WebRTC.⁶ WebRTC is an open-source framework that enables real-time peer-to-peer (P2P) media streaming directly between browsers and applications. In scenarios where P2P connections are not feasible (e.g., due to NAT traversal issues or for large group calls), Discord employs Selective Forwarding Units (SFUs).⁵¹ An SFU is a server that receives media streams from all participants in a call and then forwards only the necessary streams to each participant, optimizing bandwidth and processing. Discord's DAVE protocol, designed for end-to-end encrypted audio and video calls, also integrates with WebRTC, specifically using its encoded transform API to encrypt media frames before transmission.⁵¹

The strategic use of WebSockets for signaling, presence, and text-based messages, and WebRTC for high-bandwidth audio/video streams, is a common and effective pattern in modern real-time communication platforms. Discord's challenge lies in operating these at an unprecedented scale.

B. Message Queuing and Asynchronous Processing

To maintain responsiveness and resilience, especially for tasks that do not need to be completed synchronously with a user's immediate action, Discord utilizes message queuing and asynchronous processing.

• Google Cloud Pub/Sub for Search Indexing: As detailed previously (Section II.B.5), the critical task of indexing messages for search functionality is handled asynchronously. Messages are published to a Google Cloud Pub/Sub queue. Worker processes then consume messages from this queue in batches and submit them to Elasticsearch for indexing.³⁸ This decouples the real-time message delivery path from the indexing path. If the search indexing system is slow or temporarily unavailable, messages can accumulate in Pub/Sub without impacting the user's ability to send and receive messages in real-time. Pub/Sub's guarantee of message delivery and its ability to handle large backlogs are key benefits here.
• Elixir's GenStage for Backpressure and Flow Control: Within its Elixir-based services, Discord has reportedly used GenStage, an Elixir behavior for specifying data-processing pipelines with built-in support for backpressure.¹ This is particularly useful for handling bursts of requests or events, such as the "push request bursts of over a million per minute" mentioned in one blog post title.²⁶ GenStage allows systems to gracefully manage high loads by regulating the flow of data between producer and consumer stages, preventing downstream components from being overwhelmed.

The application of asynchronous processing and robust message queuing improves the overall stability and performance of the platform by smoothing out traffic spikes, isolating failures, and ensuring that resource-intensive background tasks do not degrade the interactive user experience.

C. Fault Tolerance and Redundancy in Real-time Services

Maintaining high availability for real-time services in the face of inevitable failures is a complex engineering challenge. Discord addresses this through multiple layers of fault tolerance and redundancy:

• Elixir/BEAM's Intrinsic Fault Tolerance: The Erlang VM (BEAM), upon which Elixir runs, is designed with fault tolerance as a primary feature.
• Isolated Processes: Each user session and guild is typically managed by its own lightweight Elixir process. These processes are isolated, meaning a crash in one process (e.g., due to an unexpected error in handling a specific user's data) does not directly affect other processes.¹
• Supervision Trees: Elixir applications use supervision trees, where supervisor processes monitor worker processes. If a worker process dies, its supervisor can automatically restart it according to predefined strategies (e.g., restart only the failed process, restart all related processes, or escalate the failure).² This "let it crash" philosophy allows for self-healing systems.
• State Management and Recovery: While individual processes manage their state in memory (process heap or shared via ETS) ⁴⁸, persistent state for guilds (like channels, roles, permissions) and users is stored in backend databases (the metadata store discussed in II.B.2). Upon a process restart, or if a guild process needs to be moved to a different node, it would likely rehydrate its necessary operational state from these persistent stores and reconstruct its in-memory view, including populating ETS tables where applicable. The specifics of how a full guild process state is recovered after a node failure (where all in-memory state on that node is lost) are not fully detailed in the provided materials but are critical for consistency.⁴⁸
• Distributed Erlang: For communication between different Elixir services running on potentially different nodes, Discord utilizes Distributed Erlang. They configure a partially meshed network using etcd for service discovery and shared configuration, rather than the default fully meshed network, which can be more manageable at scale.¹
• Gateway Heartbeating and Resume: The Discord Gateway API (WebSocket) has a built-in heartbeating mechanism.¹⁶ Clients must send heartbeat events at regular intervals. If the server doesn't receive a heartbeat, it considers the connection stale. Conversely, if the client doesn't receive a heartbeat acknowledgement (ACK) from the server, it should assume the connection has failed. In many disconnection scenarios, clients can attempt to "resume" their session, which involves reconnecting to a potentially different gateway server and providing a session ID and sequence number. If successful, the client can receive any missed events, ensuring continuity. This mechanism is crucial for handling transient network issues or brief server-side disruptions without forcing a full re-identification and state reload.
• Database Replication and Durability: At the persistence layer, databases like ScyllaDB (and previously Cassandra) ensure data reliability through replication. Data is written to multiple nodes in the cluster, so the failure of a single node (or even multiple nodes, depending on the replication factor and consistency level) does not result in data loss.⁷ This is fundamental for message persistence and metadata integrity.
• Geographically Distributed Infrastructure: Discord's services are hosted on Google Cloud Platform, leveraging its global network of data centers.⁶ This allows for deploying services closer to users in different regions, reducing latency. For large servers, a technique referred to as "server sharding" (distinct from database sharding) is used, which likely involves distributing the workload of a single large Discord server (guild) across multiple physical or logical server resources to manage performance and ensure smooth operation.⁴⁷ This global distribution also provides a level of redundancy against regional outages.

This multi-layered strategy—from the process level in Elixir to the connection management in the Gateway protocol, and down to the data replication in the persistence tier—creates a resilient system capable of weathering various types of failures while striving to maintain continuous service for users.

D. Global Content Delivery Network (CDN) Strategy for Assets and Media

To ensure fast loading times for static assets (like images, avatars, custom emojis, CSS, JavaScript files) and media content (like video attachments or Go Live streams) for its global user base, Discord employs a Content Delivery Network (CDN) strategy. CDNs cache content on servers geographically distributed around the world (Points of Presence, or PoPs). When a user requests an asset, the CDN serves it from the PoP closest to the user, significantly reducing latency and the load on Discord's origin servers.¹¹

Discord's CDN strategy appears to involve:

• Cloudflare: Discord is a prominent customer of Cloudflare.⁵⁶ Cloudflare provides a massive global network that is used not only for robust DDoS protection and as a reverse proxy (shielding Discord's origin servers on GCP) but also for its extensive CDN capabilities.⁵⁶ Cloudflare's edge servers cache static content and can help absorb traffic spikes.
• Google Cloud CDN: Given that Discord's primary infrastructure is hosted on Google Cloud Platform ⁶, it is highly probable, and suggested by some community discussions ⁵⁷, that they also leverage Google Cloud CDN. This would be particularly efficient for serving assets that originate from Google Cloud Storage or other GCP-based services.⁴³ Google Cloud CDN integrates well with GCP load balancers and storage services, offering features like global caching and support for modern protocols.⁶⁰

The likely approach is a multi-layered or hybrid CDN strategy. Google Cloud CDN might be used as the primary CDN for assets tightly integrated with their GCP backend, while Cloudflare acts as a significant edge layer, providing additional caching, a broader network reach, and its well-known security services. This combination allows Discord to optimize for both performance (low latency delivery from nearby edge servers) and resilience (DDoS mitigation and traffic absorption). Effective cache-control headers and cache invalidation strategies are crucial for ensuring users receive updated assets while maximizing cache hit ratios.⁵⁴

V. Security and Encryption Framework

Discord implements a multi-faceted security and encryption framework to protect user communications and data, addressing threats from various angles and complying with privacy expectations.

A. End-to-End Encryption for Audio/Video (DAVE Protocol)

A significant advancement in Discord's security posture is the introduction of end-to-end encryption (E2EE) for audio and video (A/V) communications. This is facilitated by their DAVE (Discord Audio & Video End-to-End Encryption) protocol.⁵¹

• Scope: DAVE is designed to provide E2EE for A/V calls within Direct Messages (DMs), Group DMs (GDMs), server voice channels (excluding stage channels), and Go Live streams.⁵¹
• Technology: The protocol leverages Messaging Layer Security (MLS) for establishing secure group key exchanges among participants.⁵¹ MLS is a modern IETF standard designed for efficient and secure group messaging. Once the MLS group is established, media encryption keys are derived. The actual media encryption uses the WebRTC Encoded Transforms API, with AES-128-GCM being the specified media ciphersuite for DAVE protocol version 1.0.⁵¹ This means that encoded audio or video frames are encrypted on the sender's side before being transmitted and decrypted only by the intended recipients.
• Security Goals: The DAVE protocol aims to achieve:
• Confidentiality: Ensuring that only active and authorized participants in a call can access the content of the A/V communications. Media encryption keys are unique per call and per group within the call, and change when participants join or leave. Neither Discord nor any other external party should have access to these media encryption keys.⁵¹
• Integrity: Protecting the A/V stream from surreptitious corruption, tampering, forgery, or replay by unauthorized parties.⁵¹
• Authenticity: Allowing participants to verify the identity of other participants in the call and trust that they are communicating with the expected individuals.⁵¹
• Verification: Users can verify the E2EE status of a call and the identity of other participants through "Voice Privacy Codes" or "Verification Codes." These codes can be compared out-of-band (e.g., via a different secure channel) to ensure all participants see the same code, confirming the integrity of the E2EE session.⁵¹ Users can also opt-in to persistent verification keys for devices they trust.⁶²
• Rollout and Limitations: E2EE A/V is being progressively rolled out, starting with updated desktop and mobile clients in September 2024, with web and console client support planned for 2025.⁵² The delay for web clients is partly due to limitations in browser WebRTC API availability for implementing certain E2EE mechanisms.⁵²
• Important Clarification: It is crucial to note that while audio and video communications are being secured with E2EE via DAVE, text messages on Discord are not end-to-end encrypted under this initiative and continue to be subject to Discord's standard content moderation practices.⁵²

The development and deployment of the DAVE protocol, including a public whitepaper and external security audits ⁵², represent a substantial commitment by Discord to enhancing the privacy of real-time voice and video interactions on its platform.

B. Data-in-Transit Encryption (TLS)

Beyond the specific E2EE for A/V, all general data transmitted between Discord clients (desktop, web, mobile) and Discord's servers, as well as between Discord's internal services, is encrypted in transit.

• This is primarily achieved using Transport Layer Security (TLS), the industry standard for securing HTTP and other network communications.⁷
• All API calls (REST), WebSocket connections (via WSS - Secure WebSockets for the Gateway API ¹⁶), and other data exchanges are protected by TLS.
• Even the MLS handshake messages used by the DAVE protocol for setting up E2EE A/V calls are themselves transport-encrypted using TLS when communicated between clients and Discord's voice gateway servers.⁵¹

TLS encryption ensures that data cannot be easily intercepted or tampered with by third parties while it travels across the internet or within Discord's own network.

C. Data-at-Rest Encryption

Protecting data when it is stored on disk (at rest) is another critical aspect of security.

• General Policy: Discord's Privacy Policy states that the company takes measures to protect user information, including "encrypting all information... at rest using technologies like TLS" (though TLS is typically for in-transit, the intent here likely refers to appropriate at-rest encryption mechanisms).⁷
• ScyllaDB Message Store: Discord's engineering blogs detailing the migration to ScyllaDB for message storage primarily focus on performance, scalability, and operational aspects.⁷ These blogs do not explicitly confirm whether Discord implements ScyllaDB's native data-at-rest encryption feature for their ScyllaDB clusters.⁷ ScyllaDB Enterprise does offer robust data-at-rest encryption capabilities, allowing tables or entire clusters to be encrypted.⁶³ ScyllaDB Cloud services also encrypt data at rest by default.⁶⁴ However, for Discord's self-managed deployment on GCP, the specific configuration choice (whether they use ScyllaDB's encryption, rely solely on GCP's disk-level encryption, or a combination) is not detailed in their technical blogs.
• Other Data Stores (e.g., Metadata): The general statement from the Privacy Policy regarding encryption at rest would presumably apply to databases storing user accounts, guild information, and other metadata. However, specific technical details about the encryption mechanisms for these stores are not provided in the available engineering snippets.
• GCP Persistent Disk Encryption: Discord utilizes GCP Persistent Disks for its "super-disk" storage solution and likely for other VM storage needs.²⁴ GCP Persistent Disks are encrypted at rest by default by Google, providing a baseline level of protection for data stored on them. This could be one layer of Discord's data-at-rest encryption strategy.

While Discord's official privacy stance indicates a commitment to encrypting data at rest, the specific technical implementations and choices (e.g., application-level encryption, database-native encryption for ScyllaDB, reliance on underlying cloud provider disk encryption) are not as transparently detailed in their engineering blogs as, for example, the DAVE protocol for A/V E2EE.

D. User Account and Platform Security Measures

Discord implements a variety of measures to secure user accounts and protect the platform from abuse:

• Multi-Factor Authentication (MFA): Users are strongly encouraged to enable MFA to add an additional layer of security to their accounts. Discord supports MFA via:
• Authenticator Apps: Time-based One-Time Password (TOTP) apps (e.g., Google Authenticator, Authy).⁶⁸
• Security Keys: Hardware security keys using U2F/WebAuthn standards (e.g., YubiKey) offer strong phishing-resistant authentication.⁶⁸ Discord has specifically blogged about modernizing their MFA with WebAuthn.²⁶
• SMS/Text-based MFA: While available, Discord generally recommends authenticator apps or security keys over SMS due to the potential vulnerabilities of SMS (e.g., SIM swapping).⁶⁸
• Password Security: Discord advises users to create strong, unique passwords and follow good password hygiene practices, such as not reusing passwords and keeping them secret.⁶⁸
• Secure Bot Tokens: For developers creating Discord bots, there are best practices for securing bot tokens and other sensitive credentials to prevent unauthorized access and abuse.⁴⁰
• Rate Limiting and Anti-Spam Measures: The platform employs rate limiting on API requests and various anti-spam measures to prevent abuse, automated attacks, and platform misuse.⁴⁰ This includes measures against common security vulnerabilities.
• Privacy Controls: Users have access to a range of privacy settings, allowing them to control who can send them direct messages, who can add them as friends, what information is visible on their profile, and more.⁴⁷
• Cloud Development Environments (CDEs): Discord's engineering teams have transitioned to using CDEs hosted on a cloud provider (using Coder's technology).⁵ This shift offers security benefits such as:
• Enhanced Security: Centralized management, better control over development environments, and built-in Identity and Access Management (IAM).
• Reproducibility and Immutability: More consistent and reproducible environments can reduce security risks associated with snowflake developer setups.
• Content Moderation and Trust & Safety: While not strictly encryption, Discord invests heavily in Trust & Safety, using a combination of automated tooling (including machine learning), human moderation, and community reporting to detect and remove content and behavior that violates their Community Guidelines.⁷⁰

These measures collectively contribute to a defense-in-depth strategy for securing user accounts, protecting platform integrity, and maintaining user privacy.

The following table summarizes key security protocols and measures employed by Discord:

Table 3: Overview of Security Protocols and Measures at Discord

VI. Infrastructure Foundation: Providers and Services

Discord's global operations, designed to deliver low-latency communication to millions of users, are built upon a foundation of robust cloud infrastructure and strategic partnerships.

A. Primary Cloud Provider: Google Cloud Platform (GCP)

Discord has made a significant commitment to Google Cloud Platform (GCP) as its primary cloud provider, running the majority of its backend services and hardware within GCP's ecosystem.⁶ The choice of a major cloud provider like GCP offers several advantages:

• Global Infrastructure: GCP operates numerous data centers and network points of presence across many regions worldwide.⁶ This global footprint is essential for Discord to serve its international user base with low latency by hosting services closer to users.
• Scalability and Elasticity: Cloud platforms provide the ability to scale resources up or down based on demand, which is critical for a platform like Discord that experiences fluctuating loads.
• Managed Services: GCP offers a wide array of managed services (databases, queuing, machine learning, etc.) that can reduce Discord's operational burden, allowing their engineering teams to focus more on application-level innovation rather than undifferentiated infrastructure management.
• Reliability: Major cloud providers invest heavily in redundant power, cooling, and networking to ensure high availability for their services.

By leveraging GCP, Discord can offload much of the complexity of managing physical data centers and instead focus on building and scaling its unique application features.

B. Key GCP Services Utilized

Discord employs a diverse range of GCP services, tailored to different aspects of its infrastructure:

• Google Compute Engine (GCE): This is GCP's Infrastructure-as-a-Service (IaaS) offering, providing virtual machines. Discord uses GCE instances to host its various backend services, including those written in Elixir, Python, and Rust, as well as its database clusters like ScyllaDB.²⁴ The custom "super-disk" storage solution for ScyllaDB, for example, is built using GCE instances equipped with both Local SSDs and Persistent Disks.²⁴
• Google Kubernetes Engine (GKE): Discord is self-described as a "heavy Kubernetes user".⁵ GKE is Google's managed Kubernetes service, used for deploying, managing, and scaling containerized applications. It is highly probable that Discord uses GKE for orchestrating its numerous microservices, including components written in Python, Rust, and Elixir, as well as for managing deployments of third-party applications like Elasticsearch (which is deployed on Kubernetes using the Elastic Kubernetes Operator - ECK).³⁸ GKE simplifies many aspects of running Kubernetes, such as cluster provisioning, upgrades, and scaling.⁷³
• Google Cloud Storage (GCS): This is a scalable and durable object storage service. Discord likely uses GCS for a variety of purposes, such as storing user-uploaded files (images, videos, attachments before they are potentially processed or served via CDN), application backups, log archival, and potentially as an origin for content served through Google Cloud CDN.¹¹ An older engineering blog also mentioned the possibility of archiving inactive channel data to GCS.¹⁸
• Google Cloud Bigtable: Discord utilizes Bigtable, a massively scalable NoSQL wide-column store, to support and deliver its machine learning (ML)-driven experiences.⁷² Its use cases likely include serving as a feature store for ML models, caching data for fast access by ML training and inference frameworks, and generally scaling ML infrastructure.
• Google Cloud Persistent Disks & Local SSDs: These are fundamental storage components for Discord's GCE instances.
• Persistent Disks: Network-attached block storage that provides durable and reliable storage, with features like on-the-fly resizing and snapshotting for backups. Discord uses these as part of their "super-disk" solution for ScyllaDB, valuing their durability and snapshot capabilities.⁷
• Local SSDs: High-performance, physically attached NVMe SSDs that offer very low latency. Discord uses these in a RAID0 configuration within their "super-disk" setup to provide a fast read cache for their ScyllaDB clusters.⁷
• Google Cloud Pub/Sub: A real-time, scalable messaging service. As discussed, Discord uses Pub/Sub to provide a reliable and durable queue for its message indexing pipeline, feeding messages asynchronously to Elasticsearch.³⁸
• Google BigQuery: This is Discord's primary data warehouse solution, capable of storing and analyzing petabytes of data.²⁵ It processes trillions of records, enabling Discord to derive insights, perform analytics, and support data-driven product development.
• Google Cloud Load Balancing: While specific products are not always named by Discord, it's virtually certain they use GCP's comprehensive load balancing services to distribute the immense volume of incoming API (HTTP/S) and WebSocket traffic across their backend fleets.¹¹ GCP's global load balancers offer features like anycast IP addresses, SSL offloading, and integration with CDN and security services, which are essential for a global application like Discord.
• Google Cloud CDN: Given Discord's extensive use of GCP and the need to serve static assets (images, JS, CSS) and media globally with low latency, they likely utilize Google Cloud CDN.⁴³ This service integrates with Google Cloud Storage and external Application Load Balancers to cache content at Google's network edge, closer to users.

Discord's strategy demonstrates a sophisticated use of GCP, blending IaaS components for deep customization and performance (like GCE with custom disk configurations) with PaaS and managed services (like GKE, Pub/Sub, BigQuery, Bigtable) to accelerate development and reduce operational overhead.

The following table provides a summary of key GCP services used by Discord:

Table 4: Key Google Cloud Platform Services Leveraged by Discord

C. Role of Cloudflare

In addition to its deep reliance on GCP, Discord strategically utilizes Cloudflare as a critical external partner, primarily for enhancing security and performance at the edge of its network.

• DDoS Protection and Web Application Firewall (WAF): A primary role of Cloudflare is to protect Discord's origin infrastructure (hosted on GCP) from Distributed Denial of Service (DDoS) attacks and other malicious web traffic.⁵⁶ Cloudflare's massive global network can absorb and filter out attack traffic before it reaches Discord's servers.
• Reverse Proxy: Cloudflare acts as a reverse proxy, meaning user traffic is routed through Cloudflare's network before reaching Discord's backend. This helps to mask the origin IP addresses of Discord's servers, adding a layer of security.
• Content Delivery Network (CDN): Cloudflare also provides extensive CDN services.⁵⁶ While Discord likely uses Google Cloud CDN for assets originating from GCP, Cloudflare's CDN can complement this by providing an additional caching layer, potentially with a wider network of PoPs or different caching strategies, further reducing latency for global users and offloading traffic from origin servers.⁵⁷
• Domain Registrar: Discord also uses Cloudflare as its domain name registrar.⁵⁷

The use of Cloudflare in conjunction with GCP creates a robust, multi-layered defense and delivery architecture. Cloudflare handles initial traffic screening, DDoS mitigation, and edge caching, while GCP provides the core backend infrastructure, application hosting, and primary data storage. This division of labor allows each provider's strengths to be leveraged effectively.

VII. Real-time Processing Capabilities

Discord's essence lies in its real-time interaction features. Handling presence updates (who is online, what they are doing) and typing indicators instantaneously for millions of concurrent users requires a highly optimized and responsive backend. Elixir and the BEAM VM are central to these capabilities.

A. Handling Presence Updates

Presence updates are a fundamental aspect of the Discord experience, informing users about the status (online, idle, do not disturb, offline) and activity (e.g., playing a game, listening to music) of their friends and other server members.

• Elixir's Role: These updates are managed and distributed by Discord's Elixir-based real-time services.¹ When a user's client application detects a change in status or activity, it sends an update to the Discord backend, typically via the persistent WebSocket connection managed by a dedicated "session" process for that user.²
• Fan-out Mechanism: This session process then communicates the change. For guild-related presence (e.g., status visible to other members of a server), the update is relayed to the relevant "guild" process(es). The guild process, which maintains a list of currently connected members (or their sessions), is then responsible for "fanning out" this presence update to all other relevant connected clients within that guild.²
• "Presence" Service: A dedicated "Presence" service, likely one of the ~20 Elixir microservices, is mentioned as being responsible for keeping track of a user's sessions across the platform.⁷⁷ This service would play a crucial role in aggregating and distributing presence information efficiently.

Managing and broadcasting presence information at Discord's scale—with potentially millions of users changing state frequently—is a significant distributed systems challenge. The lightweight nature of Elixir processes allows Discord to maintain state for each user's presence individually and to efficiently propagate updates through its WebSocket infrastructure. The architectural patterns, including the per-guild and per-session processes, along with optimizations like "relays" (discussed in VII.C), are critical to making this scalable.

B. Implementing Typing Indicators

Typing indicators ("User is typing...") provide immediate visual feedback that someone is actively composing a message in a DM or channel.

• Real-time Events: When a user starts typing in the Discord client, an event is sent to the backend via the WebSocket connection. Similarly, events are sent when the user stops typing or sends the message.⁴⁶
• Backend Processing and Fan-out: These typing events are processed by the Elixir services. The backend identifies the relevant recipients (e.g., other users in the same DM or channel) and fans out the typing indicator event to their clients, again via WebSockets.⁷⁸ Discord's Gateway API is the conduit for these event transmissions.¹⁶
• Native vs. Bot Implementation: While typing indicators are native in DMs and group DMs, some of the provided material discusses implementing server-wide typing indicators in text channels (where they are not natively present for all users) using bots and the sendTyping() API function.⁷⁸ This indicates that the underlying infrastructure and API support the transmission and display of such events.

Although seemingly simple, typing indicators can generate a high frequency of short-lived events. The low-latency, high-concurrency WebSocket infrastructure powered by Elixir is essential for ensuring these indicators appear and disappear promptly, contributing to the feeling of a live conversation. The same fan-out mechanisms used for messages and presence updates are leveraged for distributing typing status.

C. Elixir-based State Management for Guilds and Sessions (ETS, Relays)

The scalability and responsiveness of Discord's real-time features heavily depend on how it manages state for its millions of guilds and user sessions within its Elixir services.

• Per-Process State: As previously established (Section III.D), Discord's core Elixir architecture involves a dedicated GenServer process for each guild and each connected user session.² These processes hold the immediate operational state for that entity in their own memory heap. For a session, this might include connection details and subscriptions. For a guild, this includes information like the list of online members, channel states, and ongoing activities.
• ETS (Erlang Term Storage): For data that needs to be shared or accessed very quickly by multiple processes, or to reduce the memory footprint of individual large guild processes, Discord utilizes ETS.⁴ ETS provides fast, in-memory, key-value storage accessible to all processes on the same Erlang node.
• A key use case is storing the list of members for a large guild in an ETS table. This allows the main guild process to offload this potentially large dataset from its own heap, which can improve its garbage collection performance and overall responsiveness.
• Worker processes can then be spawned by the guild process to perform operations that require iterating over all members (e.g., checking permissions for an @everyone ping) by directly accessing the shared ETS table, without blocking the main guild process from handling other incoming events.⁴⁸
• Relays: As some Discord guilds grew to encompass hundreds of thousands or even over a million concurrent users (e.g., the Midjourney server ⁴⁸), the single Elixir process responsible for that guild could become a bottleneck, particularly for fanning out messages and presence updates. To address this, Discord introduced a system of "relays".²
• Relays are intermediary Elixir processes, likely running on separate nodes, that sit between the main guild process and the user session processes.
• The guild process delegates the task of fanning out updates to a subset of its connected sessions to these relay processes. Each relay might handle connections for up to 15,000 sessions.⁴⁸
• Relays perform tasks like permission checking and direct message delivery to the sessions they manage, thus distributing the fan-out workload that would otherwise be concentrated on the single guild process. The main guild process still handles operations requiring a global view of the server state and coordinates with the relays.
• Optimizations were made for relays handling very large guilds to only track member data relevant to their assigned sessions, reducing memory overhead.⁴⁸
• Fault Tolerance and State Recovery: Elixir's supervision strategy ensures that if a guild or session process crashes, it can be restarted. However, since these processes hold state in memory (process heap or ETS, which is also in-memory), a node failure would result in the loss of this in-memory state.
• Persistent State: The canonical, durable state for guilds (members, roles, channels, settings) and users (profiles, relationships) resides in persistent backend databases (the metadata store discussed in Section II.B.2).
• State Rehydration: Upon a restart (e.g., after a crash or during a deployment that moves a guild process to a new node), a guild process would need to "rehydrate" its operational state. This involves fetching the necessary persistent data from the backend database and rebuilding its in-memory representation, including populating relevant ETS tables.
• Session Resumption: For user sessions, the Gateway API's "resume" functionality ¹⁶ allows clients to reconnect after a brief disconnection and potentially receive missed events, implying some level of short-term session state or event buffering on the backend. A dedicated "Sessions" service is mentioned to persist connection information to facilitate this.⁷⁷
• The precise mechanisms for ensuring consistency and performing full state recovery for a large guild process after an unexpected node failure (beyond just restarting an empty process) are complex and not fully elaborated in the provided snippets ⁴⁸, but would be critical for system integrity. It likely involves careful coordination between the Elixir layer and the persistent metadata datastores.

Discord's Elixir-based state management architecture is a sophisticated system that balances in-memory performance for real-time operations with strategies for distributing load (relays) and sharing data efficiently (ETS). The interplay with persistent backend datastores is crucial for durability and recovery, forming a complete solution for managing state at extreme scale.

VIII. Key Architectural Takeaways and Concluding Remarks

Discord's journey to reliably serve billions of messages per day to a massive global user base is a compelling illustration of sophisticated engineering and continuous architectural evolution. Several key takeaways emerge from the analysis of their systems:

• Pragmatic Polyglotism: Discord's backend is not a monoculture. The strategic use of Elixir for its unparalleled real-time concurrency and fault tolerance, Rust for performance-critical data services requiring memory safety and low-level control, Python for the main API monolith (leveraging rapid development and a rich ecosystem), and Go for specific utilities like media processing, demonstrates a mature engineering culture that selects the best tool for each specific job. This pragmatic approach allows for optimized performance and developer productivity across different domains of the platform.
• Aggressive and Iterative Scaling of Data Storage: The evolution of the message store from MongoDB to Cassandra, and ultimately to ScyllaDB, is a clear testament to Discord's proactive approach to data scalability. Each migration was driven by emerging performance bottlenecks at new orders of magnitude. The move to ScyllaDB was not just a database replacement but involved significant re-architecture, including the introduction of Rust-based data services for request coalescing and the innovative "super-disk" storage topology on GCP. This shows a deep commitment to optimizing their most critical data pathway.
• Leveraging the Strengths of Elixir and the BEAM VM: The Erlang VM (BEAM) is fundamental to Discord's real-time capabilities. Its lightweight process model, isolated memory management, per-process garbage collection, and robust supervision strategies enable Discord to manage millions of concurrent WebSocket connections and fan out real-time events (presence, typing indicators, messages) with high efficiency and fault tolerance. Architectural patterns like per-guild and per-session processes, augmented by ETS for shared in-memory data and "relays" for load distribution in massive guilds, showcase a deep exploitation of BEAM's unique strengths.
• Continuous Architectural Evolution and Adaptation: Discord's architecture is not static; it has clearly evolved from its initial, simpler form. The shift from a purely monolithic approach with MongoDB to a hybrid system combining a central monolith with an expanding array of microservices reflects an iterative refinement process. New challenges, such as scaling specific features or integrating new technologies, are met with targeted architectural changes rather than wholesale rewrites, indicating a practical and sustainable approach to managing a complex, large-scale system.
• Strategic and Deep Utilization of Cloud Services: Discord's heavy reliance on Google Cloud Platform for its foundational infrastructure (compute, storage, networking, specialized data services like BigQuery and Pub/Sub) is a strategic choice. This allows them to leverage GCP's global scale, managed services (reducing operational overhead), and advanced capabilities. Their willingness to deeply customize IaaS components (e.g., GCE instances for "super-disks") while also adopting PaaS solutions demonstrates a balanced cloud strategy. This is further augmented by the use of Cloudflare for critical edge security (DDoS protection) and content delivery.
• Emphasis on Decoupling and Asynchronous Processing: The adoption of microservices, well-defined APIs between services, and the use of message queues (like Google Cloud Pub/Sub for search indexing and Elixir's GenStage for internal flow control) are crucial for decoupling system components. This enhances scalability by allowing independent scaling, improves resilience by isolating failures, and boosts performance by enabling asynchronous processing of background tasks.
• Commitment to Security and User Privacy: The development and rollout of the DAVE protocol for end-to-end encryption of audio and video communications is a significant investment in user privacy. This, combined with standard security practices like TLS for all data in transit, robust multi-factor authentication options, and a comprehensive Trust & Safety operation, underscores a serious commitment to protecting users and their data. While more transparency in engineering blogs regarding the specifics of data-at-rest encryption for their primary datastores would be beneficial, their stated policies indicate an intent to secure stored data.

Concluding Thoughts:

Discord's ability to handle its immense scale is not the result of a single silver bullet technology but rather the product of a carefully orchestrated symphony of diverse technologies, astute architectural decisions, and a culture of continuous engineering innovation. The platform's architecture is a living system, constantly being monitored, analyzed, and refined to meet the ever-growing demands of its global community. The journey from a rapidly prototyped application to a hyperscale distributed system offers invaluable lessons in pragmatic technology selection, iterative design, and the relentless pursuit of performance and reliability. As Discord continues to grow and introduce new features, its underlying architecture will undoubtedly continue to evolve, pushing the boundaries of what is possible in real-time communication at scale.

Works cited:

1. Real time communication at scale with Elixir at Discord, accessed May 12, 2025, https://elixir-lang.org/blog/2020/10/08/real-time-communication-at-scale-with-elixir-at-discord/
2. How Discord Can Serve Millions of Users From a Single Server - Quastor, accessed May 12, 2025, https://blog.quastor.org/p/discord-can-serve-millions-users-single-server
3. Elixir: Concurrency, Parallelism, & Fault-Tolerance - DEV Community, accessed May 12, 2025, https://dev.to/sandrockjustin/elixir-multi-threaded-fault-tolerant-33pk
4. Scaling Elixir at Discord for Massive Online User Loads, accessed May 12, 2025, https://elixirmerge.com/p/scaling-elixir-at-discord-for-massive-online-user-loads
5. How Discord Moved Engineering to Cloud Development Environments, accessed May 12, 2025, https://discord.com/blog/how-discord-moved-engineering-to-cloud-development-environments
6. Discord - Wikipedia, accessed May 12, 2025, https://en.wikipedia.org/wiki/Discord
7. How Discord Stores Trillions of Messages, accessed May 12, 2025, https://discord.com/blog/how-discord-stores-trillions-of-messages
8. How Discord Stores Trillions of Messages - ByteByteGo, accessed May 12, 2025, https://bytebytego.com/guides/how-discord-stores-trillions-of-messages/
9. Scaling Trillions of Messages: Discord's Journey from Cassandra to ScyllaDB with Rust-Powered Solutions - Blog - Saifeddine Rajhi, accessed May 12, 2025, https://seifrajhi.github.io/blog/discord-cassandra-to-scylladb/
10. Has Golang changed sinced Discord changed PL? : r/golang - Reddit, accessed May 12, 2025, https://www.reddit.com/r/golang/comments/11khs2z/has_golang_changed_sinced_discord_changed_pl/
11. Systems Design Interview: How to Design Discord - Java Challengers, accessed May 12, 2025, https://javachallengers.com/design-discord/
12. How Discord Seamlessly Upgraded Millions of Users to 64-Bit Architecture, accessed May 12, 2025, https://discord.com/blog/how-discord-seamlessly-upgraded-millions-of-users-to-64-bit-architecture
13. Microservices Architecture: Building APIs for Scalability - ThatAPICompany, accessed May 12, 2025, https://thatapicompany.com/microservices-architecture-building-apis-for-scalability/
14. How to Scale Your Game Servers for High-Traffic Events - Innovecs Games, accessed May 12, 2025, https://www.innovecsgames.com/blog/how-to-scale-your-game-servers-for-high-traffic-events/
15. API Reference | Documentation | Discord Developer Portal, accessed May 12, 2025, https://discord.com/developers/docs/reference
16. Gateway | Documentation | Discord Developer Portal, accessed May 12, 2025, https://discord.com/developers/docs/events/gateway
17. Configuring App Metadata for Linked Roles | Documentation | Discord Developer Portal, accessed May 12, 2025, https://discord.com/developers/docs/tutorials/configuring-app-metadata-for-linked-roles
18. How Discord Stores Billions of Messages, accessed May 12, 2025, https://discord.com/blog/how-discord-stores-billions-of-messages
19. Confused about Discord's reasoning for migrating from MongoDB - Reddit, accessed May 12, 2025, https://www.reddit.com/r/mongodb/comments/13ma6f2/confused_about_discords_reasoning_for_migrating/
20. How Discord Stores Billions of Messages : r/programming - Reddit, accessed May 12, 2025, https://www.reddit.com/r/programming/comments/5oynbu/how_discord_stores_billions_of_messages/
21. Discord, on the Joy of Opinionated Systems - ScyllaDB, accessed May 12, 2025, https://www.scylladb.com/2019/03/20/discord-on-the-joy-of-opinionated-systems/
22. The Case for Shared Storage - WarpStream, accessed May 12, 2025, https://www.warpstream.com/blog/the-case-for-shared-storage
23. How Discord Migrated Trillions of Messages to ScyllaDB - The New Stack, accessed May 12, 2025, https://thenewstack.io/how-discord-migrated-trillions-of-messages-to-scylladb/
24. How Discord Supercharges Network Disks for Extreme Low Latency, accessed May 12, 2025, https://discord.com/blog/how-discord-supercharges-network-disks-for-extreme-low-latency
25. How Discord Creates Insights from Trillions of Data Points, accessed May 12, 2025, https://discord.com/blog/how-discord-creates-insights-from-trillions-of-data-points
26. Discord Blog, accessed May 12, 2025, https://discord.com/category/engineering
27. Recent Instability & What's Next - Discord, accessed May 12, 2025, https://discord.com/blog/recent-instability-whats-next
28. Senior Software Engineer- Database Infrastructure - Discord | Built In, accessed May 12, 2025, https://builtin.com/job/senior-software-engineer-database-infrastructure/4823230
29. G.V() Brings Interactive Graph Visualization To Google Cloud's Spanner Graph, accessed May 12, 2025, https://gdotv.com/blog/google-cloud-spanner-graph-release/
30. Google Cloud SQL - Hyperdrive - Cloudflare Docs, accessed May 12, 2025, https://developers.cloudflare.com/hyperdrive/examples/connect-to-postgres/google-cloud-sql/
31. IAM overview | Spanner | Google Cloud, accessed May 12, 2025, https://cloud.google.com/spanner/docs/iam
32. Google Spanner | LangChain, accessed May 12, 2025, https://python.langchain.com/v0.1/docs/integrations/document_loaders/google_spanner/
33. Get Bucket Metadata with Google Cloud API on New Forum Thread Message from Discord Bot API - Pipedream, accessed May 12, 2025, https://pipedream.com/apps/discord-bot/integrations/google-cloud/get-bucket-metadata-with-google-cloud-api-on-new-forum-thread-message-from-discord-bot-api-int_pQsEzL6o
34. Google Cloud SQL for PostgreSQL - PostgresVectorStore - LlamaIndex, accessed May 12, 2025, https://docs.llamaindex.ai/en/stable/examples/vector_stores/CloudSQLPgVectorStoreDemo/
35. Does a distributed database like CockroachDB etc makes sense for a product that's local-only in a country with zero plans for global expansion? : r/AskProgramming - Reddit, accessed May 12, 2025, https://www.reddit.com/r/AskProgramming/comments/1ihf6wd/does_a_distributed_database_like_cockroachdb_etc/
36. Compare CockroachDB, accessed May 12, 2025, https://www.cockroachlabs.com/compare/
37. How Discord Uses Open-Source Tools for Scalable Data Orchestration & Transformation, accessed May 12, 2025, https://discord.com/blog/how-discord-uses-open-source-tools-for-scalable-data-orchestration-transformation
38. How Discord Indexes Trillions of Messages, accessed May 12, 2025, https://discord.com/blog/how-discord-indexes-trillions-of-messages
39. Overclocking dbt: Discord's Custom Solution in Processing ..., accessed May 12, 2025, https://discord.com/blog/overclocking-dbt-discords-custom-solution-in-processing-petabytes-of-data
40. Load balancing and distributed architectures - Comprehensive Guide to Discord Bot Development with discord.py | StudyRaid, accessed May 12, 2025, https://app.studyraid.com/en/read/7183/176836/load-balancing-and-distributed-architectures
41. Cloud Load Balancing overview | Google Cloud, accessed May 12, 2025, https://cloud.google.com/load-balancing/docs/load-balancing-overview
42. Cloud Load Balancing | Google Cloud, accessed May 12, 2025, https://cloud.google.com/load-balancing
43. Enhance your Application Infrastructure with Google Cloud's Load Balancer, accessed May 12, 2025, https://www.appsecengineer.com/blog/enhance-your-application-infrastructure-with-google-clouds-load-balancer
44. Using WebSockets | Cloud Run Documentation - Google Cloud, accessed May 12, 2025, https://cloud.google.com/run/docs/triggering/websockets
45. External Application Load Balancer overview - Google Cloud, accessed May 12, 2025, https://cloud.google.com/load-balancing/docs/https
46. Why Use WebSockets? Real-Time Communication Explained ..., accessed May 12, 2025, https://www.videosdk.live/developer-hub/websocket/why-use-websocket
47. Discord: The Tech Community's Powerhouse Platform - Technikole Consulting, accessed May 12, 2025, https://technikole.com/discord-the-tech-communitys-powerhouse-platform/
48. Maxjourney: Pushing Discord's Limits with a Million+ Online Users in ..., accessed May 12, 2025, https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server
49. Discord Scales to 1 Million+ Online MidJourney Users in a Single ..., accessed May 12, 2025, https://www.infoq.com/news/2024/01/discord-midjourney-performance/
50. WebSockets vs Server-Sent-Events vs Long-Polling vs WebRTC vs WebTransport | RxDB - JavaScript Database, accessed May 12, 2025, https://rxdb.info/articles/websockets-sse-polling-webrtc-webtransport.html
51. Discord DAVE Protocol Whitepaper, accessed May 12, 2025, https://daveprotocol.com/
52. Meet DAVE: Discord's New End-to-End Encryption for Audio & Video, accessed May 12, 2025, https://discord.com/blog/meet-dave-e2ee-for-audio-video
53. Three CDN Strategies To Lower Live Streaming Latency, accessed May 12, 2025, https://www.streamingmedia.com/Articles/Post/Blog/Three-CDN-Strategies-To-Lower-Live-Streaming-Latency-168673.aspx
54. What Is CDN? The Complete Guide for DevOps Engineers - Last9, accessed May 12, 2025, https://last9.io/blog/what-is-cdn/
55. Designing Content Delivery Network (CDN) | System Design - GeeksforGeeks, accessed May 12, 2025, https://www.geeksforgeeks.org/designing-content-delivery-network-cdn-system-design/
56. The Best 5 Google CDN Alternatives - IO River, accessed May 12, 2025, https://www.ioriver.io/blog/google-cdn-alternatives
57. What hosting provider does Discord use? : r/discordapp - Reddit, accessed May 12, 2025, https://www.reddit.com/r/discordapp/comments/160ermf/what_hosting_provider_does_discord_use/
58. Is Discord a SaaS? : r/Cloud - Reddit, accessed May 12, 2025, https://www.reddit.com/r/Cloud/comments/ng6ey1/is_discord_a_saas/
59. How to speed up images loading adding a CDN for a Google Cloud Bucket?, accessed May 12, 2025, https://stackoverflow.com/questions/79442217/how-to-speed-up-images-loading-adding-a-cdn-for-a-google-cloud-bucket
60. Media CDN overview | Google Cloud, accessed May 12, 2025, https://cloud.google.com/media-cdn/docs/overview
61. Content delivery best practices | Cloud CDN, accessed May 12, 2025, https://cloud.google.com/cdn/docs/best-practices
62. End-to-End Encryption for Audio and Video - Discord Support, accessed May 12, 2025, https://support.discord.com/hc/en-us/articles/25968222946071-End-to-End-Encryption-for-Audio-and-Video
63. Encryption at Rest | ScyllaDB Docs, accessed May 12, 2025, https://enterprise.docs.scylladb.com/stable/operating-scylla/security/encryption-at-rest.html
64. Getting Started with Database-Level Encryption at Rest in ScyllaDB Cloud, accessed May 12, 2025, https://www.scylladb.com/2024/07/09/getting-started-with-database-level-encryption-at-rest-in-scylladb-cloud/
65. How ScyllaDB Cloud Protects Your Sensitive Data, accessed May 12, 2025, https://www.scylladb.com/wp-content/uploads/scylladb-protecting-your-sensitive-data.pdf
66. Encryption at Rest | ScyllaDB Docs, accessed May 12, 2025, https://docs.scylladb.com/manual/stable/operating-scylla/security/encryption-at-rest.html
67. Encryption at Rest in ScyllaDB Enterprise, accessed May 12, 2025, https://www.scylladb.com/2019/07/11/encryption-at-rest-in-scylla-enterprise/
68. Discord Best Practices: Guidelines on how to keep Discord safe and secure - DotCIO, accessed May 12, 2025, https://itssc.rpi.edu/hc/en-us/articles/32018134944013-Discord-Best-Practices-Guidelines-on-how-to-keep-Discord-safe-and-secure
69. Transitioning Discord's Engineering Team to Cloud Development ..., accessed May 12, 2025, https://www.infoq.com/news/2024/03/discord-cloud-development-env/
70. Discord Transparency Report: April - June 2022, accessed May 12, 2025, https://discord.com/blog/discord-transparency-report-q2-2022
71. Discord Transparency Report: January - March 2022, accessed May 12, 2025, https://discord.com/blog/discord-transparency-report-q1-2022
72. Bigtable: Fast, Flexible NoSQL | Google Cloud, accessed May 12, 2025, https://cloud.google.com/bigtable
73. Google Kubernetes Engine: Architecture, Pricing & Best Practices - Spot.io, accessed May 12, 2025, https://spot.io/resources/google-kubernetes-engine/google-kubernetes-engine-architecture-pricing-best-practices/
74. Google Kubernetes Engine (GKE), accessed May 12, 2025, https://cloud.google.com/kubernetes-engine
75. Discord and Google Cloud Storage integration - N8N, accessed May 12, 2025, https://n8n.io/integrations/discord/and/google-cloud-storage/
76. Bigtable documentation - Google Cloud, accessed May 12, 2025, https://cloud.google.com/bigtable/docs/
77. Looking for Alternative Designs to Discord's Architecture : r/ExperiencedDevs - Reddit, accessed May 12, 2025, https://www.reddit.com/r/ExperiencedDevs/comments/1gmf7yg/looking_for_alternative_designs_to_discords/
78. How to Add Typing Indicators on Discord Server - UnderConstructionPage, accessed May 12, 2025, https://underconstructionpage.com/how-to-add-typing-indicators-on-discord-server/
79. How to Add Typing Indicators on Discord Server - UnderConstructionPage, accessed May 12, 2025, https://underconstructionpage.com/how-to-add-typing-indicators-on-discord-server-2/