/r/RedditEng
Welcome to the Reddit Tech Blog. We'll be talking about how we contribute to Reddit's mission to bring community and belonging to everyone. Read about how we build stuff people love and what it's like to be part of our team.
If you're interested in working with us, go to our careers page: https://www.redditinc.com/careers
Reddit Engineering Tech Blog
/r/RedditEng
12,076 Subscribers
Scaling Ads Serving: Find and Eliminate Redundant Operations
Written by Andy Zhang and Divya Bala
Introduction
The Ad Serving Platform team is thrilled to bring you this behind-the-scenes look at Reddit’s ad-serving system! Our team has the humble yet powerful job of keeping the ad magic running smoothly so that Reddit Ad’s various product teams can continue dazzling the world with endless possibilities.
Here’s what our team is responsible for:
- Ad Serving Infrastructure: We’re the architecture and operational excellence gurus, making sure our infrastructure is built like a skyscraper but flexible as a rubber band. Our system’s elasticity is crucial to our partner teams, allowing them to run their ad selection models with the reliability of your morning coffee.
- Ad Serving Platform: We own the platform that makes executing vertical teams’ models as seamless as possible. Think of us as the tech world’s “easy button” for integrating new products, simplifying onboarding, and providing robust tools for debugging when things inevitably get too exciting.
Over the past few years, our team has tackled some mission-critical projects to ensure our system remains as scalable and reliable as the Reddit communities it supports. In this post, we’ll share a few of the scaling challenges we’ve encountered, plus a recent project where we boosted system availability while reducing infrastructure cost (yes, it is possible). We hope our journey gives you some fresh ideas and maybe a little inspiration for scaling your own systems.
A brief history of Reddit Ad Serving
The functional requirements of Reddit’s Ad Serving system are refreshingly simple:
- Accept front-end requests and produce a curated set of ads.
- Incorporate various products to maximize advertisers’ ROI while keeping users engaged and interested (instead of exasperated).
Like many backend systems, we began with a simple, single-service setup that handled all the ad selection tasks in a neat little package. But as our customer base (advertisers) began to grow like Reddit comment threads, scaling limitations hit fast. Those O(N) operations that once worked smoothly started feeling like they were running on yesterday’s Wi-Fi.
So, the next logical step? Sharding our customer base. This kicked off a series of redesign phases to keep our ad-serving system humming efficiently, no matter how much our business continues to climb.
Introduction
The Ad Serving Platform team is thrilled to bring you this behind-the-scenes look at Reddit’s ad-serving system! Our team has the humble yet powerful job of keeping the ad magic running smoothly so that Reddit Ad’s various product teams can continue dazzling the world with endless possibilities
The challenges in scaling
With service architecture v2.1, we’re set up to handle some of the most resource-intensive operations—like expensive targeting and complex modeling—in a separate, scalable service dedicated to a subset of advertisers. This way, we can scale these processes independently from the Ad Selector and other shards, giving our main systems some much-needed breathing room.
But scaling isn’t just about where we store and process our data. Sometimes, it’s about how seamlessly products are integrated into the request workflow. When a product starts playing a starring role in workflow orchestration, it’s all too easy to overlook the “hidden” costs lurking in the background. Just like adding extra cheese to a pizza, a little overhead can be manageable—but too much, and suddenly you’ve got a system that’s weighed down and sluggish.
Design and Redesign
Select a single ad
The roles of Ad Selector and Ad Shards are clear and complementary:
- Ad Selector: Like a highly skilled traffic cop, Ad Selector validates and enriches incoming requests with extra context, sends them off to the individual shards, and then gathers all the responses to deliver the final ad lineup.
- Ad Shards: Each shard is a busy hive of activity, running a series of actions to choose local winners and executing a host of models from various teams to help identify the best ad candidates. Think of Ad Shards as the talent scouts of our system, making sure only the best ads make it to the spotlight.
- The challenges in scaling
With service architecture v2.1, we’re set up to handle some of the most resource-intensive operations—like expensive targeting and complex modeling—in a separate, scalable service dedicated to a subset of advertisers. This way, we can scale these processes independently from the Ad Selector and other shards, giving our main systems some much-needed breathing room
The illustration above demonstrates how we select an ad to be displayed in a designated location.
Select multiple ads
When it comes to filling multiple ad slots at once, things get a bit more complex:
- Not every ad is eligible for every slot.
- And not every ad performs equally well across all slots.
To ensure each slot maximizes advertiser ROI, we designed a specialized workflow that filters ads by eligibility for each position and scores them accurately during ranking. And here’s a key point: just because an ad doesn’t make the cut for one position doesn’t mean it’s out of the game for another slot. After all, everyone deserves a second chance, especially ads!
The workflow looks something like this:
This design utilizes the majority of the code and workflow when the concept is initially formed. We simply provide slot specific context to each shard request, and let the filtering process respect each slot context, and job done.
Identify the problem
While slot-specific processing gives ads more chances to be evaluated at the request level (great for business!), we noticed a big uptick in the load on our ad shard services. This increased load means our heavy models get invoked more frequently, putting a serious demand on our cluster’s resources.
When scaling issues come from all sides—more DAUs, more advertisers, and stricter SLAs—it’s tempting to dive into code optimizations, compromise on latency to keep availability high, or even throw more infrastructure dollars at the problem, hoping it all smooths out eventually.
But here’s the thing: sometimes, no amount of extra infrastructure can fix the bottlenecks. Your cluster might hit its node scheduling limits, adding more shards could start backfiring on upstream services, and that delicate balance between latency and availability gets harder and harder to manage.
So, what do you do?
Well, we took a step back. Instead of throwing more resources at it, we analyzed our request workflow to see if it was as efficient as we assumed. And guess what? The opportunities for improvement were much bigger than we’d anticipated.
The fix
Per-slot ad selection gives us precisely the right ads for each slot’s unique context, and that’s essential to the product. But here’s the twist: only a small slice of the actions in this selection process actually impact this “precision cut” in filtering out ineligible ads.
So, our solution? Trim out redundant operations that don’t influence outcomes or add any real business value at the per-slot level.
Here’s how we tackled it:
- In the parallel ad sourcing stage – None of the candidate sources need slot-level information here. What really matters is user context—interests, device type, that sort of thing. Slot-level specifics are just extra weight at this stage.
- At the filtering level – Less than 5% of actions, like brand safety checks or negative keyword filtering, actually need to be slot-aware. These are tied to slot context only to ensure sensitive content doesn’t accidentally end up above or below certain posts.
- In heavy model execution – Turns out, a different feature with much lower cardinality can get us the same results, letting us cut down on model invocations without losing accuracy. It’s like upgrading to a more efficient tool without sacrificing quality.
- Finally, the ranking process – Here, slot-awareness is essential. Each candidate ad has different opportunities depending on the slot it’s aiming for, so we keep this step fully slot-aware to get the right ads in the right places.
By rewiring the execution pipeline this way, we’ve brought the Adserver Shard pipeline’s workload down from O(N)—where N is the number of slots—to a sleek O(1). In doing so, we’ve stripped away a hefty portion of the execution overhead, and significantly lightened the service’s networking and middleware load. It’s like switching from rush hour traffic to an express lane—smoother, faster, and way less stressful on the system.
How we did it
To implement this project, we divided it into two parts. We opted for this approach because our serving system is highly dynamic, with multiple teams continuously contributing to the codebase. This creates challenges in making progress while keeping the live system stable and avoiding discrepancies.
Phase 1
In the first phase, we introduced new Thrift APIs for RPC calls to handle both global and slot-specific metadata. These requests were sent to AdServer Shards, where they were converted into multiple legacy requests and processed through the old pipeline in parallel.
Once the local auction results were gathered, they were parsed and merged into the new response API, minimizing changes to the shards and relying on the existing integration test suite.
Additionally, in Ad-Selector, we introduced stages to logically organize request handling, with each stage returning a unique struct response. This allowed for independent unit testing. It also provided valuable analytics and diagnostics data around global auction results at each stage.Identify the problem
While slot-specific processing gives ads more chances to be evaluated at the request level (great for business!), we noticed a big uptick in the load on our ad shard services. This increased load means our heavy models get invoked more frequently, putting a serious demand on our cluster’s resources
Phase 2
In the second phase, we removed the looping logic and legacy requests in AdServer Shard, replacing them with a new pipeline that could select ad candidates and apply slot-specific filtering and ranking. This streamlined the process, eliminating unnecessary repetition of business logic
The result
The final results from this effort were truly exciting, with large-scale operational efficiency gains across our entire serving stack:
- QPS to the Adserver Shard pipeline dropped by about 50%, cutting network-in traffic by 50% and network-out by 35%.
- QPS to our heavy model inference server dropped by 42%, giving us valuable headroom before hitting cluster capacity.
- Availability increased significantly thanks to fewer operations required per request, reducing the chance of failures.
On the cost side:
- Resource allocation for Ad Selector dropped by 30%, primarily from needing fewer Adserver Shard connections and spending less time on long-tail requests.
- Shard costs dropped by nearly 50% thanks to a lighter workload.
- Inference server costs fell by around 35%, with additional savings from reduced storage layer lookups and lowered network overhead.
All told, this optimization translates to millions in annual infrastructure savings and a substantial boost in cluster capacity, which also unblocks compute power for other product developments.
What we learned (and what we hope you'd learn from us)
Designing a scalable system is challenging, especially when it’s highly distributed with many moving parts. In a fast-paced engineering environment, we often focus heavily on techniques, tools, and the quickest route to achieving our business goals.
Hopefully, this post serves as a reminder that smart request pattern design is equally critical and can drive fundamental improvements across the system.
Special thanks to contributors to this project: Divya Bala, Emma Luukkonen, Rachael Morton, Tim Zhu, Gopai Rajpurohit, Yuxuan Wang, Andy Zhang
21:21 UTC
Product Candidate Generation for Reddit Dynamic Product Ads
Written by Simon Kim, Sylvia Wu, and Shivaram Lingamneni.
Reddit Shopping Ads Business
At Reddit, Dynamic Product Ads (DPA) plays a crucial part in putting shopping into context. DPA aims to serve the right product, to the right person at the right time on Reddit. The dynamic, personalized ads experience helps users to explore and purchase products they are interested in and makes it easier for advertisers to drive purchases.
After advertisers upload their product catalog, Dynamic Product Ads (DPA) allows advertisers to define an ad group with a set of products and let Reddit ML dynamically generate relevant products to serve at the time of request.
For example, an advertiser selling beauty products might upload a product catalog that ranges from skin care, hair care to makeup. When there is an ad request in a Reddit post seeking advice about frizzy hair, Reddit will dynamically construct a shopping ad from the catalog on behalf of the advertiser by generating relevant product candidates such as hair serum and hair oil products.
This article will delve into the DPA funnel with a focus on product candidate generation, covering its methods, benefits, and future directions.
Funnel Overview for DPA
The Dynamic Product Ads (DPA) funnel consists of several key stages that work together to deliver relevant product advertisements to users. At a high level, the funnel begins with Targeting, which defines the audience and determines who will see the ads based on various criteria, such as demographics, device or location.
Once the audience is targeted, the next step is Product Candidate Generation. This process involves generating a broad set of potential products that might be relevant to the targeted ad request. Here, a wide array of products is identified based on factors like historical engagement, content preference, product category etc.
Then, the funnel proceeds to Product Selection, where products are ranked and filtered based on various relevance and performance metrics. This light selection phase ensures that the most relevant products are presented to users.
Finally, the selected products enter the Auction stage, where an auction-based system determines which products will be shown based on bids, ad relevance, and other factors.
Why and What is Candidate Generation in DPA?
Compared to static ads, the key challenge faced by DPA is the ability to dynamically generate relevant products from hundreds of millions of products tailored to the current context, with low latency and at scale. It is impractical to do an extensive search in the vast candidate pool to find the best product for each ad request. Instead, our solution is to employ multiple candidate selectors to source products that are more likely to be recommended at the ranking stage. The candidate selectors can cover different aspects of an ad request, such as the user, the subreddit, the post, and the contextual information, and source corresponding relevant products. This way, we can narrow down a vast pool of potential product options to a manageable set of only relevant and high-potential products that are passed through the funnel, saving the cost for future evaluation while preserving the relevance of the recommendations.
Candidate Generation Approaches
At Reddit, we have developed an extensive list of candidate selectors that capture different aspects of the ad request, and work together to yield the best performance. We categorize the selectors in two dimensions, modeling and serving.
Modeling:
- Rule-Based Selection selects items based on rule-based scores, such as popular products, trending products, etc.
- Contextual-Based Selection emphasizes relevance between the product and the Reddit context, such as the subreddit and the post. For example, in a camping related post, contextual-based selectors will retrieve camping related products using embeddings search or keywords matching between post content and product descriptions.
- Behavioral-Based Selection optimizes purchase engagement between the user and the product by capturing implicit user preferences and user-product interaction history.
Currently, we use a combination of the above as they cover different aspects of the ad request and complement each other. Contextual-based models shine in conversational contexts, whereas product recommendations closely align with the user’s interest at the moment, and behavioral-based models capture the user engagement behavior and provide more personalization. We also found that while not personalized, rule-based candidates ensure content availability to alleviate cold-start problems, and allow a broader user reach and exploration in recommendations.
Serving:
- Offline methods precompute the product matching offline, and store the pre-generated pairs in databases for quick retrieval.
- Online methods conduct real-time matching between ad requests and the products, such as using Approximate Nearest Neighbor (ANN) Search to find product embeddings given a query embedding.
Both online and offline serving techniques have unique strengths in candidate generation and we adopt them for different scenarios. The offline method excels in speed and allows more flexibility in the model architectures and the matching techniques. However, it requires considerable storage, and the matching might not be available for new content and new user actions due to the lag in offline processing, while it stores recommendations for users or posts that are infrequently active. The online method can achieve higher coverage by providing high quality recommendations for fresh content and new user behaviors immediately. It also has access to real-time contextual information such as the location and time of day to enrich the model.but it requires more complex infrastructure to handle on-the-fly matching and might face latency issues.
A Closer Look: Online Approximate Nearest Neighbor Search with Behavioral-Based Two-Tower Model
Below is a classic example of candidate generation for DPA. When a recommendation is requested, the user’s features are fed through the user tower to produce a current user embedding. This user embedding is then matched against the product embeddings index with Approximate Nearest Neighbor (ANN) search to find products that are most similar or relevant, based on their proximity in the embedding space.
It enables real-time and highly personalized product recommendations by leveraging deep learning embeddings and rapid similarity searches. Here’s a deeper look at each of component:
Model Deep Dive
The two-tower model is a deep learning architecture commonly used for candidate generation in recommendation systems. The term "two-tower" refers to its dual structure, where one tower represents the user and the other represents the product. Each tower independently processes features related to its entity (user or product) and maps them to a shared embedding space.
Model Architecture, Features, and Labels
- User and Product Embeddings:
- The model takes in user-specific features (e.g., engagement, platform etc) and product-specific features (e.g., price, catalog, engagement etc).
- These features are fed into separate neural networks or "towers," each producing an embedding - a high-dimensional vector - that represents the user or product in a shared semantic space.
- Training with Conversion Events:
- The model is trained on past conversion events
- In-batch negative sampling is also used to further refine the model, increasing the distance between unselected products and the user embedding.
Model Training and Deployment
We developed the model training pipeline leveraging our in-house TTSN (Two Tower Sparse Network) engine. The model is retrained daily on Ray. Once daily retraining is finished, the user tower and product tower are deployed separately to dedicated model servers. You can find more details about Gazette and our model serving workflow in one of our previous posts.
Serving Deep Dive
Online ANN (Approximate Nearest Neighbor) Search
Unlike traditional recommendation approaches that might require exhaustive matching, ANN (Approximate Nearest Neighbor) search finds approximate matches that are computationally efficient and close enough to be highly relevant. ANN search algorithms are able to significantly reduce computation time by clustering similar items and reducing the search space.
After careful exploration and evaluation, the team decided to use FAISS (Facebook AI Similarity Search). Compared to other methods, the FAISS library provides a lot of ways to get optimal performance and balance between index building time, memory consumption, search latency and recall.
We developed an ANN sidecar that implements an ANN index and API to build product embeddings and retrieve N approximate nearest product embeddings given a user embedding. The product index sidecar container is packed together with the main Product Ad Shard container in a single pod.
Product Candidate Retrieval Workflow with Online ANN
Imagine a user browsing Home Feed on Reddit, triggering an ad request for DPA to match relevant products to the user. Here’s the retrieval workflow:
Real-Time User Embedding Generation:
- When an ad request comes in, the Ad Selector sends a user embedding generation request to the Embedding Service.
- Embedding Service constructs and sends the user embedding request along with real-time contextual features to the inference server which connects to the user tower model server and feature store and returns the user embedding. Alternatively, if this user request has been scored recently within 24 hrs, retrieve it from the cache instead.
- Ad selector passes the generated user embedding to Shopping Shard, and then Product Ad Shard.
Async Batch Product Embedding Generation:
- Product Metadata Delivery service pulls from Campaign Metadata Delivery service and Catalog Service to get all live products from live campaigns.
- At a scheduled time, Product Metadata Delivery service sends product embedding generation requests in batches to Embedding Service. The batch request includes all the live products retrieved from the last step.
- Embedding Service returns batched product embeddings scored from the product tower model.
- Product Metadata Delivery service publishes the live products metadata and product embeddings to Kafka to be consumed by Product Ad Shard.
Async ANN Index Building
- The Product Index is stored in the ANN sidecar within Product Ad Shard. The ANN Sidecar will be initialized with all the live product embeddings from PMD, and then refreshed every 30s to add, modify, or delete product embeddings to make the index space up-to-date.
Candidate Generation and Light Ranking:
- The Product Ad Shard collects request contexts from upstream services (eg, Shopping Shard), including user embedding, and makes requests to all the candidate selectors to return recommended candidate products, including the online behavioral-based selector.
- The online behavioral-based selector makes a local request to the ANN Sidecar to get top relevant products. The ANN search quickly compares this user embedding with the product embeddings index space, finding the approximate nearest neighbors. It’s important to ensure the embedding version is matched between the user embedding and the product embedding index.
- All the candidate products are unioned and go through a light ranking stage in Product Ad Shard to determine the final set of ads the user will see. The result will be passed back to the upstream services to construct DPA ads and participate in final auctions.
Impact and What’s Next
By utilizing rule-based, contextual-based and behavioral-based candidate selectors with online and offline serving, we provide comprehensive candidate generation coverage and high quality product recommendations at scale, striking a balance between speed, accuracy, and relevance. The two-tower model and online ANN search, in particular, enable real-time and highly personalized recommendations, adapting dynamically to user behaviors and product trends. It helps advertisers to see higher engagement and ROAS (Return over Ad Spend), while users receive ads that feel relevant to their immediate context and interests.
The modeling and infrastructure development in Reddit DPA has been growing rapidly in the past few months - we have launched tons of improvements that cumulatively yield more than doubled ROAS and tripled user reach, and there are still many more exciting projects to explore!
We would also like to thank the DPA v-team: Tingting Zhang, Marat Sharifullin, Andy Zhang, Hanyu Guo, Marcie Tran, Xun Zou, Wenshuo Liu, Gavin Sellers, Daniel Peters, Kevin Zhu, Alessandro Tiberi, Dinesh Subramani, Matthew Dornfeld, Yimin Wu, Josh Cherry, Nastaran Ghadar, Ryan Sekulic, Looja Tuladhar, Vinay Sridhar, Sahil Taneja, and Renee Tasso.
21:30 UTC
Open Source of Achilles SDK
Harvey Xia and Karan Thukral
TL;DR
We are thrilled to announce that Reddit is open sourcing the Achilles SDK, a library for building Kubernetes controllers. By open sourcing this library, we hope to share these ideas with the broader ecosystem and community. We look forward to the new use cases, feature requests, contributions, and general feedback from the community! Please visit the achilles-sdk repository to get started. For a quickstart demo, see this example project.
What is the Achilles SDK?
At Reddit we engineer Kubernetes controllers for orchestrating our infrastructure at scale, covering use cases ranging from fully managing the lifecycle of opinionated Kubernetes clusters to managing datastores like Redis and Cassandra. The Achilles SDK is a library that empowers our infrastructure engineers to build and maintain production grade controllers.
The Achilles SDK is a library built on top of controller-runtime. By introducing a set of conventions around how Kubernetes CRDs (Custom Resource Definitions) are structured and best practices around controller implementation, the Achilles SDK drastically reduces the complexity barrier when building high quality controllers.
The defining feature of the Achilles SDK is that reconciliation (the business logic that ensures actual state matches desired intent) is modeled as a finite state machine. Reconciliation always starts from the FSM’s first state and progresses until reaching a terminal state.
Modeling the controller logic as an FSM allows programmers to decompose their business logic in a principled fashion, avoiding what often becomes an unmaintainable, monolithic Reconcile() function in controller-runtime-backed controllers. Reconciliation progress through the FSM states are reported on the custom resource’s status, allowing both humans and programs to understand whether the resource was successfully processed.
Why did we build the Achilles SDK?
2022 was a year of dramatic growth for Reddit Infrastructure. We supported a rapidly growing application footprint and had ambitions to expand our serving infrastructure across the globe. At the time, most of our infrastructure was hand-managed and involved extremely labor-intensive processes, which were designed for a company of much smaller scope and scale. Handling the next generation of scale necessitated that we evolve our infrastructure into a self-service platform backed by production-grade automation.
We chose Kubernetes controllers as our approach for realizing this vision.
- Kubernetes was already tightly integrated into our infrastructure as our primary workload orchestrator.
- We preferred its declarative resource model and believed we could represent all of our infrastructure as Kubernetes resources.
- Our core infrastructure stack included many open source projects implemented as Kubernetes controllers (e.g. FluxCD, Cluster Autoscaler, KEDA, etc.).
All of these reasons gave us confidence that it was feasible to use Kubernetes as a universal control plane for all of our infrastructure.
However, implementing production-grade Kubernetes controllers is expensive and difficult, especially for engineers without extensive prior experience building controllers. That was the case for Reddit Infrastructure in 2022—the majority of our engineers were more familiar with operating Kubernetes applications than building them from scratch.
For this effort to succeed, we needed to lower the complexity barrier of building Kubernetes controllers. Controller-runtime is a vastly impactful project that has enabled the community to build a generation of Kubernetes applications handling a wide variety of use cases. The Achilles SDK takes this vision one step further by allowing engineers unfamiliar with Kubernetes controller internals to implement robust platform abstractions.
The SDK reached general maturity this year, proven out by wide adoption internally. We currently have 12 Achilles SDK controllers in production, handling use cases ranging from self-service databases to management of Kubernetes clusters. An increasing number of platform teams across Reddit are choosing this pattern for building out their platform tooling. Engineers with no prior experience with Kubernetes controllers can build proof of concepts within two weeks.
Features
Controller-runtime abstracts away the majority of controller internals, like client-side caching, reconciler actuation conditions, and work queue management. The Achilles SDK, on the other hand, provides abstraction at the application layer by introducing a set of API and programming conventions.
Highlights of the SDK include:
- Modeling reconciliation as a finite state machine (FSM)
- “Ensure” style resource updates
- Automatic management of owner references for child resources
- CR status management
- Tracking child resources
- Reporting reconciliation success or failure through status conditions
- Finalizer management
- Static tooling for suspending/resuming reconciliation
- Opinionated logging and metrics
Let’s walk through these features with code examples.
Defining a Finite State Machine
The SDK represents reconciliation (the process of mutating the actual state towards the desired state) as an FSM with a critical note—each reconciliation invokes the first state of the FSM and progresses until termination. The reconciler does not persist in states between reconciliations. This ensures that the reconciler’s view of the world never diverges from reality—its view of the world is observed upon each reconciliation invocation and never persisted between reconciliations.
Let’s look at an example state below:
type state = fsmtypes.State[*v1alpha1.TestCR]
type reconciler struct {
log *zap.SugaredLogger
c *io.ClientApplicator
scheme *runtime.Scheme
}
func (r *reconciler) createConfigMapState() *state {
return &state{
Name: "create-configmap-state",
Condition: achillesAPI.Condition{
Type: CreateConfigMapStateType,
Message: "ConfigMap created",
},
Transition: r.createCMStateFunc,
}
}
func (r *reconciler) createCMStateFunc(
ctx context.Context,
res *v1alpha1.TestCR,
out *fsmtypes.OutputSet,
) (*state, fsmtypes.Result) {
configMap := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: res.GetName(),
Namespace: res.GetNamespace(),
},
Data: map[string]string{
"region": res.Spec.Region,
"cloud": ,
},
}
// Resources added to the output set are created and/or updated by the sdk after the state transition function ends.
// The SDK automatically adds an owner reference on the ConfigMap pointing
// at the TestCR parent object.
out.Apply(configMap)
// The reconciler can conditionally execute logic by branching to different states.
if res.conditionB() {
return r.stateB(), fsmtypes.DoneResult()
}
return r.stateC(), fsmtypes.DoneResult()
}A CR of type TestCR is being reconciled. The first state of the FSM, createConfigMapState, creates a ConfigMap with data obtained from the CR’s spec. An achilles-sdk state has the following properties:
- Name: unique identifier for the state
- used to ensure there are no loops in the FSM
- used in logs and metrics
- Condition: data persisted to the CR’s status reporting the success or failure of this state
- Transition: the business logic
- defines the next state to transition to (if any)
- defines the result type (whether this state completed successfully or failed with an error)
We will cover some common business logic patterns.
Modifying the parent object’s status
Reconciliation often entails updating the status of the parent object (i.e. the object being reconciled). The SDK makes this easy—the programmer mutates the parent object (in this case res *v1alpha1.TestCR) passed into the state struct and all mutations are persisted upon termination of the FSM. We deliberately perform status updates at the end of the FSM rather than in each state to avoid livelocks caused by programmer errors (e.g. if two different states both mutate the same field to conflicting values the controller would be continuously triggered).
func (r *reconciler) modifyParentState() *state {
return &state{
Name: "modify-parent-state",
Condition: achillesAPI.Condition{
Type: ModifyParentStateType,
Message: "Parent state modified",
},
Transition: r.modifyParentStateFunc,
}
}
func (r *reconciler) modifyParentStateFunc(
ctx context.Context,
res *v1alpha1.TestCR,
out *fsmtypes.OutputSet,
) (*state, fsmtypes.Result) {
res.Status.MyStatusField = “hello world”
return r.nextState(), fsmtypes.DoneResult()
}Creating and Updating Resources
Kubernetes controllers’ implementations usually include creating child resources (objects with a metadata.ownerReference to the parent object). The SDK streamlines this operation by providing the programmer with an OutputSet. At the end of each state, all objects inserted into this set will be created or updated if they already exist. These objects will automatically obtain a metadata.ownerReference to the parent object. Conversely, the parent object’s status will contain a reference to this child object. Having these bidirectional links allows system operators to easily reason about relations between resources. It also enables building more sophisticated operational tooling for introspecting the state of the system.
The SDK supplies a client wrapper (ClientApplicator) that provides “apply” style update semantics—the ClientApplicator only updates the fields declared by the programmer. Non-specified fields (e.g. nil fields for pointer values, slices, and maps) are not updated. Specified but zero fields (e.g. [] for slice fields, {} for maps, 0 for numeric types, ””for string types) signal deletion of that field. There’s a surprising amount of complexity in serializing/deserializing YAML as it pertains to updating objects. For full discussion of this topic, see this doc.
This is especially useful in cases where multiple actors manage mutually exclusive fields on the same object, and thus must be careful to not overwrite other fields (which can lead to livelocks). Updating only the fields declared by the programmer in code is a simple, declarative mental model and avoids more complicated logic patterns (e.g. supplying a mutation function).
In addition to the SDK’s client abstraction, the developer also has access to the underlying Kubernetes client, giving them flexibility to perform arbitrary operations.
func (r *reconciler) createConfigMapState() *state {
return &state{
Name: "create-configmap-state",
Condition: achillesAPI.Condition{
Type: CreateConfigMapStateType,
Message: "ConfigMap created",
},
Transition: r.createCMStateFunc,
}
}
func (r *reconciler) createCMStateFunc(
ctx context.Context,
res *v1alpha1.TestCR,
out *fsmtypes.OutputSet,
) (*state, fsmtypes.Result) {
configMap := &corev1.ConfigMap{
ObjectMeta: metav1.ObjectMeta{
Name: res.GetName(),
Namespace: res.GetNamespace(),
},
Data: map[string]string{
"region": res.Spec.Region,
"cloud": ,
},
}
// Resources added to the output set are created and/or updated by the sdk after the state transition function ends
out.Apply(configMap)
// update existing Pod’s restart policy
pod := &corev1.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "existing-pod",
Namespace: “default”,
},
Spec: corev1.PodSpec{
RestartPolicy: corev1.RestartPolicyAlways,
},
}
// applies the update immediately rather than at end of state
if err := r.Client.Apply(ctx, pod); err != nil {
return nil, fsmtypes.ErrorResult(fmt.Errorf("creating namespace: %w", err))
}
return r.nextState(), fsmtypes.DoneResult()
}Result Types
Each transition function must return a Result struct indicating whether the state completed successfully and whether to proceed to the next state or retry the FSM. The SDK supports the following types:
DoneResult(): the state transition finished without any errors. If this result type is returned the SDK will transition to the next state if provided.ErrorResult(err error): the state transition failed with the supplied error (which is also logged). The SDK terminates the FSM and requeues (i.e. re-actuates), subject to exponential backoff.RequeueResult(msg string, requeueAfter time.Duration): the state transition terminates the FSM and requeues after the supplied duration (no exponential backoff). The supplied message is logged at the debug level. This result is used in cases of expected delay, e.g. waiting for a cloud vendor to provision a resource.DoneAndRequeueResult(msg string, requeueAfter time.Duration): this state behaves similarly to the RequeueResult state with the only difference being that the status condition associated with the current state is marked as successful.
Status Conditions
Status conditions are an inconsistent convention in the Kubernetes ecosystem (See this blog post for context)^(.) The SDK takes an opinionated stance by using status conditions to report reconciliation progress, state by state. Furthermore, the SDK supplies a special, top-level status condition of type Ready indicating whether the resource is ready overall. Its value is the conjunction of all other status conditions. Let’s look at an example:
conditions:
- lastTransitionTime: '2024-10-19T00:43:05Z'
message: All conditions successful.
observedGeneration: 14
reason: ConditionsSuccessful
status: 'True'
type: Ready
- lastTransitionTime: '2024-10-21T22:51:30Z'
message: Namespace ensured.
observedGeneration: 14
status: 'True'
type: StateA
- lastTransitionTime: '2024-10-21T23:05:32Z'
message: ConfigMap ensured.
observedGeneration: 14
status: 'True'
type: StateBThese status conditions report that the object succeeded in reconciliation, with details around the particular implementing states (StateA and StateB).
These status conditions are intended to be consumed by both human operators (seeking to understand the state of the system) and programs (that programmatically leverage the CR).
Suspension
Operators can pause reconciliation on Achilles SDK objects by adding the key value pair infrared.reddit.com/suspend: true to the object’s metadata.labels. This is useful in any scenario where reconciliation should be paused (e.g. debugging, manual experimentation, etc.).
Reconciliation is resumed by removing that label.
Metrics
The Achilles SDK instruments a useful set of metrics. See this doc for details.
Debug Logging
The SDK will emit a debug log for each state an object transitions through. This is useful for observing and debugging the reconciliation logic. For example:
my-custom-resource internal/reconciler.go:223 entering state {"request": "/foo-bar", "state": "created"}
my-custom-resource internal/reconciler.go:223 entering state {"request": "/foo-bar", "state": "state 1"}
my-custom-resource internal/reconciler.go:223 entering state {"request": "/foo-bar", "state": "state 2"}
my-custom-resource internal/reconciler.go:223 entering state {"request": "/foo-bar", "state": "state 3"}Finalizers
The SDK also supports managing Kubernetes finalizers on the reconciled object to implement deletion logic that must be executed before the object is deleted. Deletion logic is modeled as a separate FSM. The programmer provides a finalizerState to the reconciler builder, which causes the SDK to add a finalizer to the object upon creation. Once the object is deleted, the SDK skips the regular FSM and instead calls the finalizer FSM. The finalizer is only removed from the object once the finalizer FSM reaches a successful terminal state (DoneResult()).
func SetupController(
log *zap.SugaredLogger,
mgr ctrl.Manager,
rl workqueue.RateLimiter,
c *io.ClientApplicator,
metrics *metrics.Metrics,
) error {
r := &reconciler{
log: log,
c: c,
scheme: mgr.GetScheme(),
}
builder := fsm.NewBuilder(
&v1alpha1.TestCR{},
r.createConfigMapState(),
mgr.GetScheme(),
).
// WithFinalizerState adds deletion business logic.
WithFinalizerState(r.finalizerState()).
// WithMaxConcurrentReconciles tunes the concurrency of the reconciler.
WithMaxConcurrentReconciles(5).
// Manages declares the types of child resources this reconciler manages.
Manages(
corev1.SchemeGroupVersion.WithKind("ConfigMap"),
)
return builder.Build()(mgr, log, rl, metrics)
}
func (r *reconciler) finalizerState() *state {
return &state{
Name: "finalizer-state",
Condition: achapi.Condition{
Type: FinalizerStateConditionType,
Message: "Deleting resources",
},
Transition: r.finalizer,
}
}
func (r *reconciler) finalizer(
ctx context.Context,
_ *v1alpha1.TestCR,
_ *fsmtypes.OutputSet,
) (*state, fsmtypes.Result) {
// implement finalizer logic here
return r.deleteChildrenForegroundState(), fsmtypes.DoneResult()
}Case Study: Managing Kubernetes Clusters
The Compute Infrastructure team has been using the SDK in production for a year now. Our most critical use case is managing our fleet of Kubernetes clusters. Our legacy manual process for creating new opinionated clusters takes about 30 active engineering hours to complete. Our Achilles SDK based automated approach takes 5 active minutes (consisting of two PRs) and 20 passive minutes for the cluster to be completely provisioned, including not only the backing hardware and Kubernetes control plane, but over two dozen cluster add-ons (e.g. Cluster Autoscaler and Prometheus). Our cluster automation currently manages around 35 clusters.
The business logic for managing a Reddit-shaped Kubernetes cluster is quite complex:
FSM for orchestrating Reddit-shaped Kubernetes clusters
The SDK helps us manage this complexity, both from a software engineering and operational perspective. We are able to reason with confidence about the behavior of the system and extend and refactor the code safely.
The self-healing, continuously reconciling nature of Kubernetes controllers ensures that these managed clusters are always configured according to their intent. This solves a long standing problem with our legacy clusters, where state drift and uncodified manual configuration resulted in “haunted” infrastructure that engineers could not reason about with confidence, thus making operations like upgrades extremely risky. State drift is eliminated by control processes.
We define a Reddit-shaped Kubernetes cluster the following API:
apiVersion: cluster.infrared.reddit.com/v1alpha1
kind: RedditCluster
metadata:
name: prod-serving
spec:
cluster: # control plane properties
managed:
controlPlaneNodes: 3
kubernetesVersion: 1.29.6
networking:
podSubnet: ${CIDR}
serviceSubnet: ${CIDR}
provider: # cloud provider properties
aws:
asgMachineProfiles:
- id: standard-asg
ref:
name: standard-asg
controlPlaneInstanceType: m6i.8xlarge
envRef: ${ENV_REF} # integration with network environment
labels:
phase: prod
role: serving
orchKubeAPIServerAddr: ${API_SERVER}
vault: # integration with Hashicorp Vault
addr: ${ADDR}This simple API abstracts over the underlying complexity of the Kubernetes control plane, networking environment, and hardware configuration with only a few API toggles. This allows our infrastructure engineers to easily manage our cluster fleet and enforces standardization.
This has been a massive jump forward for the Compute team’s ability to support Reddit engineering at scale. It gives us the flexibility to architect our Kubernetes clusters with more intention around isolation of workloads and constraining the blast radius of cluster failures.
Conclusion
The introduction of the Achilles SDK has been successful internally at Reddit, though adoption and long-term feature completeness of the SDK is still nascent. We hope you find value in this library and welcome all feedback and contributions.
19:40 UTC
How Reddit Keeps Developer Tools Updated Across Thousands of Workstations
Written by Matthew Warren, Jason Phung and Nick Fohs
Why it matters: We write a lot of software at Reddit. In addition to our work on Reddit itself, we also write internal developer tooling to enhance our software development process. But with thousands of workstations, keeping these tools up to date used to be a manual and time-consuming effort. By treating our employee computers as a deployment platform, we’ve streamlined software deployment for consistency and reproducibility.
Who we are: Corporate Technology, or “CorpTech,” is Reddit’s IT department. Our mission is to Ship cool shit, build things people love, and empower Reddit to do its best work. Within CorpTech, the Endpoint Engineering team manages the computers, devices, systems, and tools our employees use to fulfill that mission every day.
The problem: Previously, engineers followed setup guides to install and configure tools on their Macs. Updates? Those were up to each person. The result? Outdated versions, wasted time, and increased support demands. This was unnecessary toil.
Our approach: We manage our workstations like a deployment platform. This means defining and publishing a structured, automated process for software deployment that’s consistent and transparent to developers. It aligns with how we think about systems, allowing teams outside of CorpTech to reason about – and even extend – our deployment processes.
How it works:
- AutoPkg automation: AutoPkg is an automation tool that detects, downloads, and prepares software updates based on “recipes” we define. Each recipe contains specific steps, like finding the latest release or creating a macOS installer, tailored to the needs of a given tool. We write custom recipes to prepare each of our tools.
- Simple guidelines: We keep things simple by publishing all our tools on our internal GitHub Enterprise server. Our single requirement is that software must be attached to a GitHub Release. This keeps things familiar to our developer teams, and reduces confusion about how or where to store assets. We like to say “if you can tag it in a Release, we can get it on our workstations.”
- CI/CD integration: Our CI/CD pipeline runs these recipes daily in isolated macOS VMs, automatically pulling new releases and distributing updates to workstations. Additionally, builds can be triggered ad-hoc whenever an internal repository is tagged with a new release. This keeps deployments reproducible and allows us to test each update before rollout.
- Version-controlled and accessible: All AutoPkg recipes and CI configurations are stored in a central Git repository open to the entire company. This transparency not only promotes collaboration but also enables any team to add or modify recipes through pull requests, making software deployment a shared responsibility.
Why it works: Within an hour of a release, our developers have the latest software installed and ready to go – without any manual effort. It’s fast, consistent, and lets developers focus on what matters.
Unexpected benefit: With our documented process and auditable pull request system, developers can now manage their own dependencies. Recently, one developer wrote an AutoPkg recipe for a new tool, which Endpoint Engineering quickly reviewed and approved – no extra meetings needed.
The bottom line: Managing our endpoints as a cohesive platform allows Reddit’s internal tooling to stay current, efficient, and hands-off. With AutoPkg, our engineers can focus on building Reddit, while CorpTech keeps the tools running smoothly.
22:38 UTC
Unbossed, But Not Undone
Written by Anthony Sandoval, u/DaveCashewsBand
It’s not a career ladder, it’s a climbing wall. Sometimes you’re moving up, other times across, and every once in a while, you just need to find a ledge.
Roughly a year ago, I was set to present my talk, Accountability Engineering, at SREcon. I only attend every 2-3 years, as my technical curiosities are quickly satisfied and long-lasting. I usually seek out as many socio-technical talks as I can. Last year, I was excited for Charity Majors to present The Engineer/Manager Pendulum Goes Mainstream – a reflection on her 2017 blog post and current perspectives on the same topic.
I was 7 years into my own management journey, leading Reddit’s SRE team, and by now very familiar with the original writing. But reading and rereading it (more than once) had never rocked my commitment to the management career ladder, track, or however I once thought of it. Sitting a few rows from the podium, in a room full of engineers, her talk introduced a fresh vantage point. It hit me differently than I expected.
You cannot just be an engineering line manager forever.
In 2017, the year the post was published, I had only been managing people for a few months. The points in the presentation were honest and relatable, but I was excited in my new role and quickly filed the concepts away into the deep crevices of my brain and bookmarks folders.
At the start of 2021 I joined Reddit, in 2.5 years, I had scaled our SRE department to 34 people, I had 2 managers reporting to me and I was exactly where I’d aspired to be 7 years earlier. I couldn’t have been prouder of what we'd accomplished! In two days, I was even about to present for the first time at SREcon. But, first, I’d watch others present.
Now, back to Majors’s talk. The first 17 minutes of the presentation encouraged the audience to take a break from management and refocus on technical skills – and how a healthy engineering culture needs to support these transitions. She also outlined a half dozen or so traps that managers could fall into. And by then, October 2023, they’d almost all happened to me.
Gobsmacked.
(To be fair, in many forums Majors also strongly encourages engineers who want to, to try the management track.)
I was staring at a slide that informed me I’d come to a fork in the road. But, was it my fork in the road? Did I want to be a director, or VP? And, if so, was direct ascent up the management ladder the best way to get there?
Still, I was reluctant to consider a move. Why?
- I wasn’t burnt out or unhappy as a manager
- I’d been a manager longer than I’d been an engineer – was it even possible that I could become an engineer again?
A seed had been planted. I began to develop a small, but growing concern that I had too few job options. The words on the slide emblazoned in my mind read in bold: “You cannot just be an engineering line manager forever.”
I wasn’t unhappy as a manager
Sure, there were times it was frustrating. But, I love the job.
For months after the conference, the sentiment Majors described seemed to be moving through and extending beyond the tech industry. I found articles focused on middle management burnout. Much of it, I believe stemming from research published by Gartner and Gallup. It was clickbait-y.
But, then in April, read David Brooks' piece in the NYTimes, In Praise of Middle Managers. In the first paragraph, he calls middle managers the “unsung heroes of our age” and quickly establishes that he’s writing about “ethical leadership” (not just management). I saw myself in it. However the undertone was that it was “uncelebrated work, day after day.” It didn’t feel great to read, even if it was “praising” my profession–and incongruent with my own experiences.
Reddit managers are some of the best people I’ve ever worked with. They care about their reports, their quality of life, and the ways they contribute to this amazing product powering the world's online communities.
In line with Brooks’ points about managers, for me too, the most satisfying part of my career has been coaching, mentoring, and investing my time into the teams I’ve worked with. If I stepped away from my role as manager, I could continue to create opportunities to mentor, but it would become an implicit rather than explicit responsibility. And my people management skills were what I believed created the most value for Reddit.
The very same day Brooks’ article was published, The Boston Consulting Group (BCG) released an episode of their “Imagine This…” podcast titled, The End of Middle Management (for Real). The head of BCG’s Behavioral Science Lab, Julia Dhar and her cohosts–one of which is an AI agent GENE–discuss the evolution of the knowledge workforce and the place of middle managers in it. Please, don’t get me started on AI disrupting our careers. I’ve lost track of which industrial or technological revolution we’re currently in, but I acknowledge its power. I know the supervisory role of a manager has changed dramatically with the prominence of remote work – and I’m sure management isn’t out of the reach of AI’s impact.
In contradiction with Brooks’, the hosts asked the forward-looking question: Do companies need managers for employees to feel valued and to grow?
Whoa. I’m an open minded person, so I listened. Unexpectedly, the conversation aligned on Majors’ points. The topic unexpectedly pivoted and challenged the notion that the prescribed “climbing of a ladder” was the most efficient path for growth.
I’d been a manager longer than I’d been an engineer
I have never had a clear, direct career trajectory in my life. For as long as I could remember, I’d been doing exactly what Majors encouraged and what Dhar refers to as a “honeycomb career” (ironic, because Majors founded honeycomb.io).
The road that led me to engineering management was paved by equal parts technical and non-technical experiences. As a new manager, I felt initially that I had some advantages over (many but not all of) my peers who came from strictly engineering backgrounds. But, with the passage of time I’ve observed those well suited for the role–now with years of managerial experience–could develop a both technical and organizational strategy.
I want to grow and extend my career at Reddit.
I worked with my manager, the VP of Infrastructure to evaluate my strengths, and identify opportunities for development. In addition to my people skills, I’m intensely detail oriented, a strong communicator, organized, and a technical generalist. Combined that with an accumulated depth of Reddit specific knowledge and that combination lends itself well to a number of different roles.
Thankfully, Reddit has a great engineering culture. When it’s appropriate, a swing on the pendulum is supported by the company. My career moves would never have been possible if our leadership wasn’t investing in career growth and internal mobility. In fact, at Reddit, every employee receives the “Mobility Monthly” newsletter which lists open positions and spotlights a Snoo (employee) who recently moved into a new position.
Unbossing Yourself
That same month, after my transfer, I stumbled across the term “Unbossing” in Rachel Feintzeig’s piece Will ‘Unbossing’ Yourself Kill Your Career? in The Wall Street Journal. (Google the term, it’s kind of trendy these days.)
Spoiler: It won’t kill your career.
Companies need managers, and I’d love to be one again someday.
I don’t disagree that people management roles can weigh heavily. If you care about the people you manage, detaching yourself from the emotion and stress that comes with the responsibilities requires intention and discipline. And, I don’t believe the organizational evolution of the post-Covid remote workplace is finished, I expect the role of people manager still needs to evolve and adapt to it.
But, the media focus on the negative sentiment of managers is unfair, and the simple narrative of listing the hardships of the career rings hollow. Placing an emphasis on the switching roles for the purposes of development–for both the individual and the organization–are much more compelling. Dhar describes “reshuffling” as a way of reinfusing the organization with people capable of promoting productivity.
It turns out it’s a great deal of fun, too!
I’d never seen a manager spotlighted in our Mobility Monthly, but I’m amongst more than a handful. In July, I transferred to the Tech Program Management Office (PMO) team at Reddit. I’m extremely happy in my new role as a Senior Technical Program Manager (TPM) and I’ve found the new cross-functional domains and inter-disciplinary areas of the business to be both exciting and challenging. I’m eager to make my mark–and expect I’ll have more than a handful of fun TPM adventures to write about on this blog next year.
00:04 UTC
We brought a group of women engineers from Reddit to Grace Hopper. Here’s how it went…
Written by Briana Nations, Nandika Donthi, and Aarin Martinez (leaders of WomEng @ Reddit)
Pictured: Aarin (on the left) and Bri (in the middle) and Nandika (on the right)
This year, Reddit sent a group of 15 amazing women engineers to the 2024 Grace Hopper Celebration in Philadelphia!
These women engineers varied in level, fields, orgs, and backgrounds all united by their participation in Reddit’s Women in Engineering (WomEng) ERG and interest in the conference. For some engineers, this was a long anticipated reunion with the celebration in a post-pandemic setting. Other engineers were checking off a bucket list conference. And some engineers were honestly just happy to be there with their peers.
Although 15 members seems like a small group, in a totally remote company, a gathering of 15 women engineers felt like a rare occasion. You could only imagine the shock factor of the world’s largest IRL gathering of women and non-binary technologists.
Speakers
Right off the bat, the conference kicked off with a powerful opening ceremony featuring an AMA from America Ferrara (from Barbie). Her message about how “staying in the room even when it's uncomfortable is the only way you make change” was enough to inspire even the most cynical of attendees to lean into what the conference was really about: empowerment.
The following day, our members divided into smaller groups to participate in talks on a range of themes: Emotional Intelligence in the Workplace, Designing Human-Centered Tech Policy, Climbing the Career Ladder, etc. Although there were technical insights gained from these discussions, the most valuable takeaway was that nearly every participant left each session having formed a new connection. Many of these connections were also invited to our happy hour networking event that we hosted Wednesday night!
Networking Event
Putting up decorations at the networking event
Going into the conference, we wanted to create an opportunity for our women engineers to connect with other engineers who were attending the conference in a more casual setting. We planned a networking event at a local Philly brewery and hosted over 80 GHC attendees for a fun night of sharing what we do over snacks and drinks! We got to meet folks from diverse backgrounds, each pursuing their own unique career paths from various corners of the globe. It was incredibly inspiring to be surrounded by such driven and open-minded engineers. We each left the event with energized spirits and 10+ new LinkedIn connections.
BrainDates
One unexpected highlight at the conference (that none of us leads had seen before) was the opportunity to go on 'BrainDates’. Through the official GHC app, attendees could join or initiate in-person discussions with 2 to 10 other participants on a chosen topic. The most impactful BrainDate us leads attended was on a topic we proposed: how to bring value in the ERG space (shocker). By chance, a CTO from another company joined our talk and bestowed her valuable insights on women in engineering upon us, drawing from her past experience in creating impactful programs at her previous organization. While we obviously spent some time forcing her into an impromptu AMA on being a girl boss, she also taught us that you don’t always have to bring people away from their work to bring meaning to our ERG. Women engineers want to talk about their work and often don’t feel like people care to listen or that their work isn’t worth talking about. We have the power to change that both in our orgs and company wide.
Main Takeaways
Our Reddit WomEng conference group on the last night of GHC
Throughout the entirety of the conference we heard so many different perspectives both internally and externally about what being a woman in technology meant to them. Many only had good things to say about the field and were trying to give back and uplift other women in the field. Many had harder times believing that diversity and inclusion were truly a priority in hiring processes. And some were trying to do what they could to fill the gaps wherever they saw them. All of these points of views were valid and the reason conferences like these are so important. Regardless of whether you are motivated or jaded, when you bring women together there is a collective understanding and empowerment that is so vital. When women come together, we hear each other, get stuff done, and make change happen. We ultimately left the conference inspired to create more upskilling/speaking opportunities for our current women engineers and to also hold our own leaders accountable to practice the inclusive values they preach. We also maybe know a little more about GraphQL, cybersecurity, and K-pop?
All in all, to the readers who were maybe hoping for a “hotter take” on the conference: sorry (not sorry) to disappoint, though we admit the title is a little clickbaity. To the readers who need to hear it: you being the only ___ in the room matters. We know that it can feel like everyone is eager to de-prioritize or even invalidate DEI initiatives, especially given the way the industry has hit some downturns recently. We strongly believe though, that in these times when there are less sponsors and less flashy swag, it is essential to remind each other why diversity, equity, and inclusion are an integral part of a successful and fair workforce. It’s time to start “BrainDating” each other more often and not wait around for a yearly conference to remind ourselves of the value we bring to the table!
P.S. to all the allies in the chat, we appreciate you for making it this far. We challenge you to ask a woman engineer you may know about their work. You never know what misconception you could be breaking with just 2 minutes of active listening.
22:52 UTC
Spooky Szn at Reddit
Written by Chase Sturgill
u/dodai_taboada’s spooky Reddit logo design
Hey y’all! This week we are taking a break from our typical technical content to showcase some of the fun things that happen behind the scenes at Reddit. One of my favorite things about working at Reddit is that there is no shortage of fun things happening, both for our in-office and virtual Snoos. With Halloween right around the corner, this is no exception! We would like to showcase the amazing decorations our teams have put up in our offices around the world. Be sure to let us know in the comments which office you think has the best decorations!
While some may argue this is too early for Halloween decorations, I challenge that it isn’t early enough. And long may they reign until “All I For Christmas is You” tops the charts once again.
Decorations are only a small part of the fun though! On October 31st, all of our offices will be hosting pizza parties, with the pizza of choice being chosen via polling of local Snoos. Our NYC Snoos couldn’t make up their minds, so both Emmy Squared and John’s of Bleecker will be served. Really wishing I was based in NYC right about now 🙂
And of course it wouldn’t be a Halloween celebration without a costume contest! Every Snoo is encouraged to submit pictures of their Halloween costumes and company-wide awards will be given for the following categories:
- Most Reddit-y
- Best Team Costume
- Best in Show (for the pets)
- Spookiest
- ThereWasAnAttempt (a costume that shows you tried…)
Additionally, every office will crown their own office winner for the Snoo who wears the best costume to the office on Halloween.
It’s an exciting time for our Snoos and we hope you’ve enjoyed this glimpse into some of the fun things that happen within Reddit - wishing you all a happy spooky szn!
20:36 UTC
Title: Snoosweek Recap (Reddit’s Internal Hack-a-thon)
Written by Mackenzie Greene
Hey friends - We’ve just wrapped up another exciting Snoosweek here at Reddit this past August! For those who have been following r/RedditEng for a bit (past Snoosweek blog post), you know it’s a special time. But if you’re new to the concept, you’re probably wondering, “What is Snoosweek?” Well, let us take you behind the scenes of this unique event where we break from our everyday routines to work on something different from usual.
What is a Snoosweek (and why it’s special)
Snoosweek is Reddit’s internal hackathon week where employees are encouraged to step away from their day to day and pursue any project that sparks their interest. It’s a dedicated time for creativity, innovation, and collaboration. We have 2 weeks dedicated to Snoosweek each year - one in Q1 and one in Q3.
Whether it’s addressing long standing technical challenges, building dream features, or brainstorming future Reddit, Snoosweek empowers employees to explore their boldest ideas. By fostering team collaboration, it opens up new avenues for problem solving and provides fresh perspectives on both internal processes and user facing features. Some of these ideas even make it into a product roadmap! Snoosweek is both fun and impactful.
There are Demos!
At the end of Snoosweek, we host a Demo Day, where teams have the opportunity to present their projects in a quick 60-second demo video. This showcase, hosted by our Chief Technology Officer (CTO) Chris Slowe and Chief Product Officer (CPO) Pali Bhat, allows our leaders and the broader company to see the creative solutions developed during the week, It’s a chance for teams to share their achievements and for everyone to witness the potential impact these projects could have on Reddit.
These are the stats from the most recentt Snoosweek demos!
There are Awards!
Following Demo Day, a hand selected group of judges evaluates the demos and selects winners for six distinct awards. The awards and this year's winners are listed below.
This year, we introduced a new award - the A11Y Ally to recognize and celebrate projects that enhance accessibility on Reddit, making the platform more inclusive and user-friendly for everyone. This award encourages innovative solutions that improve the Reddit experience for users of all abilities, helping to foster a truly inclusive community for all.
And there’s Swag!
Each Snoosweek, we host a design contest where one employee’s artwork is selected to feature on the official T-shirt, which is then given to all participants as a memorable keepsake of the week.
This is the design that won, created by Dylan Glenn.
Thanks!
Snoosweek has become one of our most beloved traditions and a cornerstone of our company culture. Beyond the tangible benefits we've highlighted, it’s an incredible opportunity for our Snoos to connect and collaborate with colleagues beyond their usual teams. As Reddit continues to grow, we see Snoosweek evolving and expanding, becoming an even bigger and better part of our company’s traditions. Thank you to the Eng Branding team, the judges, Chris Slowe and Pali Bhat for their Executive support, and all the Snoos that come excited to participate each Snoosweek.
15:02 UTC
Bringing Learning to Rank to Reddit - LTR modeling
Written by Sahand Akbari.
In the previous series of articles in the learning to rank series, we looked at how we set up the training data for the ranking model, how we did feature engineering, and optimized our Solr clusters to efficiently run LTR at scale. In this post we will look at learning to rank ML modeling, specifically how to create an effective objective function.
To recap, imagine we have the following training data for a given query.
| Query | Post ID | Post Title | F1: Terms matching post title | F2: Terms matching posts body text | F3: Votes | Engagement Grade |
|---|---|---|---|---|---|---|
| Cat memes | p1 | Funny cat memes | 2 | 1 | 30 | 0.9 |
| Cat memes | p2 | Cat memes ? | 2 | 2 | 1 | 0.5 |
| Cat memes | p3 | Best wireless headphones | 0 | 0 | 100 | 0 |
For simplicity, imagine our features in our data are defined per each query-post pair and they are:
- F1: Terms in the query matching the post title
- F2: Terms in the query matching the post body
- F3: number of votes for this post
Engagement grade is our label per query-post pair. It represents our estimation of how relevant the post is for the given query. Let’s say it’s a value between 0 and 1 where 1 means the post is highly relevant and 0 means it’s completely irrelevant. Imagine we calculate the engagement grade by looking at the past week's data for posts redditors have interacted with and discarding posts with no user interaction. We also add some irrelevant posts by randomly sampling a post id for a given query (i.e negative sampling). The last row in the table above is a negative sample. Given this data, we define an engagement-based grade as our labels: click through rate (CTR) for each query-post pair defined by ratio of total number of clicks on the post for the given query divided by total number of times redditors viewed that specific query-post pair.
Now that we have our features and labels ready, we can start training the LTR model. The goal of an LTR model is to predict a relevance score for each query-post pair such that more relevant posts are ranked higher than less relevant posts. Since we don’t know the “true relevance” of a post, we approximate the true relevance with our engagement grade.
One approach to predicting a relevance score for each query-post is to train a supervised model which takes as input the features and learns to predict the engagement grade directly. In other words, we train a model so that its predictions are as close as possible to the engagement grade. We’ll look closer at how that can be done. But first, let’s review a few concepts regarding supervised learning. If you already know how supervised learning and gradient descent work, feel free to skip to the next section.
Machine Learning crash course – Supervised Learning and Gradient Descent
Imagine we have d features ordered in a vector (array) x = [x1, x2, …, xd]and a label g(grade).
Also for simplicity imagine that our model is a linear model that takes the input x and predicts y as output:
We want to penalize the model when y is different from g. So we define a Loss function that measures that difference. An example loss function is squared error loss (y-g)^2. The closer y is to g the smaller the loss is.
In training, we don’t have just one sample (x, g) but several thousands (or millions) of samples. Our goal is to change the weights w in a way that makes the loss function over all samples as small as possible.
In the case of our simple problem and loss function we can have a closed-form solution to this optimization problem, however for more complex loss functions and for practical reasons such as training on large amounts of data, there might not be an efficient closed-form solution. As long as the loss function is end-to-end differentiable and has other desired mathematical properties, one general way of solving this optimization problem is using stochastic gradient descent where we make a series of small changes to weights w of the model. These changes are determined by the negative of the gradient of the loss function L. In other words, we take a series of small steps in the direction that minimizes L. This direction is approximated at each step by taking the negative gradient of L with respect to w on a small subset of our dataset.
At the end of training, we have found a w that minimizes our Loss function to an acceptable degree, which means that our predictions y are as close as possible to our labels g as measured by L. If some conditions hold, and we’ve trained a model that has learned true patterns in the data rather than the noise in the data, we'll be able to generalize these predictions. In other words, we’ll be able to predict with reasonable accuracy on unseen data (samples not in our training data).
One thing to remember here is that the choice of weights w or more generally the model architecture (we could have a more complex model with millions or billions of weights) allows us to determine how to get from inputs to the predictions. And the choice of loss function L allows us to determine what (objective) we want to optimize and how we define an accurate prediction with respect to our labels.
Learning to rank loss functions
Now that we got that out of the way, let’s discuss choices of architecture and loss. For simplicity, we assume we have a linear model. A linear model is chosen only for demonstration and we can use any other type of model (in our framework, it can be any end to end differentiable model since we are using stochastic gradient descent as our optimization algorithm).
An example loss function is (y-g)^2. The closer y is to g on average, the smaller the loss is. This is called a pointwise loss function, because it is defined for a single query-document sample.
While these types of loss functions allow our model output to approximate the exact labels values (grades), this is not our primary concern in ranking. Our goal is to predict scores that produce the correct rankings regardless of the exact value of the scores (model predictions). For this reason, learning to rank differs from classification and regression tasks which aim to approximate the label values directly. For the example data above, for the query “cat memes”, the ranking produced by the labels is [p1 - p2 - p3]. An Ideal LTR loss function should penalize the predictions that produce rankings that differ from the ranking above and reward the predictions that result in similar rankings.
Side Note: Usually in Machine learning models, loss functions express the “loss” or “cost” of making predictions, where cost of making the right predictions is zero. So lower values of loss mean better predictions and we aim to minimize the loss.
Pairwise loss functions allow us to express the correctness of the ranking between a pair of documents for a given query by comparing the rankings produced by the model with rankings produced by the labels given a pair of documents. In the data above for example, p1 should be ranked higher than p2 as its engagement grade is higher. If our model prediction is consistent, i.e. the predicted score for p1 is higher than p2, we don’t penalize the model. On the other hand, if p1’s score is higher than p2, the loss function assigns a penalty.
Loss for a given query q is defined as the sum of pairwise losses for all pairs of documents i,j.
1(g_i > g_j) is an indicator function. It evaluates to 1 when g_i > g_j and to 0 otherwise. This means that if the grade of document i is larger than the grade of document j, the contribution of i,j to loss is equal to max(0, 1 - (y_i - y_j)). In other words, if g_i > g_j, loss decreases as (y_i - y_j) increases because our model is ranking document i higher than document j. Loss increases when the model prediction for document j is higher than document i.
One downside of using pairwise loss is the increase in computational complexity relative to pointwise solutions. For each query, we need to calculate the pairwise loss for distinct document pairs. For a query with D corresponding posts, the computation complexity is O(D^2) while for a pointwise solution it is O(D). In practice, we usually choose a predefined number of document pairs rather than calculating the loss for all possible pairs.
In summary, we calculate how much the pairwise difference of our model scores for a pair of documents matches the relative ranking of the documents by labels (which one is better according to our grades). Then we sum the loss for all such pairs to get the loss for the query. The loss of a given dataset of queries can be defined as the aggregation of loss for each queries.
Having defined the loss function L and our model f(x), our optimization algorithm (stochastic gradient descent) finds the optimal weights of the model (w and b) that minimizes the loss for a set of queries and corresponding documents.
In addition to pointwise and pairwise ranking loss functions, there's another category known as listwise. Listwise ranking loss functions assess the entire ranked list, assigning non-zero loss to any permutation that deviates from the ideal order. Loss increases with the degree of divergence.
These functions provide the most accurate formulation of the ranking problem, however, to compute a loss based on order of the ranked list, the list needs to be sorted. Sorting is a non-differentiable and non-convex function. This makes the gradient based optimization methods a non-viable solution. Many studies have sought to create approximate listwise losses by either directly approximating sorting with a differentiable function or by defining an approximate loss that penalizes deviations from the ideal permutation order. The other challenge with listwise approaches is computationally complexity as these approaches need to maintain a model of permutation distribution which is factorial in nature. In practice, there is usually a tradeoff between degree of approximation and computational complexity.
For learning to rank at Reddit Search, we used a weighted pairwise loss called LambdaRank. The shortcoming of the pairwise hinge loss function defined above is that different pairs of documents are treated the same whereas in search ranking we usually care more about higher ranked documents. LambdaRank defines a pairwise weight (i.e. LambdaWeight), dependent on positions of the documents, to assign an importance weight for each comparison. Our pairwise hinge loss with lambda weight becomes:
There are different ways to define the importance of comparisons. We use NDCG lambda weight which calculates a weight proportionate to the degree of change in NDCG after a swap is made in the comparison.
Side Note: We still need to sort the ranking list in order to calculate the LambdaWeight and since sorting is not a differentiable operation, we must calculate the LambdaWeight component without gradients. In tensorflow, we can use tf.stop_gradient to achieve this.
One question that remains: how did we choose f(x)? We opted for a dense neural network (i.e. multi-layer perceptron). Solr supports the Dense Neural network architecture in the Solr LTR plugin and we used tensorflow-ranking for training the ranker and exporting to the Solr LTR format. Practically, this allowed us to use the tensorflow ecosystem for training and experimentation and running LTR at scale within Solr. While gradient boosted trees such as LambdaMart are popular architectures for learning to rank, using end-to-end differentiable neural networks allows us to have a more extensible architecture by enabling only minimal modifications to the optimization algorithm (i.e. stochastic gradient descent) when adding new differentiable components to the model (such as semantic embeddings).
We have our model! So how do we use it?
Imagine the user searches for “dog memes”. We have never seen this query and corresponding documents in our training data. This means that we don’t have any engagement grades. Our model trained by the Pairwise loss, can now predict scores for each query - document pair. Sorting the model scores in a descending order will result in a ranking of documents that will be returned to the user.
| Query | Post ID | Post Title | F1: Terms matching post title | F2: Terms matching posts body | F3: Votes | Engagement Grade | Model Predicted Score |
|---|---|---|---|---|---|---|---|
| dog memes | p1 | Funny dog memes | 2 | 1 | 30 | ? | 10.5 |
| dog memes | p2 | Dog memes | 2 | 2 | 1 | ? | 3.2 |
| dog memes | p3 | Best restaurant in town? | 0 | 0 | 100 | ? | 0.1 |
Conclusion
In this post, we explored how learning-to-rank (LTR) objectives can be used to train a ranking model for search results. We examined various LTR loss functions and discussed how we structure training data to train a ranking model for Reddit Search. A good model produces rankings that put relevant documents at the top. How can we measure if a model is predicting good rankings? We would need to define what “good” means and how to measure better rankings. This is something we aim to discuss in a future blog post. So stay tuned!
17:50 UTC
A Million Connection Problem
Written by Anton Kuklin, edited by René Treffer
Background
Hey folks, Anton from the Transport team here. We, as a team, provide a network platform for Reddit Infrastructure for both North/South and East/West pillars. In addition to that, we are responsible for triaging & participating in sitewide incidents, e.g. increased 5xx on the edge. Quite often it entails identifying a problematic component and paging a corresponding team. Some portion of incidents are related to a “problematic” pod, and usually is identified by validating whether this is the only pod that is erroring and solved by rescheduling it. However, during my oncall shift in the first week of June, the situation changed drastically.
First encounter
In that one week, we received three incidents, related to different services, with a number of slow responding and erroring pods. It became clear that something was wrong on the infra level. None of the standard k8s metrics showed anything suspicious, so we started going down the stack.
As most of our clusters are currently running Calico CNI in a non-ebpf mode, they require kube-proxy, which relies on conntrack. While going through node-level linux metrics, we found that we were starting to have issues on nodes, which were hitting one million conntrack rows. This was certainly unexpected, because our configuration specified max conntrack rows by ~100k * Cores numb. In addition, we saw short timeframes (single digits of seconds), when spikes of ~20k+ new connections appeared on a single node.
At this point, we pondered three questions:
- Why are we hitting a 1M limit? These nodes have 96 cores, which should result in a 9.6M limit; the numbers don’t match.
- How did we manage to get 1M connections? The incidents were related to normal kubernetes worker nodes, so such a number of connections was unreasonable.
- Where are these 20k new connections per second spikes coming from?
As these questions affected multiple teams, a dedicated workgroup was kicked off.
Workgroup
At the very beginning we defined two main goals:
- Short term: fix max conntrack limit. This would prevent recurring incidents and give us time for further investigations.
- Mid term: figure out the cause and fix the large number of connections per node.
The first goal was solved relatively quickly as a conntrack config change was mistakenly added into a base AMI and kube-proxy setting was overwritten as a result. By fixing it, we managed to stop incidents from recurring. However, the result scared us even more: right after the fix, some bad nodes had 1.3M conntrack rows.
After some manual digging into conntrack logs (you can do the same by running conntrack -L on your node) and labeling corresponding IP’s, we managed to identify the client/server pair that contributed the most. It was a graphql service making a ton of connections to one of the core services. And here comes the most interesting part: our standard protocol for internal service communication is gRPC, which is built on top of HTTP/2. As HTTP/2 implies long-lived connections, it establishes connections to all of the target pods and performs client-side load balancing, which we already knew. However, there were a number of compounding factors at the wrong time and place.
At Reddit, we have a few dozen clusters. We still oversee a few gigantic, primary clusters, which are running most of Reddit’s services. We are already proactively working on scaling them horizontally, equally distributing the workload.
These clusters run GQL API services, which are written in Python. Due to the load the API receives, this workload runs on over ~2000 pods. But, due to GIL, we run multiple (35 to be more precise) app processes within one pod. There’s a talk by Ben Kochie and Sotiris Nanopolous at SRECON, which describes how we are managing this: SREcon23 Europe/Middle East/Africa - Monoceros: Faster and Predictable Services through In-pod....The GQL team is in the process of gradually migrating this component from Python to Go, which should significantly decrease the number of pods required to run this workload and the need to have multiple processes per serving container.
Doing some simple math, we calculated that 2,000 GQL pods, running 35 processes each, results in 75,000 gRPC clients. To illustrate how enormous this is, the core service mentioned above, which GQL makes calls to, has ~500 pods. As each gRPC client opens a connection to each of target pods, this will result in 75,000 * 500 = 37.5M connections.
However, this number was not the only issue. We now have everything to explain the spikes. As we are using headless service, when a new pod is getting spawned, it will be discovered after a DNS record gets updated with a new pod IP added to a list of IPs. Our kube-dns cache TTL is set to 10s, and as a result, newly spawned pods targeted by GQL will receive 75K of new connections in a timeframe of 10s.
After some internal discussions, we agreed on the following decision. We needed some temporary approach, which would reduce a number of connections, until the load from GQL Python would be migrated to Go in a matter of months. The problem boils down to a very simple equation: we have N clients and M servers, which results in N*M connections. By putting a proxy in between, we can replace N*M with N*k + M*k, where k is the number of proxy instances. As proxying is cheap, we can assume that k < N/2 and k < M/2, which means N*k + M*k < N*M. We heavily use envoy for ingress purposes and we have already used it for intra-cluster proxy in some special cases. Because of that, we decided to spin up a new envoy deployment for this test, proxy traffic from GQL to that core service using it and see how it would change the situation. And … it reduced the number of opened connections by GQL by more than 10x. That was huge! We didn’t see any negative changes in request latencies. Everything worked seamlessly.
At this point, the question became, how many connections per node are acceptable? We didn’t have a plan to migrate all of the traffic to run via an envoy proxy from GQL servers to targets, so we needed some sort of a line in the sand, some number, where we could say, “okay, this is enough and we can live with this until GQL migration and clusters horizontal scaling are finished”. A conntrack row size is 256 bytes, which you can check by running `cat /proc/slabinfo | grep nf_conntrack`. As our nodes have ~100 MB L3 cache size, which is ~400K conntrack rows, we decided that we normally want 90%+ of nodes in our clusters to fit into this limit, and in case it goes lower than 85%, we would migrate more target services to envoy proxy or re-evaluate our approach
After the work group successfully achieved its result, we in the transport team realized that what we actually could and should improve is our L3/4 network transparency. We should be able to identify workloads much quicker and outside of L7 data that we collect via our network libraries used by applied engineers in their service. Ergo, a “network transparency” project was born, which I will share more about in a separate post or talk. Stay tuned.
19:34 UTC
Snappy, Not Crappy: An Android Health & Performance Journey
Written by Lauren Darcey, Rob WcWhinnie, Catherine Chi, Drew Heavner, Eric Kuck
How It Started
Let’s rewind the clock a few years to late 2021. The pandemic is in full swing and Adele has staged a comeback. Bitcoin is at an all-time high, Facebook has an outage and rebrands itself as Meta, William Shatner gets launched into space, and Britney is finally free. Everyone’s watching Squid Game and their debt-ridden contestants are playing games and fighting for their lives.
Meanwhile, the Reddit Android app is supporting communities talking and shitposting about all these very important topics while struggle-bugging along with major [tech] debt and growing pains of its own. We’ve also grown fast as a company and have more mobile engineers than ever, but things aren’t speeding up. They’re slowing down instead.
Back then, the Android app wasn’t winning any stability or speed contests, with a crash-free rate in the 98% range (7D) and startup times over 12 seconds at p90. Yeah, I said 12 seconds. Those are near-lethal stats for an app that supports millions of users every day. Redditors were impatiently waiting for feeds to load, scrolling was a janky mess, the app did not have a coherent architecture anymore and had grown quickly into a vast, highly coupled monolith. Feature velocity slowed, even small changes became difficult, and in many critical cases there was no observability in place to even know something was wrong. Incidents took forever to resolve, in part, because making fixes took a long time to develop, test, deploy. Adding tests just slowed things down even more without much obvious upside, because writing tests on poorly written code invites more pain.
These were dark times, friends, but amidst the disruptions of near-weekly “Reddit is down” moments, a spark of determination ignited in teams across Reddit to make the mobile app experiences suck less. Like a lot less. Reddit might have been almost as old as dial-up days, but there was no excuse for it still feeling like that in-app in the 2020s.
App stability and performance are not nice-to-haves, they’re make-or-break factors for apps and their users. Slow load times lead to app abandonment and retention problems. Frequent crashes, app not responding events (ANRs), and memory leaks lead to frustrated users uninstalling and leaving rage-filled negative reviews. On the engineering team, we read lots of them and we understood that pain deeply. Many of us joined Reddit to help make it a better product. And so began a series of multi-org stability and performance improvement projects that have continued for years, with folks across a variety of platform and feature teams working together to make the app more stable, reliable, and performant.
This blog post is about that journey. Hopefully this can help other mobile app teams out there make changes to address legacy performance debt in a more rational and sustainable way.
Snappy, Not Crappy
You might be asking, “Why all the fuss? Can’t we just keep adding new features?” We tried that for years, and it showed. Our app grew into a massive, complex monolith with little cleanup or refactoring. Features were tightly coupled and CI times ballooned to hours. Both our ability to innovate and our app performance suffered. Metrics like crash rates, ANRs, memory leaks, startup time, and app size all indicated we had significant work to do. We faced challenges in prioritization, but eventually we developed effective operational metrics to address issues, eliminate debt, and establish a sustainable approach to app health and performance.
The approach we took, broadly, entailed:
- Take stock of Android stability and performance and make lots of horrified noises.
- Bikeshed on measurement methods, set unrealistic goals, and fail to hit them a few times.
- Shift focus on outcomes and burndown tons of stability issues, performance bottlenecks, and legacy tech debt.
- Break up the app monolith and adopt a modern, performant tech stack for further gains.
- Improve observability and regression prevention mechanisms to safeguard improvements long term. Take on new metrics, repeat.
- Refactor critical app experiences to these modern, performant patterns and instrument them with metrics and better observability.
- Take app performance to screen level and hunt for screen-specific improvement opportunities.
- Improve optimization with R8 full mode, upgrade Jetpack Compose, and introduce Baseline Profiles for more performance wins.
- Start celebrating removing legacy tech and code as much as adding new code to the app.
We set some north star goals that felt very far out-of-reach and got down to business.
From Bikeshedding on Metrics to Focusing On Burning Down Obvious Debt
Well, we tried to get down to business but there was one more challenge before we could really start. Big performance initiatives always want big promises up-front on return on investment, and you’re making such promises while staring at a big ball of mud that is fragile with changes prone to negative user impact if not done with great care.
When facing a mountain of technical debt and traditional project goals, it’s tempting to set ambitious goals without a clear path to achieve them. This approach can, however, demoralize engineers who, despite making great progress, may feel like they’re always falling short. Estimating how much debt can be cleared is challenging, especially within poorly maintained and highly coupled code.
“Measurement is ripe with anti-patterns. The ways you can mess up measurement are truly innumerable” - Will Larson, The Engineering Executive's Primer
We initially set broad and aggressive goals and encountered pretty much every one of the metrics and measurement pitfalls described by Will Larson in "The Engineering Executive's Primer." Eventually, we built enough trust with our stakeholders to move faster with looser goals and shifted focus to making consistent, incremental, measurable improvements, emphasizing solving specific problems over precise performance metrics goals upfront and instead delivered consistent outcomes after calling those shots. This change greatly improved team morale and allowed us to address debt more effectively, especially since we were often making deep changes capable of undermining metrics themselves.
Everyone wants to build fancy metrics frameworks but we decided to keep it simple as long as we could. We took aim at simple metrics we could all agree on as both important and bad enough to act on. We called these proxy metrics for bigger and broader performance concerns:
- Crashlytics crash-free rate (7D) became our top-level stability and “up-time” equivalent metric for mobile.
- When the crash-free rate was too abstract to underscore user pain associated with crashing, we would invert the number and talk about our crashing user rates instead. A 99% starts to sound great, but 1% crashing user rate still sounds terrible and worth acting on. This worked better when talking priorities with teams and product folks.
- Cold start time became our primary top-level performance metric.
- App size and modularization progress became how we measured feature coupling.
These metrics allowed us to prioritize effectively for a very long time. You also might wonder why stability matters here in a blog post primarily about performance. Stability turns out to be pretty crucial in a performance-focused discussion because you need reliable functionality to trust performance improvements. A fast feature that fails isn’t a real improvement. Core functionality must be stable before performance gains can be effectively realized and appreciated by users.
Staying with straightforward metrics to quickly address user pain allowed us to get to work fixing known problems without getting bogged down in complex measurement systems. These metrics were cheap, easy, and available, reducing the risk of measurement errors. Using standard industry metrics also facilitated benchmarking against peers and sharing insights. We deferred creating a perfect metrics framework for a while (still a work in progress) until we had a clearer path toward our goals and needed more detailed measurements. Instead, we focused on getting down to business and fixing the very real issues we saw in plain sight.
In Terms of Banana Scale, Our App Size & Codebase Complexity Was Un-a-peeling
Over the years, the Reddit app had grown due to the continuous feature development, especially in key spaces, without corresponding efforts around feature removals or optimization. App size is important on its own, but it’s also a handy proxy for assessing an app’s feature scope and complexity. Our overall app size blew past our peers’ sizes as our app monolith grew in scope in complexity under-the-hood.
Figure 1: The Reddit Android App Size: Up, Up and Away!
App size was especially critical for the Android client, given our focus on emerging markets where data constraints and slower network speeds can significantly impact user acquisition and retention. Drawing from industry insights, such as Google’s recommendations on reducing APK size to enhance install conversion rates, we recognized the need to address our app’s size was important, but our features were so tightly coupled we were constrained on how to reduce app size until we modularized and decoupled features enough to isolate them from one another.
We prioritized making it as easy to remove features as to add them and explored capabilities like conditional delivery. Worst case? By modularizing by feature with sample apps, we were ensuring that features operated more independently and ownership (or lack of it) was obvious. This way, if worse came to worse, we could take the modernized features to a new app target and declare bankruptcy on the legacy app. Luckily, we made a ton of progress on modularization quickly, those investments began to pay off and we did not have to continue in that direction.
As of last week, our app nudged to under 50Mb for the first time in three years and app size and complexity continue to improve with further code reuse and cleanups. We are working to explore more robust conditional delivery opportunities to deliver the right features to our users. We are also less tolerant of poorly owned code living rent-free in the app just in case we might need it again someday.
How we achieved a healthier app size:
- We audited app assets and features for anything that could be removed: experiments, sunsetted features, assets and resources
- We optimized our assets and resources for Android, where there were opportunities like webp. Google Play was handy for highlighting some of the lowest hanging fruit
- We experimented with dynamic features and conditional delivery, shaving about a third of our app install size
- We leveraged R8 full mode for improved minification
- We worked with teams to have more experiment cleanup and legacy code sunset plans budgeted into projects
- We made app size more visible in our discussions and introduced observability and CI checks to catch any accidental app size bloat at the time of merge and deploy
Finally, we leaned in to celebrating performance and especially removing features and unnecessary code as much as adding it, in fun ways like slack channels.
Cold Start Improvements Have More Chill All The Time
When we measured our app startup time to feed interactions (a core journey we care about) and it came in at that astronomical 12.3s @ p90, we didn’t really need to debate that this was a problem that needed our immediate attention. One of the first cross-platform tiger teams we set up focused on burning down app startup debt. It made sense to start here because when you think about it, app startup impacts everything: every time a developer starts the app or a tester runs a test, they pay the app startup tax. By starting with app start, we could positively impact all teams, all features, all users, and improve their execution speeds.
How we burned more than 8 seconds off app start to feed experience:
- We audited app startup from start to finish and classified tasks as essential, deferrable or removable
- We curated essential startup tasks and their ordering, scrutinizing them for optimization opportunities
- We optimized feed content we would load and how much was optimal via experimentation
- We optimized each essential task with more modern patterns and worked to reduce or remove legacy tech (e.g. old work manager solutions, Rx initialization, etc.)
- We optimized our GraphQL calls and payloads as well as the amount of networking we were doing
- We deferred work and lazy loaded what we could, moving those tasks closer to the experiences requiring them
- We stopped pre-warming non-essential features in early startup
- We cleaned up old experiments and their startup tasks, reducing the problem space significantly
- We curated essential startup tasks and their ordering, scrutinizing them for optimization opportunities
- We modularized startup and put code ownership around it for better visibility into new work being introduced to startup
- We introduced regression prevention mechanisms as CI checks, experiment checks and app observability in maintain our gains long term
- We built an advisory group with benchmarking expertise and better tooling, aided in root causing regressions, and provided teams with better patterns less likely to introduce app-wide regressions
These days our app start time is a little over 3 seconds p90 worldwide and has been stable and slowly decreasing as we make more improvements to startup and optimize our GQL endpoints. Despite having added lots of exciting new features over the years, we have maintained and even improved on our initial work. Android and iOS are in close parity on higher end hardware, while Android continues to support a long tail of more affordable device types as well which take their sweet time starting up and live in our p75+ range. We manage an app-wide error budget primarily through observability, alerting and experimentation freezes when new work impacts startup metrics meaningfully. There are still times where we allow a purposeful (and usually temporary) regression to startup, if the value added is substantial and optimizations are likely to materialize, but we work with teams to ensure we are continuously paying down performance debt, defer unnecessary work, and get the user to the in-app experience they intended as quickly as possible.
Tech Stack Modernization as a Driver for Stability & Performance
Our ongoing commitment to mobile modernization has been a powerful driver for enhancing and maintaining app stability and performance. By transforming our development processes and accelerating iteration speeds, we’ve significantly improved our ability to work on new features while maintaining high standards for app stability and performance; it’s no longer a tradeoff teams have to regularly make.
Our modernization journey centered around transitioning to a monorepo architecture, modularized by feature, and integrating a modern, cutting-edge tech stack that developers were excited to work in and could be much more agile within. This included adopting a pure Kotlin, Anvil, GraphQL, MVVM, Compose-based architecture and leveraging our design system for brand consistency. Our modernization efforts are well-established these days (and we talk about them at conferences quite often), and as we’ve progressed, we’ve been able to double-down on improvements built on our choices. For example:
- Going full Kotlin meant we could now leverage KSP and move away from KAPT. Coroutine adoption took off, and RxJava disappeared from the codebase much faster, reducing feature complexity and lines of code. We’ve added plugins to make creating and maintaining features easy.
- Going pure GQL meant having to maintain and debug two network stacks, retry logic and traffic payloads was mostly a thing of the past for feature developers. Feature development with GQL is a golden path. We’ve been quite happy leveraging Apollo on Android and taking advantage of features, like normalized caching, for example, to power more delightful user experiences.
- Going all in on Anvil meant investing in simplified DI boilerplate and feature code, investing in devx plugins and more build improvements to keep build times manageable.
- Adopting Compose has been a great investment for Reddit, both in the app and in our design system. Google’s commitment to continued stability and performance improvements meant that this framework has scaled well alongside Reddit’s app investments and delivers more compelling and performant features as it matures.
Our core surfaces, like feeds, video, and post detail page have undergone significant refactors and improvements for further devx and performance gains, which you can read all about on the Reddit Engineering blog as well. The feed rewrites, as an example, resulted in much more maintainable code using modern technologies like Compose to iterate on, a better developer experience in a space pretty much all teams at Reddit need to integrate with, and Reddit users get their memes and photoshop battle content hundreds of milliseconds faster than before. Apollo GQL’s normalized caching helped power instant comment loading on the post details page. These are investments we can afford to make now that we are future focused instead of spending our time mired in so much legacy code.
These cleanup celebrations also had other upsides. Users noticed and sentiment analysis improved. Our binary got smaller and our app startup and runtime improved demonstrably. Our testing infrastructure also became faster, more scalable, and cost-effective as the app performance improved. As we phased out legacy code, maintenance burdens on teams were lessened, simplifying on-call runbooks and reducing developer navigation through outdated code. This made it easier to prioritize stability and performance, as developers worked with a cleaner, more consistent codebase. Consequently, developer satisfaction increased as build times and app size decreased.
Figure 4: App Size & Complexity Go Down. Developer Happiness Go Up.
By early 2024, we completed this comprehensive modularization, enabling major feature teams—such as those working on feeds, video players, and post details—to rebuild their components within modern frameworks with high confidence that on the other side of those migrations, their feature velocity would be greater and they’d have a solid foundation to build for the future in more performant ways. For each of the tech stack choices we’ve made, we’ve invested in continuously improving the developer experience around those choices so teams have confidence in investing in them and that they get better and more efficient over time.
Affording Test Infrastructure When Your CI Times Are Already Off The Charts
By transitioning to a monorepo structure modularized by feature and adopting a modern tech stack, we’ve made our codebase honor separation of concerns and become much more testable, maintainable and pleasant to work in. It is possible for teams to work on features and app stability/performance in tandem instead of having to choose one or the other and have a stronger quality focus. This shift not only enhanced our development efficiency but also allowed us to implement robust test infrastructure. By paying down developer experience and performance debt, we can now afford to spend some of our resources on much more robust testing strategies. We improved our unit test coverage from 5% to 70% and introduced intelligent test sharding, leading to sustainable cycle times. As a result, teams could more rapidly address stability and performance issues in production and develop tests to ensure ongoing
Figure 5: Android Repo Unit Test Coverage Safeguarding App Stability & Performance
Our modularization efforts have proven valuable, enabling independent feature teams to build, test, and iterate more effectively. This autonomy has also strengthened code ownership and streamlined issue triaging. With improved CI times now in the 30 minute range @ p90 and extensive test coverage, we can better justify investments in test types like performance and endurance tests. Sharding tests for performance, introducing a merge queue to our monorepo, and providing early PR results and artifacts have further boosted efficiency.
By encouraging standardization of boilerplate, introducing checks and golden paths, we’ve decoupled some of the gnarliest problems with our app stability and performance while being able to deliver tools and frameworks that help all teams have better observability and metrics insights, in part because they work in stronger isolation where attribution is easier. Teams with stronger code ownership are also more efficient with bug fixing and more comfortable resolving not just crashes but other types of performance issues like memory leaks and startup regressions that crop up in their code.
Observe All The Things! …Sometimes
As our app-wide stability and performance metrics stabilized and moved into healthier territory, we looked for ways to safeguard those improvements and make them easier to maintain over time.
We did this a few key ways:
- We introduced on-call programs to monitor, identify, triage and resolve issues as they arose, when fixes are most straightforward.
- We added reporting and alerting as CI checks, experiment checks, deployment checks, Sourcegraph observability and real-time production health checks.
- We took on second-degree performance metrics like ANRs and memory leaks and used similar patterns to establish, improve and maintain those metrics in healthy zones
- We leveraged Google’s ANR guide to make improvements.
- We leveraged Square’s LeakCanary 2 and 3 for memory leaks
- We scaled our beta programs to much larger communities for better signals on app stability and performance issues prior to deployments
- We introduced better observability and profiling tooling for detection, debugging, tracing and root cause analysis, Perfetto for tracing and Bitdrift for debugging critical-path beta crashes
- We introduced screen-level performance metrics, allowing teams to see how code changes impacted their screen performance with metrics like time-to-interactive, time to first draw, and slow and frozen frame rates.
Today, identifying the source of app-wide regressions is straightforward. Feature teams use screen-specific dashboards to monitor performance as they add new features. Experiments are automatically flagged for stability and performance issues, which then freeze for review and improvements.
Our performance dashboards help with root cause analysis by filtering data by date, app version, region, and more. This allows us to pinpoint issues quickly:
- Problem in a specific app version? Likely from a client update or experiment.
- Problem not matching app release adoption? Likely from an experiment.
- Problem across Android and iOS? Check for upstream backend changes.
- Problem in one region? Look into edge/CDN issues or regional experiments.
We also use trend dashboards to find performance improvement opportunities. For example, by analyzing user engagement and screen metrics, we've applied optimizations like code cleanup and lazy loading, leading to significant improvements. Recent successes include a 20% improvement in user first impressions on login screens and up to a 70% reduction in frozen frame rates during onboarding. Code cleanup in our comment section led to a 77% improvement in frozen frame rates on high-traffic screens.
These tools and methods have enabled us to move quickly and confidently, improving stability and performance while ensuring new features are well-received or quickly reverted if necessary. We’re also much more proactive in keeping dependencies updated and leveraging production insights to deliver better user experiences faster.
Obfuscate & Shrink, Reflect Less
We have worked closely with partners in Google Developer Relations to find key opportunities for more performance improvements and this partnership has paid off over time. We’ve resolved blockers to making larger improvements and built out better observability and deployment capabilities to reduce the risks of making large and un-gateable updates to the app. Taking advantage of these opportunities for stability, performance, and security gains required us to change our dependency update strategy to stay closer to current than Reddit had in the past. These days, we try to stay within easy update distance of the latest stable release on critical dependencies and are sometimes willing to take more calculated upgrade risks for big benefits to our users because we can accurately weigh the risks and rewards through observability, as you’ll see in a moment.
Let’s start with how we optimized and minified our release builds to make our app leaner and snappier. We’d been using R8 for a long time, but enabling R8 “Full Mode” with its aggressive optimizations took some work, especially addressing some code still leveraging legacy reflection patterns and a few other blockers to strategic dependency updates that needed to be addressed first. Once we had R8 Full Mode working, we kept it baking internally and in our beta for a few weeks and timed the release to be a week when little else was going to production, in case we had to roll it back. Luckily, the release went smoothly and we didn’t need to use any contingencies, which then allowed us to move on to our next big updates. In production, we saw an immediate improvement of about 20% to the percentage of daily active users who experienced at least one Application Not Responding event (ANR). In total, we saw total ANRs for the app drop by about 30%, largely driven by optimizations improving setup time in dependency injection code, which makes sense. There’s still a lot more we can do here. We still have too many DEX files and work to improve this area, but we got the rewards we expected out of this effort and it continues to pay off in terms of performance. Our app ratings, especially around performance, got measurably better when we introduced these improvements.
Major Updates Without Major Headaches
You can imagine with a big monolith and slow build times, engineers were not always inclined to update dependencies or make changes unless absolutely necessary. Breaking up the app monolith, having better observability and incident response turnaround times, and making the developer experience more reasonable has led to a lot more future-facing requests from engineering. For example, there's been a significant cultural shift at Reddit in mobile to stay more up-to-date with our tooling and dependencies and to chase improvements in frameworks APIs for improved experiences, stability, and performance, instead of only updating when compelled to.
https://i.redd.it/exu8w2jd69pd1.gif
We’ve introduced tooling like Renovate to help us automate many minor dependency updates but some major ones, like Compose upgrades, require some extra planning, testing, and a quick revert strategy. We had been working towards the Compose 1.6+ update for some time since it was made available early this year. We were excited about the features and the performance improvements promised, especially around startup and scroll performance, but we had a few edge-case crashes that were making it difficult for us to deploy it to production at scale.
We launched our new open beta program with tens of thousands of testers, giving us a clear view of potential production crashes. Despite finding some critical issues, we eventually decided that the benefits of the update outweighed the risks. Developers needed the Compose updates for their projects, and we anticipated users would benefit from the performance improvements. While the update caused a temporary dip in stability, marked by some edge case crashes, we made a strategic choice to proceed with the release and fix forward. We monitored the issues closely, fixed them as they arose, and saw significant improvements in performance and user ratings. Three app releases later, we had reported and resolved the edge cases and achieved our best stability and performance on Android to date.
Results wise? We saw improvements across the app and it was a great exercise in testing all our observability. We saw app-wide cold start app startup improvements in the 20% range @ p50 and app-wide scroll performance improvements in the 15% range @ p50. We also saw marked improvements on lower-end device classes and stronger improvements in some of our target emerging market geos. These areas are often more sensitive to app size, startup ANRs and performance constrained so it makes sense they would see outsized benefits on work like this.
Figure 7: App Start Benchmark Improvements
We also saw:
- Google Play App Vitals: Slow Cold Start Over Time improved by ~13%, sustained.
- Google Play App Vitals: Excessive Frozen Frames Over Time improved by over ~10%, sustained.
- Google Play App Vitals: Excessive Slow Frames Over Time improved by over ~30%, sustained.
We saw sweeping changes, so we also took this opportunity to check on our screen-level performance metrics and noted that every screen that had been refactored for Compose (almost 75% of our screens these days) saw performance improvements. We saw this in practice: no single screen was driving the overall app improvements from the update. Any screen that has modernized (Core Stack/Compose) saw benefits. As an example, we focused on the Home screen and saw about a 15% improvement in scroll performance @ p50, which brought us into a similar performance zone as our iOS sister app, while p90s are still significantly worse on Android mostly due to supporting a much broader variety of lower-end hardware available to support different price points for worldwide Android users
Figure 8: App-Wide Scroll Performance Improvements & Different Feeds Impacted By the Compose Update
The R8 and Compose upgrades were non-trivial to deploy in relative isolation and stabilize, but we feel like we got great outcomes from this work for all teams who are adopting our modern tech stack and Compose. As teams adopt these modern technologies, they pick up these stability and performance improvements in their projects from the get-go, not to mention the significant improvements to the developer experience by working solely in modularized Kotlin, MVVM presentation patterns, Compose and GraphQL. It’s been nice to see these improvements not just land, but provide sustained improvements to the app experiences.
Startup and Baseline Profiles As the Cherry On Top of the Banana Split That Is Our Performance Strategy
Because we’ve invested in staying up-to-date in AGP and other critical dependencies, we are now much more capable of taking advantage of newer performance features and frameworks available to developers. Baseline profiles, for example, have been another way we have made strategic performance improvements to feature surfaces. You can read all about them on the Android website.
Recently, Reddit introduced and integrated several Baseline Profiles on key user journeys in the app and saw some positive improvements to our performance metrics. Baseline profiles are easy to set up and leverage and sometimes demonstrate significant improvements to the app runtime performance. We did an audit of important user journeys and partnered with several orgs, from feeds and video to subreddit communities and ads, to leverage baseline profiles and see what sorts of improvements we might see. We’ve added a handful to the app so far and are still evaluating more opportunities to leverage them strategically.
Adding a baseline profile to our community feed, for example, led to:
- ~15% improvement in time-to-first-draw @ p50
- ~10% improvement to time-to-interactive @ p50
- ~35% improvement in slow frames @ p50
We continue to look for more opportunities to leverage baseline profiles and ensure they are easy for teams to maintain.
Cool Performance Metrics, But How Do Users Feel About Them?
Everyone always wants to know how these performance improvements impact business metrics and this is an area we are investing in a lot lately. Understanding how performance improvements translate into tangible benefits for our users and business metrics is crucial, and we are still not good at flexing this muscle. This is a focus of our ongoing collaboration with our data science team, as we strive to link enhancements in stability and performance to key metrics such as user growth, retention, and satisfaction. Right now? We really want to be able to stack rank the various performance issues we know about to better prioritize work.
We do regularly get direct user validation for our improvements and Google Play insights can be of good use on that front. Here’s a striking example of this is the immediate correlation we observed between app-wide performance upgrades and a substantial increase in positive ratings and reviews on Google Play. Notably, these improvements had a particularly pronounced impact on users with lower-end devices globally, which aligns seamlessly with our commitment to building inclusive communities and delivering exceptional experiences to users everywhere.
Figure 9: Quelle Surprise: Reddit Users Like Performance Improvements
So What’s Next?
Android stability and performance at Reddit are at their best in years, but we recognize there is still much more to be done to deliver exceptional experiences to users. Our approach to metrics has evolved significantly, moving from a basic focus to a comprehensive evaluation of app health and performance. Over time, we’ve incorporated many other app health and performance signals and expanded our app health programs to address a wider range of issues, including ANRs, memory leaks, and battery life. Not all stability issues are weighted equally these days. We’ve started prioritizing user-facing defects much higher and built out deployment processes as well as automated bug triaging with on-call bots to help maintain engineering team awareness of production impacts to their features. Similarly on the performance metrics side, we moved beyond app start to also monitor scroll performance and address jank, closely monitor video performance, and we routinely deep-dive screen-based performance metric regressions to resolve feature-specific issues.
Our mobile observability has given us the ability to know quickly when something is wrong, to root-cause quickly, and to tell when we’ve successfully resolved a stability or performance issue. We can also validate that updates we make, be it a Compose update or an Exoplayer upgrade, is delivering better results for our users and use that observability to go hunting for opportunities to improve experiences more strategically now that our app is modularized and sufficiently decoupled and abstracted. While we wouldn’t say our app stability and performance is stellar yet, we are on the right path and we’ve clawed our way up into the industry standard ranges amongst our peers from some abysmal numbers. Building out great operational processes, like deployment war rooms and better on-call programs has helped support better operational excellence around maintaining those app improvements and expanding upon them.
These days, we have a really great mobile team that is committed to making Android awesome and keeping it that way, so if these sorts of projects sound like compelling challenges, please check out the open roles on our Careers page and come take Reddit to the next level.
ACKs
These improvements could not have been achieved without the dedication and support of every Android developer at Reddit, as well as our leadership’s commitment to prioritizing stability and performance, and fostering a culture of quality across the business. We are also deeply grateful to our partners in performance on the Google Developer Relations team. Their insights and advice has been critical to our success in making improvements to Android performance at scale with more confidence. Finally, we appreciate that the broader Android community is open and has such a willingness to talk shop, and workshop insights, tooling ideas, architecture patterns and successful approaches to better serve value to Android users. Thank you for sharing what you can, when you can, and we hope our learnings at Reddit help others deliver better Android experiences as well.
23:15 UTC
An issue was re-port-ed
Written by Tony Snook
AI Generated Image of Hackers surrounding a laptop breaking into a secured vault.
tl;dr
A researcher reported that we had an endpoint exposed to the Internet leaking metrics. That exposure was relatively innocuous, but any exposure like this carries some risk, and it actually ended up tipping us off about a larger, more serious exposure. This post discusses that incident, provides a word of warning for NLB usage with Kubernetes, and shares some insight into Reddit’s tech stack.
Background
Here at Reddit, the majority of our workloads are run on self-managed Kubernetes clusters on EC2 instances. We also leverage a variety of controllers and operators to automate various things. There’s a lot to talk about there, and I encourage you to check out this upcoming KubeCon talk! This post focuses specifically on the controller we use to provision our load balancers on AWS.
The Incident
On June 26th, 2024, we received a report from a researcher showing how they could pull Prometheus metrics from an exposed port on a random IP address that supposedly belonged to us, so we promptly kicked off an incident to rally the troops. Our initial analysis of the metrics led us to believe the endpoint belonged to one particular business area. As we pulled in representatives from the area, we started to believe that it may be more widespread. One responder asked, “do we have a way to grep across all Reddit allocated public IP addresses (across all of our AWS accounts)?” We assumed this was coming from EC2 due to our normal infrastructure (no reason to believe this is rogue quite yet). With our config and assets database, it was as simple as running this query:
SELECT * FROM aws_ec2_instances WHERE public_ip_address = “<the IP address from the report>”;That returned all of the instance details we wanted, e.g. name, AWS account, tags, etc. From our AWS tags, we could tell it was a Kubernetes worker node, and the typical way to expose a service directly on a Kubernetes worker node is via NodePort. We knew what port number was used from the report, so we focused on identifying which service was associated with it. You can use things like a Kubernetes Web UI, but knowing which cluster to target, one of our responders just used kubectl directly:
kubectl get svc -A | grep <the port from the report>Based on the service name, we knew what team to pull in, and they quickly confirmed it would be okay to nuke that service. In hindsight, this was the quick and dirty way to end the exposure, but could have made further investigation difficult, and we should have instead blocked access to the service until we determined the root cause.
That all happened pretty quickly, and we had determined that the exposure was innocuous (the exposed information included some uninteresting names related to acquisitions and products already known to the public, and referenced some technologies we use, which we would happily blog about anyway), so we lowered the severity level of the incident (eliminating the expectation of urgent, after-hours work from responders), moved the incident into monitoring mode (indicating no work in progress), and started capturing AIs to revisit during business hours.
We then started digging into why it was exposed the way it was (protip: use the five-whys method). The load balancer for this service was supposed to be “internal”. We also started wondering why our Kyverno policies didn’t prevent this exposure. Here’s what we found…
The committed code generated a manifest that creates an “internal” load balancer, but with a caveat: no configuration for the “loadBalancerSourceRanges” property. That property specifies the CIDRs that are allowed to access the NLB, and if left unset, it defaults to ["0.0.0.0/0"] (ref). That is an IP block containing all possible IP addresses, meaning any source IP address would be allowed to access the NLB. That configuration by itself would be fine, because the Network Load Balancer (NLB) doesn't have a public IP address. But in our case, because of design decisions for our default AWS VPC made many years ago, these instances have publicly addressable IPs.
AWS instances by default do not allow any traffic to any ports; they also need security groups (think virtual firewalls) configured for this. So, why on earth would we have a security group rule exposing that port to the Internet? It has to do with how we provision our AWS load balancers, and a specific nuance between Network Load Balancers (NLBs) and Classic Elastic Load Balancers (ELBs). Here’s the explanation from u/grumpimusprime:
NLBs are weird. They don't work like you'd expect. A classic ELB acts as a proxy, terminating a connection and opening a new one to the backend. NLBs, instead, act as a passthrough, just forwarding the packets along. Because of this, the security group rules of the backing instances are what apply. The Kubernetes developers are aware of this behavior, so to make NLBs work, they dynamically provision the security group rule(s) necessary for traffic to make it to the backing instance. The expectation, of course, being that if you're using an internal load balancer, your instances aren't directly Internet-exposed, so this isn't problematic. However, we've hit an edge case.
Ah ha! But wait… Does that mean we might have other services exposed in other clusters? Yup.
We immediately bumped the Severity back up and tagged responders to assess further. We ended up identifying a couple more innocuous endpoints, but the big “oh shit” finding was several exposed ingresses for sensitive internal gRPC endpoints. Luckily we were able to patch these up quickly, and we found no signs of exploitation while exposed. :phew:
Takeaways
- Make sure that the “loadBalancerSourceRanges” property is set on all LoadBalancer Services, and block creation of LoadBalancer Services when the value of that property contains “0.0.0.0/0”. These are relatively simple to implement via Kyverno.
- Consider swapping to classic ELBs instead of NLBs for external-facing service exposure, because practically the NLB config means that the nodes themselves must be directly exposed. That may be fine, but this creates a sharp edge, of which most of our Engineers aren’t aware.
- Our bug bounty program tipped us off about this problem, as it has many others. I cannot overstate the importance and value of our bug bounty program!
- Reliable external surface monitoring is important. Obviously, we want to prevent inadvertent exposures like this, but failing prevention, we should also detect it before our researchers or any malicious actors. We pay for a Continuous Attack Surface Monitoring (CASM) service but it can’t handle the ephemeral nature of our fleet (our Kubernetes nodes only last for 2 weeks tops). We’re discussing a simple nmap-based external scanning solution to alert on this scenario moving forward, as well as investing more in posture monitoring (i.e. alerts based on vulnerabilities apparent in our config database).
- Having convenient tooling (Slack bots, chat ops, integrations, notifications, etc.) is a huge enabler. It is also really important to have severity ratings codified (details on what severity levels mean, and expectations for response). That helps get things going smoothly, as everyone knows how to prioritize the incident as soon as they are pulled in. And, having incident roles well-defined (i.e. commanders and scribes have different responsibilities than responders) keeps people focused on their specific tasks and maximizes efficiency during incident response..
I want to thank everyone who helped with this incident for keeping Reddit secure, and thank you for reading!
20:40 UTC
“Breaking Barriers: Enhancing Accessibility to Reddit with AI” at KDD 2024
Written by Rosa Català.
At Reddit, our mission is to bring community, belonging, and empowerment to everyone, everywhere. This year, our team had the incredible opportunity to present a hands-on tutorial titled "Breaking Barriers: AI-Enabled Accessibility to Social Media Content" [paper, repo] at the ACM SIGKDD 2024 conference in Barcelona, Spain. We presented in front of a very engaged audience on August 26th. This tutorial highlighted our efforts and commitment to making Reddit content accessible and inclusive for all, especially for individuals with disabilities.
Why Accessibility Matters
User generated content platforms like Reddit offer endless opportunities for individuals to connect, share, and access information. However, accessing and interacting with content can be significantly challenging for individuals with disabilities. Ensuring that our platform is accessible to everyone is not just a goal—it's a responsibility. We see accessibility (a11y) as a fundamental aspect of inclusivity. By removing barriers and ensuring content is easy for all users to navigate, understand, and enjoy, we aim to empower everyone to participate fully in our community and share their perspectives.
The Power of AI in Accessibility
Our tutorial at KDD 2024 focused on leveraging Artificial Intelligence (AI) to enhance multimodal content accessibility for individuals with different disabilities, including hearing, visual, and cognitive impairments. Recent advancements in Multimodal Large Language Models (MLLMs) have empowered AI to analyze and understand diverse media formats, such as text, images, audio, and video. These capabilities are crucial for creating more accessible and inclusive social media environments.
Tutorial Objectives and Key Takeaways
The tutorial was designed to bridge the gap between AI research and real-world applications, providing participants with hands-on experience in designing and implementing AI-based solutions for accessibility:
- Image Short Captions: Participants learned how to deploy and prompt various multimodal LLMs, such as LLaVA, Phi-3-Vision, and imp-v1-3b, to generate short, descriptive captions for social media images. This helps users with visual impairments understand and engage with visual content.
- Audio Clip Transcripts and Video Descriptions: We demonstrated how to use open-source speech-to-text models (like Whisper) to transcribe audio clips to text and produce closed captions. For video content, we guided participants through a pipeline combining keyframe extraction, image captioning, and audio transcript summarization using LLMs, enhancing accessibility for hearing-impaired users.
- Complex Post Summarization: Addressing the needs of users with cognitive impairments, we explored how to use LLMs to summarize lengthy or complex media posts, making them easier to understand and engage with the platform conversation.
- Bonus Use Case - Text to Speech: For participants who progressed quickly, we introduced a bonus session on using open-source models, such as SpeechT5 and Bark, to convert text to speech, aiding users with visual impairments.
Throughout the tutorial, we emphasized the strengths and limitations of each technique, providing a comprehensive overview of the challenges and opportunities for future development in this space.
Impact on Society
AI-enabled accessibility has immense potential for transformative societal impact. By enhancing accessibility, we can foster a more inclusive, equitable, and accessible society where individuals with disabilities are empowered to actively engage in the digital world. Some of the key benefits include:
- Inclusion and Empowerment: Providing equal access to social media platforms allows individuals with disabilities to connect, share experiences, and contribute fully to the digital world.
- Reduced Isolation: Breaking down barriers to social interaction reduces feelings of isolation and fosters a sense of belonging.
- Improved Educational Outcomes: Enhancing accessibility allows students with disabilities equitable access to learning resources and discussions.
- Greater Civic Participation: Enabling individuals with disabilities to engage in online political and social discussions helps shape public discourse and advocate for their rights.
- Increased Employment Opportunities: Improving access to information and communication tools can support individuals with disabilities in seeking and securing employment.
- Economic Benefits: By increasing the participation of individuals with disabilities in the digital economy, AI-enabled accessibility can contribute to economic growth and innovation.
Looking Ahead
Our tutorial was met with great enthusiasm, with over 30 participants engaging in lively discussions and sharing valuable insights. The positive feedback we received highlights the importance of accessibility in the digital age and the role AI can play in making social media more inclusive.
We hope to continue raising awareness about the importance of accessibility and look forward to further collaborations to develop and implement AI-driven solutions that make digital content more accessible to all.
For more details, you can explore our tutorial materials on GitHub here and read the full paper on the ACM Digital Library here.
Together, let’s break barriers and build a more inclusive world.
23:33 UTC
Engineering Practices for Accessible Feature Development
Written by the Reddit Accessibility Team.
This is the first of what I hope will be several blog posts about accessibility at Reddit. Myself and several others have been working full time on accessibility since last year, and I’m excited to share some of our progress and learnings during that time. I’m an iOS Engineer and most of my perspective will be from working on accessibility for the Reddit iOS app. But the practices discussed in this blog post apply to how we develop for all platforms.
I think it’s important to acknowledge that, while I’m very proud of the progress that we’ve made so far in making Reddit more accessible, there is still a lot of room for improvement. We’re trying to demonstrate that we will respond to accessibility feedback whilst maintaining a high quality bar for how well the app works with assistive technologies. I can confidently say that we care deeply about delivering an accessibility experience that doesn’t just meet the minimum standard but is actually a joy to use.
Reddit’s mission is to bring community, belonging, and empowerment to everyone in the world, and it’s hard not to feel the gravity of that mission when you’re working on accessibility at Reddit. We can’t accomplish our company’s mission if our apps don’t work well with assistive technologies. As an engineer, it’s a great feeling when you truly know that what you’re working on is in perfect alignment with your company’s mission and that it’s making a real difference for users.
I want to kick things off by highlighting five practices that we’ve learned to apply while building accessible features at Reddit. These are practices that have been helpful whether we are starting off a new project from scratch, or making changes to an existing feature. A lot of this is standard software engineering procedure; we don’t need to reinvent the wheel. But I think it’s important to be explicit about the need for these practices because they remind us to keep accessibility in our minds through all phases of a project, which is critical to ensuring that accessibility in our products continues to improve.
1 - Design specs
Accessibility needs to be part of the entire feature design and development process, and that all starts with design. Reddit follows a typical feature design process where screens are mocked up in Figma. The mockup in Figma gives an engineer most of the information they’ll need to build the UI for that feature, including which components to use, color and font tokens, image names, etc.
What we realized when we started working full time on accessibility is that these specs should also include values for the properties we need to set for VoiceOver. VoiceOver is the screen reader built into the iOS and macOS operating systems. Screen readers provide a gestural or keyboard interface that works by moving a focus cursor between on-screen elements. The attributes that developers apply to on-screen elements control what the screen reader reads for an element and other aspects of the user experience, such as text input, finding elements, and performing actions.
On iOS there are several attributes that can be specified on an element to control its behavior: label, hint, value, traits, and custom actions. The label, hint, and value all affect what VoiceOver reads for an element, all have specific design guidance from Apple on how to write them, and all require localized copy that makes sense for that feature.
The traits and custom actions affect what VoiceOver reads as well as how a user will interact with the element. Traits are used to identify the type of element and also provide some details about what state the element is in. Custom actions represent the actions that can be performed on the focused element. We use them extensively for actions like upvoting or downvoting a post or comment.
Having an accessibility spec for these properties is important because engineers need to know what to assign to each property, and because there are often many decisions to make regarding what each property should be set to. It’s best to have the outcome of those decisions captured in a design spec.
Team members need to be asking each other how VoiceOver interaction will work with feature content, and the design phase is the right time to be having these conversations. The spec is where we decide which elements are grouped together, or hidden because they are decorative. It’s also where we can have discussions about whether an action should be a focusable button, or if it should be provided as a custom action.
In our early design discussions for how VoiceOver would navigate the Reddit feed, the question came up of how VoiceOver would focus on feed cells. Would every vote button, label, and other action inside of the cell be focusable, or would we group the elements together with a single label and a list of custom actions? If we did group the elements together, should we concatenate accessibility labels together in the visual order, or base them on which information is most important?
Ultimately we decided that it was best to group elements together so that the entire feed cell becomes one focusable element with accessibility labels that are concatenated in the visual order. Buttons contained within the cell should be provided as custom actions. For consistency, we try to apply this pattern any time there is a long list of repeated content using the same cell structure so that navigation through the list is streamlined.
We think it’s important to make the VoiceOver user experience feel consistent between different parts of the app. There are platform and web standards when it comes to accessibility, and we are doing our best to follow those best practices. But there is still some ambiguity, especially on mobile, and having our own answer to common questions that can be applied to a design spec is a helpful way of ensuring consistency.
Writing a design spec for accessibility has been the best way to make sure a feature ships with a good accessibility experience from the beginning. Creating the design spec makes accessibility part of the conversation, and having the design spec to reference helps everyone understand the ideal accessibility experience while they are building and testing the feature.
2 - Playtests
Something that I think Reddit does really well are internal playtests. A playtest might go by many names at other companies, such as a bug bash. I like the playtest name because the spirit of a playtest isn’t just to file bugs – it’s to try out something new and find ways to make it better.
Features go through several playtests before they ship, and the accessibility playtest is a new one that we’ve added. The way it works is that the accessibility team and a feature team get together to test with assistive technologies enabled. What I like the most about this is that everyone is testing with VoiceOver on - not just the accessibility team. The playtest helps us teach everyone how to test for and find accessibility issues. It’s also a good way to make sure everyone is aware of the accessibility requirements for the feature.
We typically are able to find and fix any major issues after these playtests, so we know they’re serving an important role in improving accessibility quality. Further, I think they’re also of great value in raising the awareness of accessibility across our teams and helping more people gain proficiency in developing for and testing with assistive technologies.
Custom actions are one example of a VoiceOver feature that comes up a lot in our playtests. Apple introduced custom actions in iOS 8, and since then they’ve slowly become a great way to reduce the clutter of repetitive actions that a user would otherwise have to navigate through. Instead of needing to focus on every upvote and downvote button in the Reddit conversation view, we provide custom actions for upvoting and downvoting in order to streamline the conversation reading experience. But many developers don’t know about them until they start working on accessibility.
One of the impulses we see when people start adding custom actions to accessibility elements is that they’ll add too many. While there are legitimate cases in Reddit where there are over 10 actions that can be performed on an element like a feed post, where possible we try to limit the available actions to a more reasonable number.
We typically recommend presenting a more actions menu with the less commonly used actions. This presented action sheet is still a list of focusable accessibility elements, so it still works with VoiceOver. Sometimes we see people try to collapse those actions into the list of custom actions instead, but we typically want to avoid that so that the primary set of custom actions remain streamlined and easy to use.
Holding a playtest allows us to test out the way a team has approached screen reader interaction design for their feature. Sometimes we’ll spot a way that custom actions could improve the navigation flow, or be used to surface an action that wouldn’t otherwise be accessible. The goal is to find accessibility experiences that might feel incomplete and improve them before the feature ships.
3 - Internal documentation
In order to really make the entire app accessible, we realized that every engineer needs to have an understanding of how to develop accessible features and fix accessibility issues in a consistent way. To that end, we’ve been writing internal documentation for how to address common VoiceOver issues at Reddit.
Simply referring developers to Apple’s documentation isn’t as helpful as explaining the full picture of how to get things done within our own code base. While the Reddit iOS app is a pretty standard native UIKit iOS app, familiarity with the iOS accessibility APIs is only the first step to building accessible features. Developers need to use our localization systems to make sure that our accessibility labels are localized correctly, and tie into our Combine publishing systems to make sure that accessibility labels stay up to date when content changes.
The accessibility team isn’t fixing every accessibility issue in the app by ourselves: often we are filing tickets for engineers on other teams responsible for that feature to fix the issue. We’ve found that it’s much better to have a documentation page that clearly explains how to fix the issue that you can link in a ticket. The issues themselves aren’t hard to fix if you know what to look for, but the documentation reduces friction to make sure the issue is easy for anyone to fix regardless of whether or not they have worked on accessibility before.
The easier we can make it for anyone at Reddit to fix accessibility issues, the better our chances of establishing a successful long-term accessibility program, and helpful documentation has been great for that purpose.
Internal documentation is also critical for explaining any accessibility requirements that have a subjective interpretation, such as guidelines for reducing motion. Reduce motion has been a staple of iOS accessibility best practices for around a decade now, but there are varying definitions for what that setting should actually change within the app.
We created our own internal documentation for all of our motion and autoplay settings so that teams can make decisions easily about what app behavior should be affected by each setting. The granularity of the settings helps users get the control they need to achieve the app experience they’re looking for, and the documentation helps ensure that we’re staying consistent across features with how the settings are applied in the app.
4 - Regression testing
We’re trying to be very careful to avoid regressing the improvements that we have made to accessibility by using end to end testing. We’ve implemented several different testing methodologies to try and cover as much area as we can.
Traditional unit tests are part of the strategy. Unit tests are a great way to validate accessibility labels and traits for multiple different configurations of a view. One example of that might be toggling a button from a selected to an unselected state, and validating that the selected trait is added or removed.
Unit tests are also uniquely able to be used to validate the behavior of custom actions. We can write asynchronous test expectations that certain behavior will be invoked when the custom action is performed. This plays very well with mock objects which are a core part of our existing unit test infrastructure.
Accessibility snapshot tests are another important tool that we’ve been using. Snapshot tests have risen in popularity for quickly and easily testing multiple visual configurations of a view. A snapshot image captures the appearance of the view and is saved in the repository. On the next test run, a new image is captured for the same view and compared to the previous image. If the two images differ, the test fails, because a change in the view’s appearance was not expected.
We can leverage these snapshot tests for accessibility by including a visual representation of each view’s accessibility properties, along with a color coding that indicates the view’s focus order within its container. We’re using the AccessibilitySnapshot plugin created by Cash app to generate these snapshots.
This technique allows us to fail a test if the accessibility properties of a view change unexpectedly, and since the snapshot tests are already great for testing many different configurations we’re able to achieve a high degree of coverage for each of the ways that a view might be used.
Apple also added a great new capability in Xcode 17 to run Accessibility Audits during UI Automation tests. We’ve begun adding these audits to some of our automated tests recently and have been pleased with the results. We do find that we need to disable some of the audit types, but the audit system makes it easy to do that, and for the audit types where we do have good support, this addition to our tests is proving to be very useful. I hope that Apple will continue to invest in this tool in the future, because there is a lot of potential.
5 - User feedback
Above all, the best thing that we can do to improve accessibility at Reddit is to listen to our users. Accessibility should be designed and implemented in service of its users' needs, and the best way to be sure of that is to listen to user feedback.
We’ve conducted a lot of interviews with users of many different assistive technologies so that we can gather feedback on how our app performs with VoiceOver enabled, with reduced motion enabled, with larger font sizes, and with alternative input mechanisms like voice control or switch control. We are trying to cover all of the assistive technologies to the best of our abilities, and feedback has driven a lot of our changes and improvements over the last year.
Some of the best feedback we’ve gotten involves how VoiceOver interacts with long Reddit posts and comments. We have clear next steps that we’re working on to improve the experience there.
We also read a lot of feedback posted on Reddit itself about the app’s accessibility. We may not respond to all of it, but we read it and do our best to incorporate it into our roadmap. We notice things like reports of unlabeled buttons, feedback about the verbosity of certain content, or bugs in the text input experience. Bugs get added to the backlog, and feedback gets incorporated into our longer term roadmap planning. We may not always fix issues quickly, but we are working on it.
The road goes on forever and the journey never ends
The work on accessibility is never finished. Over the last year, we systematically added accessibility labels, traits, and custom actions to most of the app. But we’ve learned a lot about accessibility since then, and gotten a ton of great feedback from users that needs to be incorporated. We see accessibility as much more than just checking a box to say that everything has a label; we’re trying to make sure that the VoiceOver experience is a top tier way of using the app.
Reddit is a very dense app with a lot of content, and there is a balance to find in terms of making the app feel easy to navigate with VoiceOver and ensuring that all of the content is available. We’re still actively working on improving that balance. All of the content does need to be accessible, but we know that there are better ways of making dense content easier to navigate.
Over the coming months, we’ll continue to write about our progress and talk more specifically about improvements we’re making to shipping features. In the meantime, we continue to welcome feedback no matter what it is.
If you’ve worked on accessibility before or are new to working on accessibility, let us know what you think about this. What else would you like to know about our journey, and what has been helpful to you on yours?
18:09 UTC
Bringing the Cybersecurity Community together through SnooSec
Written by: mattjay
Matt Johansen giving opening remarks at the first SnooSec in San Francisco on April 3, 2024
When I was first getting into cybersecurity, social media was in its infancy and big regional conferences were one of the main ways we got together. These were great but were a really big deal for my broke as a joke self. I had to rub a few pennies together, share badges, sleep on couches, etc. But it was at my first few conferences that I met the next 15 years of future bosses who I’ve worked with.
Also during this time we had smaller local meetups and conferences starting to form, from OWASP chapters, to the very first BSides, all the way to the citysec meetups like NYsec, Baysec, Sillisec, etc. - But during Covid a lot of these more casual smaller local meetups took a real hit. Coupled with our industry absolutely exploding in size, the tight knit sense of community started to feel like it was a nostalgic memory.
We missed these events and decided to step up in an attempt to do our part to bring them back by launching a new SnooSec meetup series. SnooSec is Reddit's new meetup series designed to bring the local cybersecurity community together for a fun night of casual learning, networking, and fun. Afterall, Reddit is all about community and most of my personal favorite subreddits are niche interest or hyper local.
The last two SnooSec meetups were a huge success. We had 50-70 people at both of them, ironed out some of the logistical challenges, and now have a huge pipeline of people looking to attend or present at future events.
Our plan is to run these meetups quarterly, alternating between our offices in San Francisco and New York. We’re still figuring out our best way to handle all the interest in giving talks. Stay tuned on that, but for now just reach out to us if you’re interested in speaking.
Join the r/SnooSec community to stay up to date on future SnooSec events.
16:14 UTC
Day in the Life of an Infrastructure Intern at Reddit
Written by Haley Patel
Hello world! My name is Haley, and I am thrilled to be a Snootern on Reddit’s Observability Team working from NYC this summer. My time at Reddit has been a transformative and unforgettable experience, and I’m excited to share this journey with all of you. Join me as I give you an inside look into a day in the life of an infrastructure intern at Reddit.
View from below of our office in the sky
Unlike many other interns spending the summer in NYC, I commute to the office from New Jersey using two trains: NJTransit and PATH. In my state, it is actually quite common to travel to out-of-state cities via train for work on the daily. To ensure I arrive at the office on time, I start my mornings early by waking up at 6:00 a.m., giving myself enough time to thoughtfully stare at my closet and select a stylish outfit for the day. One of my favorite aspects about working at Reddit is the freedom to wear clothes and jewelry that express my personality, and I love seeing my colleagues do the same (while remaining office appropriate of course).
Once I am ready to face the day, I head to the train station for my hour-long commute to the office. I find the commute relaxing as I use the time to read books and listen to music. The NYC Reddit office has an excellent selection of books that I enjoy browsing through during my breaks. Currently, I am reading Which Way is North, a book I discovered in our office’s little library. Engaging in these activities provides a valuable buffer for self-care and personal time before starting my day.
Once I arrive at the office, I head straight to the pantry for some free breakfast, whether it is a cup of iced coffee, Greek yogurt, or a bagel. Since we do not have any syrups for flavoring coffee, I devised my own concoction: Fairlife Vanilla Flavored Milk swirled into my iced latte base to create a vanilla protein iced latte. Thank me later …
Starting the morning in the canteen with my Vanilla Protein Latte
In the Flow
I like to start my day diving right into what I was working on the day before while my mind is fresh. I work on the Observability Team, which builds tools and systems that enable other engineers and technical users at Reddit to analyze the performance, behavior, and cost of their applications. Observability allows teams to monitor and understand what is happening inside of their applications, using that information to optimize performance, reduce costs, debug errors, and improve overall functionality. By providing these tools, we help other engineers at Reddit ensure their applications run smoothly, efficiently, and cost-effectively.
My intern project was concerned with improving the efficiency of collecting and routing metrics within our in-house built logging infrastructure. I built a Kubernetes operator in Go that dynamically and automatically scales metrics aggregators within all Reddit clusters. A major highlight of my project was deploying it to production and witnessing its real impact on our systems. I saw the operator prevent disruptions to our platforms during multiple major incidents, and observed a 50% reduction in costs associated with running the aggregators! Overall, it was a broadly scoped project, in which I learned a lot about distributed systems, Kubernetes, Go, and the open source components of our monitoring stack such as Grafana and Prometheus. It was an amazing opportunity to work on such an impactful project at Reddit’s scale and see the results firsthand!
I have to admit, when I first started this internship, I did not have any experience with the aforementioned technologies. Although I was eager to learn what I needed to complete the project, I was thankful to have a mentor to guide me along the way and demonstrate to me how each tool was implemented within the team’s specific environment. My mentor was the most amazing resource for me throughout my internship, and he definitely showed me the ropes of being a part of Observability and Infrastructure at Reddit. I am glad that Reddit pairs every intern with a mentor on their respective team, as it provides an opportunity to learn more about the team’s functions and project contexts.
When I was not working on my project or meeting with my team, I liked to engage in coffee chats with other Reddit employees, learning skills relevant to my project, and participating in the engaging activities organized by the Emerging Talent team for us Snooterns. I particularly enjoyed the coffee chats, where I had the chance to learn about others’ journeys to and through tech, as well as connect over shared hobbies and interests outside of work. Building friendships and connections with other Snoos at Reddit was a vital part of my experience, and I am excited to come out of this experience with lifelong friends.
5-9 After the 9-5
The Emerging Talent team at Reddit does an amazing job with organizing fun events during and after work to bond with other interns. Us Snooterns do seem to love baseball. Earlier in the summer, we all went to support the Snoo York Yankees (Reddit’s own softball team) during their game at Central Park. Exactly a month later, we were at Yankees Stadium watching the real Yankees play against the Mets.
The excitement in the air at Yankees Stadium was spectacular.
Going to the game with my fellow Snooterns was a fun activity, and it is safe to say that we definitely enjoyed the free food vouchers that we received. Thanks Reddit!
Key Takeaways
Interning at Reddit was a full-circle moment for me, as Reddit was one of the first social platforms I ever used. Frequenting Reddit mainly to discuss video games I enjoyed, I found like-minded communities that had lasting impacts on me. Through Reddit, I connected with people passionate about programming game mods, and even developing their own games, from which I joined a small developer team to help create a videogame that reached 12,000 players! That experience truly solidified my interest in programming, and now I have the opportunity to be part of the engineering team at Reddit and help bring community and belonging to everyone in the world!
One key takeaway that I gained from this experience is that software engineering is such a vast field, making it important to stay curious, retain a growth mindset, and learn new things along the way. Engineering decisions are results of compromise, built upon knowledge gained from past experiences and learnings. At Reddit, I learned about the importance of admitting when I did not know something, as it provided an opportunity to learn something new! Additionally, I have come to appreciate Reddit’s culture of promoting knowledge sharing and transparency, with Default Open being one of its core values that I resonate with.
In the 12 weeks I’ve been here at Reddit, I feel that I have grown immensely personally and professionally. The Reddit internship program gave me an opportunity to go above and beyond, teaching me that I can accomplish anything that I put my mind to, and breaking the boundaries imposter syndrome had set onto me. The support from Emerging Talent, my team, and other Snoos at Reddit made my summer worthwhile, and I am excited to come out of this internship with a network of lifelong friends and mentors. I could not have asked for a better way to spend my summer! With that being said, thank you for joining me today in my day in the life as an infrastructure intern. I hope reading this has given you a better insight into what it is like to be a Snootern at Reddit, and if you’re considering joining as an intern, I hope you’re convinced!
19:51 UTC
How Reddit Uses Signals-Joiner in Its Real-Time Safety Systems
Written by Vignesh Raja and Jerry Chu
Background and Motivation
Acting on policy-violating content as quickly as possible is a top priority of Reddit’s Safety team and is accomplished through technologies such as Rule-Executor-V2 (REV2), a real-time rules-engine that processes streams of events flowing through Reddit.
While a low time-to-process latency, measured as the time it takes for some activity on the site to flow through REV2, is an important metric to optimize for, it is equally important for REV2 to be able to identify more sophisticated policy-violating content. Concretely, in the context of Trust and Safety, our real-time actioning needs to balance two important factors:
- Latency: the time it takes for some activity on the site to flow through REV2
- Coverage: the breadth of policy-violating content detected by REV2
How do we balance these two important factors, latency and coverage, in our real-time actioning? One way is by ~performing preliminary enrichment~ to ensure that a plethora of contextual information about each piece of content flowing through Reddit is available within REV2. This method is effective, but has a low enrichment rate when enriching more complex signals that aren’t immediately available at the time of REV2’s processing.
For example, one common scenario at Reddit is that a Machine Learning (ML) system generating signals for a piece of content runs independently of REV2. If REV2 wanted to access these ML signals, a standard approach would be to store the signals in a database (DB) that REV2 could then query. However, ML inferencing typically carries a much higher latency compared to executing a rule within REV2. As a result, we would often observe a ~race-condition~ where for a piece of content, REV2 would attempt to query a DB storing the signal, but would find it not available yet.
# Race-condition encountered when REV2 consumes a signal written by an ML Service
To improve the availability of more complex signals in REV2 while maintaining its real-time nature, we developed Signals-Joiner to enrich the events that REV2 processes.
Signals-Joiner
Now that we’ve discussed the motivation for Signals-Joiner, let’s dive into its architecture in more detail. Signals-Joiner is a stream processing application written in Java that runs on ~Apache Flink~ and performs stream joins on various signal streams that live in Kafka.
What are Stream Joins?
You may be wondering what exactly a stream join is, so here’s a quick primer before getting into the weeds. We can think of a stream join as similar to a regular SQL join. However, the key distinction is that SQL joins are performed on finite datasets while stream joins are performed on infinite and continuously changing data streams.
How can we perform a join on an infinite data stream? The solution here is to break down the stream into smaller windows of time within which data is joined by a specified key. A finite window of data is stored within the streaming application’s state (options include purely in-memory, on-disk, etc.) until the corresponding time window expires.
Many popular stream processing frameworks support stream joins these days and we use Flink to accomplish this at Reddit. ~Here~ is some useful Flink documentation illustrating windowing and stream joins in further detail.
High-Level Architecture
Below is a diagram depicting how Signals-Joiner fits into the Safety team’s real-time processing pipeline.
# High-level architecture of Signals-Joiner
In Kafka, we start with our preliminary enriched content (could be posts, comments, etc.) that is in JSON format. As mentioned earlier, the content at this point has been enriched with basic contextual information but lacks more complex signals. We also have other Kafka topics storing various ML signals in Protobuf format that are produced by independent ML services.
Signals-Joiner reads from the base, Preliminary Stream and joins the various Signal Streams based on content ID, and finally outputs the fully enriched content to a separate topic that REV2 reads from. Effectively, the fully enriched JSON, now containing the complex signals, is a superset of the preliminary enriched JSON flowing into Signals-Joiner.
As a result of waiting some extra time for the availability of all input signals being joined, the fully enriched topic has some delay. For this reason, REV2 continues to read directly from the Preliminary Stream in addition to reading from the new, Fully Enriched StreamAs a result of waiting some extra time for the availability of all input signals being joined, the fully enriched topic has some delay. For this reason, REV2 continues to read directly from the Preliminary Stream in addition to reading from the new, Fully Enriched Stream. If a high confidence decision can be made based on just the preliminary enrichment, we want to do so to minimize REV2’s time-to-action latency.
Flink Topology
Signals-Joiner is powered by Flink which provides stateful stream processing and a ~Datastream API~ with a suite of operators. Below is an illustration of Signals-Joiner’s Flink topology. Note that in the diagram, only two signals (Signals 1 and 2) are joined for conciseness.
If a high confidence decision can be made based on just the preliminary enrichment, we want to do so to minimize REV2’s time-to-action latency.
# Signals-Joiner’s Flink topology
Starting with our preliminary enriched content, we chain left joins (via the ~CoGroup operator~) with some additional signals to build up a final, fully enriched output.
Windowing Strategy
Flink offers many ~windowing strategies~ and Signals-Joiner uses an ~event time~ based ~Tumbling Window~. At a high-level, Tumbling Windows assign incoming events to fixed, non-overlapping time windows. We experimented with some other strategies like Sliding Windows, Session Windows, and Interval Joins, but found that Tumbling Windows performed well empirically based on our join-rate metric (defined as # events containing a signal / # events that should have a signal).
Starting with our preliminary enriched content, we chain left joins (via the ~CoGroup operator~) with some additional signals to build up a final, fully enriched output.
Handling Unavailable Signals
You may be wondering what happens if an upstream service goes down and as a result, one of the signals we are attempting to join is unavailable. We’ve taken a few measures to mitigate this scenario.
First, we use the Preliminary Stream as the left stream for our left joins so that if any signal is unavailable, Signals-Joiner continues to emit unenriched messages after the join window expires. You can think of the Fully Enriched Stream as being a delayed equivalent to the Preliminary Stream in the case that all signals are unavailable.
Second, we leverage a ~Flink configuration~ to specify the allowed idleness of a stream. This way, even if one of the signal streams is idle for a certain period of time during an outage, we continue to advance ~watermarks~ which allows Flink to close windows.
Deployment
At Reddit, our Flink applications are deployed to Kubernetes (K8s) using the ~Flink K8s Operator~. The operator is great for simplifying Flink deployment configurations like ~High Availability (HA)~ mode, ~Checkpointing / Savepointing~, job upgrade strategies, and the Flink version.
Evaluation
In a streaming application like Signals-Joiner, small configuration changes can significantly impact performance. As such, we implemented comprehensive testing and monitoring for the system.
We make use of the ~MiniClusterWithClientResource~ JUnit rule to perform testing of windowing and joins against a local, lightweight Flink mini-cluster. Additionally, we have a set of ~smoke-tests~ that are triggered whenever a pull-request is created. These smoke-tests spin up Flink and Kafka clusters in a test K8s environment, write events to Kafka topics, and verify that the system achieves an acceptable join-rate.
The join-rate metric is monitored closely in production to prevent regressions. Additionally, we closely monitor Kafka consumer lag as a good indicator of application latency.
Future Work
Signals-Joiner has done well to enrich REV2 input data with complex signals, but as always, there is room for improvement. Primarily, we’d like to expand the suite of signals and breadth of input content that Signals-Joiner enriches. Additionally, we’d like to continue tuning our Flink windowing strategy in order to optimize join-rates.
Conclusion
Within Safety, we’re excited to continue building great products to improve the quality of Reddit’s communities. If ensuring the safety of users on one of the most popular websites in the US excites you, please check out our ~careers page~ for a list of open positions.
Thanks for reading!
17:47 UTC
Modular YAML Configuration for CI
Written by Lakshya Kapoor.
Background
Reddit’s iOS and Android app repos use YAML as the configuration language for their CI systems. Both repos have historically had a single .yml file to store the configuration for hundreds of workflows/jobs and steps. As of this writing, iOS has close to 4.5K lines and Android has close to 7K lines of configuration code.
Dealing with these files can quickly become a pain point as more teams and engineers start contributing to the CI tooling. Overtime, we found that:
- It was cumbersome to scroll through, parse, and search through these seemingly endless files.
- Discoverability of existing steps and workflows was poor, and we’d often end up with duplicated steps. Moreover, we did not deduplicate often, so the file length kept growing.
- Simple changes required code reviews from multiple owners (teams) who didn’t even own the area of configuration being touched.
- This meant potentially slow mean time to merge
- Contributed to notification fatigue
- On the flip side, it was easy to accidentally introduce breaking changes without getting a thorough review from truly relevant codeowners.
- This would sometimes result in an incident for on-call(s) as our main development branch would be broken.
- Difficult to determine which specific team(s) own which part of the CI configuration
- Resolving merge conflicts during major refactors was a painful process.
Overall, the developer experience of working in these single, extremely long files was poor, to say the least.
Introducing Modular YAML Configuration
CI systems typically expect a single configuration file at build time. However, they don’t need to be singular in the codebase. We realized that we could modularize the YML file based on purpose/domain or ownership in the repo, and stitch them together into a final, single config file locally before committing. The benefits of doing this were immediately clear to us:
- Much shorter YML files to work with
- Improved discoverability of workflows and shared steps
- Faster code reviews and less noise for other teams
- Clear ownership based on file name and/or codeowners file
- More thorough code reviews from specific codeowners
- Historical changes can be tracked at a granular level
Approaches
We narrowed down the modularization implementation to two possible approaches:
- Ownership based: Each team could have a
.ymlfile with the configuration they own. - Domain/Purpose based: Configuration files are modularized by a common attribute or function the configurations inside serve.
We decided on the domain/purpose based approach because it is immune to organizational changes in team structure or names, and it is easier to remember and look up the config file names when you know which area of the config you want to make a change in. Want to update a build config? Look up build.yml in your editor instead of trying to remember what the name for the build team is.
Here’s what our iOS config structure looks like following the domain-based approach:
.ci_configs/
├── base.yml# 17 lines
├── build.yml # 619
├── data-export.yml # 403
├── i18n.yml # 134
├── notification.yml # 242
├── release.yml # 419
├── test-post-merge.yml # 280
├── test-pre-merge.yml # 1275
└── test-scheduled.yml # 1016base.yml as the name suggests, contains base configurations, like the config format version, project metadata, system-wide environment variables, etc. The rest of the files contain workflows and steps grouped by a common purpose like building the app, running tests, sending notifications to GitHub or Slack, releasing the app, etc. We have a lot of testing related configs, so they are further segmented by execution sequence to improve discoverability.
Lastly, we recommend the following:
- Any new YML files should be named broad/generic enough, but also limited to a single domain/purpose. This means shared steps can be placed in appropriately named files so they are easily discoverable and avoid duplication as much as possible. Example:
notifications.ymlas opposed toslack.yml. - Adding multiline bash commands directly in the YML file is strongly discouraged. It unnecessarily makes the config file verbose. Instead, place them in a Bash script under a tools or scripts folder (ex:
scripts/build/download_build_cache.sh) and then call them from the script invocation step. We enforce this using a custom ~Danger~ bot rule in CI.
File Structure
Here’s an example modular config file:
# file: data-export.yml
# description: Data export (S3, BigQuery, metrics, etc.) related workflows and steps.
workflows:
#
# -- SECTION: MAIN WORKFLOWS --
#
Export_Metrics:
before_steps:
- _checkout_repo
- _setup_bq_creds
steps:
- _calculate_nightly_metrics
_ _upload_metrics_to_bq
- _send_slack_notification
#
# -- SECTION: UTILITY / HELPER WORKFLOWS --
#
_calculate_nightly_metrics:
steps:
- script:
title: Calculate Nightly Metrics
inputs:
- content: scripts/metrics/calculate_nightly.sh
_ _upload_metrics_to_bq:
steps:
- script:
title: Upload Metrics to BigQuery
inputs:
- content: scripts/data_export/upload_to_bq.sh <file>Stitching N to 1
Flow
$ make gen-ci -> yamlfmt -> stitch_ci_config.py -> ./ci_configs/generated.yml -> validation_util ./ci-configs/generated.yml -> DoneThis command does the following things:
- Formats
./ci_configs/*.ymlusing ~yamlfmt~ - Invokes a Python script to stitch the YML files
- Orders
base.ymlin first position, lines up rest as is - Appends value of workflows keys from rest of YML files
- Outputs a single
.ci_configs/generated.yml
- Orders
- Validates generated config matches the expected schema (i.e. can be parsed by the build agent)
- Done
- Prints a success or helpful failure message if validation fails
- Prints a reminder to commit any modified (i.e. formatted by
yamlfmt) files
Local Stitching
The initial rollout happened with local stitching. An engineer had to run the make gen-ci command to stitch and generate the final, singular YAML config file, and then push up to their branch. This got the job done initially, but we found ourselves constantly having to resolve merge conflicts in the lengthy generated file.
Server-side Stitching
We quickly pivoted to stitching these together at build time on the CI build machine or container itself. The CI machine would check out the repo and the very next thing it would do is to run the make gen-ci command to generate the singular YAML config file. We then instruct the build agent to use the generated file for the rest of the execution.
Linting
One thing to be cautious about in the server-side approach is that invalid changes could get pushed. This would cause CI to not start the main workflow, which is typically responsible for emitting build status notifications, and as a result not notify the PR author of the failure (i.e. build didn’t even start). To prevent this, we advise engineers to run the make gen-ci command locally or add a Git pre-commit hook to auto-format the YML files, and perform schema validation when any YML files in ./ci_configs are touched. This helps keep the YML files consistently formatted and provide early feedback on breaking changes.
Note: We disable formatting and linting during the server-side generation process to speed it up.
$ LOG_LEVEL=debug make gen-ci
✅ yamlfmt lint passed: .ci_configs/*.yml
2024-08-02 10:37:00 -0700 config-gen INFO Running CI Config Generator...
2024-08-02 10:37:00 -0700 config-gen INFO home: .ci_configs/
2024-08-02 10:37:00 -0700 config-gen INFO base_yml: .ci_configs/base.yml
2024-08-02 10:37:00 -0700 config-gen INFO output: .ci_configs/generated.yml
2024-08-02 10:41:09 -0700 config-gen DEBUG merged .ci_configs/base.yml
2024-08-02 10:41:09 -0700 config-gen DEBUG merged .ci_configs/release.yml
2024-08-02 10:41:09 -0700 config-gen DEBUG merged .ci_configs/notification.yml
2024-08-02 10:41:09 -0700 config-gen DEBUG merged .ci_configs/i18n.yml
2024-08-02 10:41:09 -0700 config-gen DEBUG merged .ci_configs/test-post-merge.yml
2024-08-02 10:41:10 -0700 config-gen DEBUG merged .ci_configs/test-scheduled.yml
2024-08-02 10:41:10 -0700 config-gen DEBUG merged .ci_configs/data-export.yml
2024-08-02 10:41:10 -0700 config-gen DEBUG merged .ci_configs/test-pre-merge.yml
2024-08-02 10:41:10 -0700 config-gen DEBUG merged .ci_configs/build.yml
2024-08-02 10:41:10 -0700 config-gen DEBUG merged .ci_configs/test-mr-merge.yml
2024-08-02 10:37:00 -0700 config-gen INFO validating '.ci_configs/generated.yml'...
2024-08-02 10:37:00 -0700 config-gen INFO ✅ done: '.ci_configs/generated.yml' was successfully generated.Output from a successful generation in local.
Takeaways
- If you’re annoyed with managing your sprawling CI configuration file, break it down into smaller chunks to maintain your sanity.
- Make it work for the human first, and then wrangle them together for the machine later.
18:50 UTC
Bringing Learning to Rank to Reddit Search - Operating with Filter Queries
Written by Chris Fournier.
In earlier posts, we shared how Reddit's search relevance team has been working to bring Learning to Rank - ML for search relevance ranking - to optimize Reddit’s post search. Those posts covered our Goals and Training Data and Feature Engineering. In this post, we go into some infrastructure concerns.
When starting to run the Learning to Rank (LTR) plugin to perform reranking in Solr, we ran into some cluster stability issues at low levels of load. This details one bit of performance tuning performed to run LTR at scale.
Background
Reddit operates Solr clusters that receive hundreds to thousands of queries per second and indexes new documents in near-real time. Solr is a Java-based search engine that – especially when serving near-real time indexing and query traffic – needs its Java Virtual Machine (JVM) garbage collection (GC) tuned well to perform. We had recently upgraded from running Solr 7 on AWS VMs to running Solr 9 on Kubernetes to modernize our clusters and began experiencing stability issues as a result. These upgrades required us to make a few configuration changes to the GC to get Solr to run smoothly. Specifically, using the G1 GC algorithm, we prevented the Old Generation from growing too large and starving the JVM’s ability to create many short-lived objects. Those changes fixed stability for most of our clusters, but unfortunately did not address a stability issue specific to our cluster serving re-ranking traffic. This issue appeared to be specific to our LTR cluster, so we dove in further.
Investigation
On our non-re-ranking Solr clusters, when we increased traffic on them slowly, we would see some stress that was indicated by slightly increased GC pause times, frequency, and slightly higher query latencies. In spite of the stress, Solr nodes would stay online, follower nodes would stay up-to-date with their leaders, and the cluster would be generally reliable.
However, on our re-ranking cluster, every time we started to ramp up traffic on the cluster, it would invariably enter a death spiral where:
- GC pause times would increase rapidly to a point where they were too long, causing:
- Solr follower nodes to be too far behind their leaders so they started replication (adding more GC load), during which:
- GC times would increase even further, and we’d repeat the cycle until individual nodes and then whole shards were down and manual intervention was required to get the nodes back online.
Such a death-spiral example is shown below. Traffic (request by method) and GC performance (GC seconds per host) reaches a point where nodes (replicas) start to go into either a down or recovery state until manual intervention (load shedding) is performed to right the cluster state.
Zooming in, this effect was even visible at small increases in traffic, e.g. from 5% to 10% of total; garbage collection jumps up and continues to rise until we reach an unsustainable GC throughput and Solr nodes go into recovery/down states (shown below).
It looked like we had issues with GC throughput. We wanted to fix this quickly so we tried vertically and horizontally scaling to no avail. We then looked at other performance optimizations that could increase GC throughput.
Critically, we asked the most basic performance optimization question: can we do less work? Or put another way, can we put less load on garbage collection? We dove into what was different about this cluster: re-ranking. What do our LTR features look like? We know this cluster runs well with re-ranking turned off. Are some of our re-ranking features too expensive?
Something that we began to be suspicious of was the effects of re-ranking on filter cache usage. When we increased re-ranking traffic, we saw the amount of items in the filter cache triple in size (note that the eviction metric was not being collected correctly at the time) and warm up time jumped. Were we inserting a lot of filtered queries to the filter cache? Why the 3x jump with 2x traffic?
To understand the filter cache usage, we dove into the LTR plugin’s usage and code. When re-ranking a query, we will issue queries for each of the features that we have defined our model to use. In our case, there were 46 Solr queries, 6 of which were filter queries like the one below. All were fairly simple.
{
"name": "title_match_all_terms",
"store": "LTR_TRAINING",
"class": "org.apache.solr.ltr.feature.SolrFeature",
"params":
{
"fq":
[
"{!edismax qf=title mm=100% v=\"${keywords}\"}"
]
}
},We had assumed these filter queries should not have been cached, because they should not be executed in the same way in the plugin as normal queries are. Our mental model of the filter cache corresponded to the “fq” running during normal query execution before reranking. When looking at the code, however, the plugin makes a call to getDocSet when filter queries are run.
getDocSethas a Javadoc description that reads:
"Returns the set of document ids matching all queries. This method is cache-aware and attempts to retrieve the answer from the cache if possible. If the answer was not cached, it may have been inserted into the cache as a result of this call*. …"
So for every query, we re-rank and make 6 filtered queries which may be inserting 6 cache entries into the filter cache scoped to the document set. Note that the filter above depends on the query string (${keywords}) which combined with being scoped to the document set results in unfriendly cache behavior. They’ll constantly be filling and evicting the cache!
Solution
Adding and evicting a lot of items in the filter cache could be causing GC pressure. So could simply issuing 46 queries per re-ranking. Or using any filter queries in re-ranking. Any of those could have been issues. To test which was the culprit, we devised an experiment where we would try 10% traffic with each of the following configurations:
- LTR: Re-ranking with all features (known to cause high GC)
- Off: No reranking
- NoFQ: Re-ranking without filter query features
- NoCache: Re-ranking but with filter query features and a no-cache directive
The NoCache traffic had its features re-written as shown below to include cache=false:
{
"name": "title_match_all_terms",
"store": "LTR_TRAINING",
"class": "org.apache.solr.ltr.feature.SolrFeature",
"params":
{
"fq":
[
"{!edismax cache=false qf=title mm=100% v=\"${keywords}\"}"
]
}
},We then observed how GC load changed as the load was varied between these four different configurations (shown below). Just increasing re-ranking traffic from 5% to 10% (LTR) we observed high GC times that were slowly increasing over time resulting in the familiar death spiral. After turning off re-ranking (Off) GC times plummeted to low levels.
There was a short increase in GC time when we changed collection configs (Changed configs) to alter the re-ranking features, and then when we started re-ranking again without the filter query features, GC rose again, but not as high, and was stable (not slowly increasing over time). We thought we had found our culprit, the additional filter queries in our LTR model features. But, we still wanted to use those features, so we tried enabling them again but in the query indicating that they should not cache (NoCache). There was no significant change in GC time observed. We were then confident that it was specifically the caching of filter queries from the re-ranking that was putting pressure on our GC.
Looking at our items in the filter cache and warm up time we could also see that NoCache had a significant effect; item count and warm up time were low, indicating that we were putting fewer items into the filter cache (shown below).
During this time we maintained a relatively constant p99 latency except for periods of instability during high GC with the LTR configuration and when configs were changed (Changed configs) with a slight dip in latency between starting Off (no re-ranking) and NoFQ (starting re-ranking again) because we were doing less work overall.
With these results in hand we were confident to start adding more load onto the cluster using our LTR re-ranking features configured to not cache filtered queries. Our GC times stayed low enough to prevent the previously observed death spirals and we finally had a more reliable cluster that could continue to scale.
Takeaways
After this investigation we were reminded/learned that:
- For near-real time query/indexing in Solr, GC performance (throughput and latency) is important for stability
- When optimizing performance, look at what work you can avoid doing
- For the Learning to Rank plugin, or other online machine learning, look at the cost of the features being computed and their potential effects on immediate (e.g. filter cache) or transitive (e.g. JVM GC) dependencies.
21:01 UTC
A Day in the Life of a Reddit SWE Intern in NYC
Written by Alex Soong (u/besideagardenwall)
Introduction
It may be surprising to some - including myself - that an intern could be given any company platform to talk on. Luckily, this summer, I’ve had the opportunity to work at Reddit as a Software Engineering Intern. Our mission here is to bring community and belonging to everyone in the world and thus, I’ve truly been treated like an equal human being here - no corralling coffees.
Perhaps you’re here because you’re genuinely interested in what I work on. Perhaps you’re a prospective Reddit intern, scrolling through this sub to imagine yourself here, just as I did. Or perhaps you’re my manager, making sure I’m actually doing work. Regardless, this is ~r/RedditEng~’s first exposure to the Reddit internship ever so I hope I do it justice.
The Morning
I work out of Reddit’s NYC office. We got to choose between working in NYC, SF, or remotely. I’m living in the Financial District (FiDi) this summer so I have the luxury of taking a brief 10 minute walk to the office. We’re allowed to work from home, but many other interns and I elect to go in for a monitor, free food, socialization, and powerful AC - a must in the brutal NYC summer. When I get into the office, I make a beeline for the kitchen and grab a cold brew. I normally hop onto Notion and plan out what I want to accomplish that day. It’s also imperative to my work that I have music playing throughout the day. Recently, I’ve had The Beach Boys and Laufey on repeat, with berlioz for focus sessions. This morning, we were treated with catering from Playa Bowls for breakfast, which I got to enjoy while diving into our codebase.
A beautiful array of Playa bowls.
I am on the Tech PMO Solutions team. Our primary product is Mission Control. It’s Reddit’s internal tool which tracks virtually every initiative across the company, from product launches to goals to programs. Mission Control has been built entirely in-house, curated to fit Reddit’s exact needs. Our team is small but mighty. At Reddit, interns are assigned a manager and mentor. Staying in touch with my mentor and manager has helped me connect to my team, despite the fact that we’re working all across the country.
Since the rest of my team works remotely, I get to sit with my fellow interns. Or rather, Snooterns - a portmanteau of Snoo, Reddit’s alien mascot, and interns. We sit in Snootern Village and are by far the most rambunctious section of the NYC office. My apologies to the full-time employees who work near us. Come by at any point of the day and you’ll see us coding away, admiring the view of Manhattan from the windows, or eating snacks from the everflowing kitchen.
Snooterns hard at work in Snootern Village, as per usual.
Noon and After
In the NYC office, we’re very lucky to get free lunch Monday through Thursday. The cuisine varies every day but my favorites have been barbeque and Korean food. On Fridays, Smorgasburg - a large gathering of assorted food stalls - happens right outside our doors next to the Oculus, which is a fun little break from work.
After lunch, I’m getting back into the code. This summer, I’ve been programming in Python and Typescript, with which I’ve gained experience in full-stack website development. My team sets itself apart from others in the company as we function more as a small startup within Reddit, building Mission Control from the ground up, as opposed to a traditional team. There are always new features to improve MC’s capabilities or our users’ (fellow Snoos/Reddit employees) experiences, ultimately optimizing how Reddit is accomplishing its goals. This summer, my schedule is relatively light on meetings, which is much appreciated as I get many uninterrupted time blocks to focus.
My main internship project this summer has been to create data visualizations for metrics on how large initiatives are doing and implement them into Mission Control. There’s rhetorical power in seeing data rather than just reading it - some meaningful takeaways may only come to light when visualized. In theory, these graphs will help teams understand and optimize their progress. Most of my days are spent working on these visualizations and sometimes squashing random bugs, working from my desk or random spots in the office when I need a change of scenery.
Throughout the summer, I’ve had the opportunity to organically meet and chat with several Snoos in different roles across the company. I’ve found the culture at Reddit to be very welcoming and candid. There are plenty of opportunities to learn from people who have come before you. The Emerging Talent team also organizes different seminars and career development events throughout the weeks.
Finally, the clock strikes 5.
A Note-ably Eventful Evening
The Emerging Talent (ET) team plans several fun events for us Snooterns throughout the summer. Today, they took us to a VR experience at Tidal Force VR in the Flatiron District. There’s a relatively large intern cohort in NYC compared to SF and remote, so we played in smaller groups. This was my first time ever doing anything like this, and it was shocking how immersive it truly was. It was great bonding, even though my stats showed my biggest enemy in the game wasn’t the actual villain, rather, a fellow intern who kept shooting me… Post-VR, we all headed to wagamama across the street for dinner. Many kudos to the ET team for planning this event.
A wild pack of Snooterns looking especially fierce shooting at VR enemies.
After the official festivities, a subset of the interns went to Blue Note, one of the most notable jazz clubs in New York. Seeing jazz live is one of my great joys in life so I was excited to check this venue off my bucket list. It’s disorienting to realize that we were all strangers to one another so recently. These people have truly helped this summer fly by. With just a few more weeks left of the internship, I hope we get to make many more memories together - while concluding our projects, of course.
Snooterns happy after creative stimulation at Blue Note.
TL;DR
Choosing to intern at Reddit is one of the most fruitful decisions I’ve made in my life. I’ve gained so much technically and professionally, and made many invaluable connections along the way. To me, the timeboxed nature of an internship makes every moment - every approved pull request, shared meal, coffee chat, and even bugs - ever more valuable. My experience here has only been made possible by the Emerging Talent team and my team, Tech PMO Solutions, for bearing with all of my questions and investing in my growth.
My inspiration to write this blog post stemmed from searching high and low for interns’ experiences when I was deciding where to intern. Whatever your purpose is in reading this post, I hope it offers a clarifying perspective on what it’s like to intern at Reddit from behind the scenes.
21:09 UTC
Scaling Ads Pacing: from Singleton to Sharded
Written by David Yang & Yan Wang
Introduction
Welcome to our technical blog from the Ads Experimentation Platform team at Reddit. Our team plays a pivotal role in advancing the Ads Pacing Infrastructure and Marketplace Experimentation Platform.
- Ads Pacing Infrastructure: At the core of our mission is the development of scalable, reliable, and modular pacing infrastructures. These systems are designed to empower partner teams, enabling them to efficiently develop and evolve pacing control algorithms. By providing robust foundations, we aim to optimize ad delivery strategies, ensuring optimal performance and user experience.
- Marketplace Experimentation Platform: In parallel, our team is dedicated to enhancing the throughput, velocity, and quality of our experimentation capabilities across various Ads product areas, empowering teams to understand the impact of changes swiftly and confidently.
What is Ads Pacing?
In the ads marketplace, pacing refers to the strategic distribution of advertisements over a specified period to optimize performance and budget utilization. It involves managing the frequency and timing of ad placements to ensure they align with campaign objectives such as reaching target audiences effectively and economically without exhausting budget too quickly. Effective pacing aims to maintain a balanced delivery of ads throughout the campaign duration, preventing oversaturation or underperformance. You’ll often hear the term PID controller in related literature, which is not the main topic here but definitely worth its own topic for future.
Challenges in Pacing Systems
We can simplify the overall ad serving and pacing flow as a feedback loop shown as below:
For each ad campaign, the pacing system takes in three inputs: budget to spend, time in the life span, and past spendings, then calculates a set of signals, which control the rate of spending in ad serving (common controls are probability thresholding, bid modification).
In this feedback loop, the pacing system needs to react smartly and swiftly to meet the changing marketplace dynamics and continuous spendings from live campaigns:
- Smartly: the system needs to apply a sophisticated model on top of rich amounts of data from the past, e.g. a time series of per-minute clicks of last 12 hours, to derive well balanced signals per minute,
- Swiftly: the system needs to both read the data and calculate the model in a fast way, we adopt the mandate to ensure that all campaigns’ signals are recalculated at least once per minute, which translates to a cap of 1 min on the read-compute time over all campaigns,
With the number of ad campaigns growing drastically over the last couple of years and more complex controllers being introduced, both data size and computation cost grew drastically, and triggered our decision to re-architect the system for higher scalability and durability.
Design and Architecture
The old pacing system was built on the Spark batch processing architecture (diagram above). The driver is conducting all pacing calculations, the executors are mainly used for fetching and aggregating data in batches from various data stores:
- Campaign management database: a Cassandra table that stores all campaign data,
- Unverified Tracking Events: a Cassandra table that stores realtime unverified ad tracking events for providing fast-loop spendings data,
- Verified Tracking Events: an S3 bucket that stores hourly pipeline verified ad tracking events for providing the truthful spendings data,
The pacing job periodically loads in all live campaign data and fetches up-to-date spendings from both tracking events sources, calculates the pacing signals for all live campaigns, and sends the pacing signals to each ad server pod through Thrift API calls.
Why two sources of tracking events? The Verified Tracking Events data provides the ultimate truth. However, it goes through an hourly delayed verification pipeline. To mitigate the gap between now and the last available hour of verified data, we fill in with the spendings from real-time Unverified Tracking Events (aka bots/duplication unfiltered) for swift pacing control.
This singleton architecture ran into its bottleneck with more campaigns onboarding Reddit:
- The driver pod memory and cpu usages creeped up to a level where further scaling up a single pod became impossible,
- The pacing runtime surpassed the 1 min cap as it needs to process more campaigns all at once, due to the batch processing.
To address the above issues, we need to partition the inputs and shard the system (see below).
We spent one and a half years gradually turning the old system from a singleton spark job into a sharded system with partitioned streamed inputs (diagram above, the diff parts are in green).
In the new architecture,
- The campaign budget input is turned into a budget update stream on Kafka, which is partitioned on the advertiser id. The campaign budget update stream is published by a new Budgeting System, which hosts the budgeting logic extracted from the old job,
- All tracking events sources are turned into keyed data stores: Redis for unverified tracking events, Druid for the verified source, see this
presentationfrom our colleagueNagalakshmi Ramasubramanianfor details, - The pacing job is refactored into a scala
statefulset apprunning in a K8S cluster, where each shard only consumes a subset of partitions of the campaign budget updates, - Instead of fetching data in batches, each shard now reads in the spendings from both tracking events data sources by campaign ids,
- Instead of pacing all campaigns in one driver, each shard now only paces the campaigns under the partition of advertisers it consumes,
- Instead of calling each ad server pod directly by the pacer, the pacer broadcasts the pacing signals to a Kafka stream from which the ad servers consume the updates.
How does a shard know what partitions to consume? Instead of relying on Kafka for dynamic partition assignments (aka using a consumer group), we adopt a stable mapping between shards and the budget update topic partitions through range sharding:
- The sharded pacing system runs as a statefulset job with multiple stateful pods,
- Each shard pod is assigned with a unique numeric ID (between 0 and #shards),
- The number of topic partitions is fixed at 64, which is enough for a foreseeable future,
- Each shard ID is mapped to a continuous range between 0 and 63, and the mapped ranges are mutually exclusive among different IDs,
- Each shard only consumes the campaign budget updates from its mapped partitions,
- As campaign budget updates are partitioned on advertiser id, it’s guaranteed that no two shards consume the same campaign budget.
What is the budgeting system? Budgeting decides the daily budget for each campaign. Previously, its logic was embedded in the singleton job as a prerequisite step to pacing. Under the new architecture, we extracted the budgeting logic out of the pacing system into a dedicated system that runs independently and asynchronously. The budgeting system publishes the campaign budget updates to a Kafka stream and partitions the updates on the advertiser_id (an advertiser can own multiple campaigns). In this way, the campaign budget data source became naturally partitioned through Kafka for the downstream pacing system to consume.
Gains from New Architecture
We ran the sharded pacing system alongside the singleton job on the same set of campaigns for 4 weeks’ comparisons. The sharded system demonstrated a linear scalability boost on our business scale at the time, aka 1/n pacing runtime with n shards, shown as the graph below.
Path towards the New Design
The pacing system is a busy area where multiple teams actively work on it at any given time. Although the pacing system became drastically different from its singleton version, we did the refactoring and migration in a smooth and non-interrupting way, so our partner teams kept their pace on developing new pacing controllers without noticing much from the architecture change.
We first changed all data sources and their client fetching logic into sharding friendly solutions,
|| || |Component|Old (backend + client)|New (backend + client)| |Campaign management DB|Cassandra + Spark|PostgreSql + thrift API| |Unverified tracking events|Cassandra + Spark|Redis + Jedis| |Verified tracking events|S3 + Spark|Druid + Rest API| |Checkpoints|S3 + Spark|Redis + Jedis|
Then we extracted the budgeting logic out of the pacing job into a dedicated system and refactored the input of campaign budget updates into a partitioned Kafka stream.
After the above two steps, the pacing job (still in spark) was effectively transformed into a single pacing shard (aka the driver pod) that consumes and paces all campaign budgets, without any significant change to the core controller logic.
Lastly, we turned the pacing spark job (in Scala) into a statefulset application (in Scala), by setting up a new deployment pipeline and introducing the range sharding in the consumer initialization code for partitions assignment.
Future Development
In order to partition the campaign budget data, we introduced a new standalone system for budgeting and publishing the updates to Kafka, which is a lightweight and low-frequency job. The budgeting system was initially built as a singleton job.
With the ad business growing fast, the budgeting system is now facing similar challenges to pacing, therefore we are planning to partition the budgeting system in the coming quarters.
We are also planning to introduce event-based budget updates on advertiser made changes, which will provide a more reactive experience to the advertisers.
16:26 UTC
Decomposing the Analytics Monoschema!
Written by Will Pruyn.
Hello! My name is Will Pruyn and I’m an engineer on Reddit’s Data Ingestion Team. The Data Ingestion team is responsible for making sure that Analytics Events are ingested and moved around reliably and efficiently at scale. Analytics Events are chunks of data that describe a unique occurrence on Reddit. Think any time someone clicks on a post or looks at a page, we collect some metadata about this and make it available for the rest of Reddit to use. We currently manage a suite of applications that enable Reddit to collect over 150 billion behavioral events every day.
Over the course of Reddit’s history, this system has seen many evolutions. In this blog, we will discuss one such evolution that moved the system from a single monolithic schema template to a set of discrete schemas that more accurately model the data that we collect. This move allowed us to greatly increase our data quality, define clear ownership for each event, and protect data consumers from garbage data.
A Stitch in Time Saves Nine
Within our Data Ingestion system, we had a monolithic schema template that caused a lot of headaches for producers, processors, and consumers of Analytics Events. All of our event data was stored in a single BigQuery table, which made interacting with it or even knowing that certain data existed very difficult. We had very long detection cycles for problems and no way to notify the correct people when a problem occurred, which was a terrible experience. Consumers of this data were left to wade through over 2,400 columns, with no idea which were being populated. To put it simply it was a big ball of mud that needed to be cleaned up.
We decided that we could no longer maintain this status quo and needed to do something before it totally blew up in our faces. Reddit was growing as a company and this simply wouldn’t scale. We chose to evolve our system to enable discrete schemas to describe all of the different events across Reddit. Our previous monolithic schema was represented using Thrift and we chose to represent our new discrete schemas using Protobuf. We made this decision because Reddit as a whole was shifting to gRPC and Protobuf would allow us to more easily integrate with this ecosystem. For more information on our shift to gRPC, check out this excellent r/redditeng blog!
Evolving in Place
To successfully transition away from a single monolithic schema, we knew we had to evolve our system in a way that would allow us to enforce our new schemas, without necessitating code changes for our upstream or downstream customers. This would allow us to immediately benefit from the added data quality, clear ownership, and discoverability that discrete schemas provide.
To accomplish this, we started by creating a single repository to house all of the Protobuf schemas that represent each type of occurrence. This new repository segmented events by functional area and provided us a host of benefits:
- It gave us a single place to easily consume every schema.
- It allowed us to assign ownership to groups of events, which greatly improved our ability to triage problems when event errors occur.
- Having the schemas in a single place also allowed our team to easily be in the loop and apply consistent standards during schema reviews.
Once we had a place to put the schemas, we developed a new component in our system whose job it was to ensure that events conformed to both the monolithic schema and the associated discrete schema. To make this work, we ensured that all of our discrete schemas followed the same structure as our monolithic schema, but with less fields. We then applied a second check to each event, that ensured the event conformed to the discrete schema associated with it. This allowed us to transparently apply tighter schema checks without requiring all of our systems that emitted events to change a thing! We also added functionality to allow different actions to be taken when a schema failure occurred, which let us monitor the impact of enforcing our schemas without risking any data loss.
Next, we updated our ingestion services to accept the new schema format. We wrote new endpoints to enable ingestion via Protobuf, giving us a path forward to eventually update all of the systems emitting events to send them using their discrete schemas.
Finding Needles in the Haystack
In order to move to discrete schemas, we first had to get a handle on what exactly was flowing through our pipes. Our initial analysis yielded some shocking results. We had over 1 million different event types. That can’t be right… This made it apparent that we were receiving a lot of garbage and it was time to take out the trash.
Our first step to clean up this mess was to write a script that applied a set of rules to our existing types to filter out all of the garbage values. Most of these garbage values were the result of random bytes being tacked onto the field that specified what type an event was in our system, an unfortunately common bug. This got us down to around ~9,000 unique types. We also noticed that a lot of these types were populating the exact same data, for the exact same business purpose. Using this, we were able to get the number of unique types down to around ~3,400.
Once we had whittled down the number of schemas, we began an effort to determine what functional area each one belonged to. We did a lot of “archeology”, digging through old commit histories and jira tickets to figure out what functional area made sense for the event in question. After we had established a solid baseline, we made a big spreadsheet and started shopping around to teams across Reddit to figure out who cared about what. One of the awesome things about working at Reddit is that everyone is always willing to help (did I mention we’re hiring 😉) and using this strategy, we were able to assign ownership to 98% of event types!
Automating Creation of Schemas
After we got a handle on what was out there, it was clear that we would need to automate the creation of the 3,400 Protobuf schemas for our events. We wrote a script that was able to efficiently dig through our massive events table, figure out what values had been populated in practice, and produce a Protobuf schema that matched. The script did this with a gnarly SQL query that did the following:
- Convert every row to its JSON representation.
- Apply a series of regular expressions to each row to ensure key/value pairs could be pulled out cleanly and no sensitive data went over the wire.
- Filter out keys with null values.
- Group by key name.
- Return counts of which keys had been populated.
With this script, we were able to fully populate our schema repository in less than a business day. We then began monitoring these schemas for inaccuracies in production. This process lasted around 3 months as we worked with teams across Reddit to correct anything wrong with their schemas. Once we had a reasonable level of confidence that enforcing the schemas would not cause data loss, we turned on enforcement across the board and began rejecting events that were not related to a discrete schema.
Results
At the end of this effort, we finally have a definitive source of truth for what events are flowing through our system, their shape, and who owns them. We stopped ingesting garbage data and made the system more opinionated about the data that it accepts. We were able to go from 1 million unique types with a single schema to ~3,400 discrete types with less than 50 fields a piece. We were also able to narrow down ownership of these events to ~50 well-defined functional areas across Reddit.
Future Plans
This effort laid the foundation for a plethora of projects within the Data Ingestion space to build on top of. We have started migrating the emission of all events to use these new discrete schemas and will continue this effort this year. This will enable us to break down our raw storage layer, enhance data discoverability, and maintain a high level of data quality across the systems that emit events!
If you’re interested in this type of work, check out our careers page!
16:42 UTC
Happy Holiday week!
r/redditeng is taking a little break to celebrate the two holidays this week, Canada Day and Independence Day. We'll be back next week but, for now, we'll pay for our absence with Cat Tax. Meet Sam and Daniel.
13:05 UTC
Enriching Data for Reddit Safety’s Rules Engine in Real Time
Written by: Stephan Weinwurm, Bhavani Balasubramanyam, and Jerry Chu.
Background
With the mission of keeping the platform safe and welcoming, Reddit’s Safety org is committed to detecting and acting on policy-violating content in real time. In September 2023, the Safety Signals team published a blog introducing our real-time site-wide rules engine (REV2) to curb policy-violating content. This blog describes our follow-up efforts in data enrichment, which feeds necessary contextual information to the REV2 rules engine to further increase its efficacy.
To conduct site-wide Safety moderations, REV2 consists of many different rules that listen to various Kaka topics (e.g. creations and editions of posts, comments and subreddits etc). To decide whether to action a piece of content, REV2 needs to gather comprehensive contextual information, such as which user account created the content, in which subreddit the content was posted, etc. This information needs to be enriched in near real-time so REV2 can act swiftly. Since the enriched context is shared across all rules that listen to the same type of content (e.g. posts), we aim to enrich it once upstream of the rules engine, instead of enriching multiple times for each rule separately.
After we modernized the rules engine in 2023, the enrichment logic was still running in Reddit’s Python monolith–a big heap of Spaghetti-code with limited test coverage. To continue our investment in modernizing Reddit’s tech infrastructure, we set out to migrate and modernize the enrichment logic into its own micro-service. This enabled significant performance improvements. For example, end-to-end enrichment latencies were reduced by 80-90% across all percentiles.
Taming the Spaghetti Monster
The main challenge of this migration is ensuring data fidelity. More specifically, all events flowing into the Rules Engine from the new micro-service are required to be fully backwards compatible with those produced by the monolith.
For each event we have to fetch contextual information for multiple layers. For example, a new post needs information such as title, body, upvotes and downvotes, etc. We also need extra information about the author as well as the subreddit that it was posted in. This was solved as a recursion resulting in a nested event structure. The enriched events are fairly large JSON blobs without any schema definition (up to 20MB uncompressed). While we did do some minor structural clean-ups and consistency fixes along the way, we were ultimately able to maintain the structure without any significant regression.
The second challenge arose from the fact that the retrieval of various contextual information in the old enrichment logic was implemented by accessing data stores (or interfaces) inside the monolith. To completely move away from the monolith, our new enrichment microservice integrated with APIs that had already been broken out of the monolith, and we also implemented a few new ones along the way. Now the microservice utilizes a total 30+ internal APIs to fetch the required contextual information.
Lastly, we also updated the microservice from Python 2 to 3 via Reddit’s internal Baseplate framework to simplify the migration and refactored the business logic to improve maintainability.
Backwards Compatibility
As mentioned in the previous section, our main challenge was to maintain full backwards compatibility, yet we didn’t have a schema to work against. We started to tackle it by deriving some approximate schemas from the existing events so we had at least a derived structure to compare to. After this step, we developed a deep understanding of the existing code by performing some code archeology. Over the course of several quarters, we ported over all parts and implemented adequate test coverage.
Testing in Production (aka when Software Engineering meets reality)
After standing up the deployment, we relied on tap-comparing shadow traffic in production because the new microservice didn’t complete any side-effects other than writing to Kafka topics. To partially automate the comparison, we wrote a script that sampled events produced by the new microservice, reset offsets on the Kafka topics produced by the monolith, and performed a deep comparison using dictdiffer. However, due to the clean-ups and consistency improvements mentioned above, the script initially surfaced differences that were expected, so we improved the script to ignore these changes. We achieved this by building a very basic JSON path-like notation along with applied transformations per path, such as renaming fields, changing the format of the field etc.
The script output is an overview of how many times a given difference has occurred. For example, if all of the 100 compared events miss a certain field, the script outputs 100 (remove) post/author/field_1indicating that field_1 was missing from all Author objects embedded in the Post object. The script helped us to quickly identify discrepancies so we could address them before moving onto the final stages.
Productionisation
During our initial shadow-traffic tests in production, we noticed that tail latencies were in the range of minutes, compared to the median of around 2-3 seconds. By digging deeper, we discovered that the main drivers were some deeply nested events where we had to enrich almost all context details.
We identified two main low-hanging fruits to curb tail latencies:
- Leveraging Gevent to enrich parts of the message concurrently or at least as much as possible in Python, given the Global Interpreter Lock. While this required some code refactoring, it yielded fairly good results while the business logic is mostly busy waiting for network responses. Gevent is able to leverage the network-IO wait times to perform other calls in the meantime.
- After diving into the operational metrics, we noticed a couple of places in code where we called dependencies with high frequency to enrich details such as subreddit names. Such data fields are fairly static, being a great candidate for simple caching strategy. We implemented in-process caching via cachetools which, after the warm-up time, reduced call volume to some dependencies by as much as 90%. As a future improvement, we may build a distributed cache to avoid having to warm up the cache as new K8s pods come online as part of scaling or deploying.
These improvements mitigated the tail latencies, and we were ready to support production traffic.
Shifting Traffic Between Monolith and Microservice
The majority of the hard work to ensure backward compatibility was done by addressing data discrepancies revealed by our script explained in the “Testing in Production” section above. With confidence in our eventing structure, we started to gradually shift traffic topic-by-topic from monolith to the new microservice for the final cut-over, and ensure that at any sign of problems we could revert back immediately with little impact.
We achieved this gradual rollout using Reddit’s internal experimentation framework where each content ID in the event would get sent to the experimentation library in the monolith to receive a mutually exclusive decision on which deployment should process the event. This guaranteed that only one of the two deployments would process the event and the other one would skip it.
This allowed us to increase the rollout slowly from 0.1% to 1% to 5% and so on, monitoring logs and dashboards for any impact.
Ultimately the rollout went smoothly, aside from minor bug fixes, we were able to move to 100% of events processed by the new microservice.
Currently, the microservice processes around 600 messages per second under normal traffic. P90 latency of data enrichment is under a second, significantly down from the previous batch-driven deployment in the monolith, allowing us to significantly shorten the cap for our site-wide rules engine to catch policy-violating content.
Future Plan
Currently all messages for enrichment arrive via RabbitMQ procured by some remaining code of the Reddit monolith, which has been set on the deprecation path. We are planning on consuming events from our main service event bus so we can further decouple from the monolith.
Within Safety, we’re excited to continue building great products to improve the quality of Reddit’s communities. If ensuring the safety of users on one of the most popular websites in the US excites you, please check out our careers page for a list of open positions.
18:27 UTC
Building Reddit’s Frontend with Vite
Written by Jim Simon. Acknowledgements: Erin Esco and Nick Stark.
Hello, my name is Jim Simon and I’m a Staff Engineer on Reddit’s Web Platform Team. The Web Platform Team is responsible for a wide variety of frontend technologies and architecture decisions, ranging from deployment strategy to monorepo tooling to performance optimization.
One specific area that falls under our team’s list of responsibilities is frontend build tooling. Until recently, we were experiencing a lot of pain with our existing Rollup based build times and needed to find a solution that would allow us to continue to scale as more code is added to our monorepo.
For context, the majority of Reddit’s actively developed frontend lives in a single monolithic Git repository. As of the time of this writing, our monorepo contains over 1000 packages with contributions from over 200 authors since its inception almost 4 years ago. In the last month alone, 107 authors have merged 679 pull requests impacting over 300,000 lines of code. This is all to illustrate how impactful our frontend builds are on developers, as they run on every commit to an open pull request and after every merge to our main branch.
A slow build can have a massive impact on our ability to ship features and fixes quickly and, as you’re about to see, our builds were pretty darn slow.
The Problem Statement
Reddit’s frontend build times are horribly slow and are having an extreme negative impact on developer efficiency. We measured our existing build times and set realistic goals for both of them:
| Build Type | Rollup Build Time | Goal |
|---|---|---|
| Initial Client Build | ~118 seconds | Less than 10 seconds |
| Incremental Client Build | ~40 seconds | Less than 10 seconds |
Yes, you’re reading that correctly. Our initial builds were taking almost two full minutes to complete and our incremental builds were slowly approaching the one minute mark. Diving into this problem illustrated a few key aspects that were causing things to slow down:
- Typechecking – Running typechecking was eating up the largest amount of time. While this is a known common issue in the TypeScript world, it was actually more of a symptom of the next problem.
- Total Code Size – One side effect of having a monorepo with a single client build is that it pushes the limits of what most build tooling can handle. In our case, we just had an insane amount of frontend code being built at once.
Fortunately we were able to find a solution that would help with both of these problems.
The Proposed Solution – Vite
To solve these problems we looked towards a new class of build tools that leverage ESBuild to do on-demand “Just-In-Time” (JIT) transpilation of our source files. The two options we evaluated in this space are Web Dev Server and Vite, and we ultimately landed on adopting Vite for the following reasons:
- Simplest to configure
- Most module patterns are supported out of the box which means less time spent debugging dependency issues
- Support for custom SSR and backend integrations
- Existing Vite usage already in the repo (Storybook, “dev:packages”)
- Community momentum
Note that Web Dev Server is a great project, and is in many ways a better choice as it’s rooted in web standards and is a lot more strict in the patterns it supports. We likely would have selected it over Vite if we were starting from scratch today. In this case we had to find a tool that could quickly integrate with a large codebase that included many dependencies and patterns that were non-standard, and our experience was that Vite handled this more cleanly out of the box.
Developing a Proof of Concept
When adopting large changes, it’s important to verify your assumptions to some degree. While we believed that Vite was going to address our problems, we wanted to validate those beliefs before dedicating a large amount of time and resources to it.
To do so, we spent a few weeks working on a barebones proof of concept. We did a very “quick and dirty” partial implementation of Vite on a relatively simple page as a means of understanding what kind of benefits and risks would come out of adopting it. This proof of concept illuminated several key challenges that we would need to address and allowed us to appropriately size and resource the project.
With this knowledge in hand, we green-lit the project and began making the real changes needed to get everything working. The resulting team consisted of three engineers (myself, Erin Esco, and Nick Stark), working for roughly two and a half months, with each engineer working on both the challenges we had originally identified as well as some additional ones that came up when we moved beyond what our proof of concept had covered.
It’s not all rainbows and unicorns…
Thanks to our proof of concept, we had a good idea of many of the aspects of our codebase that were not “Vite compatible”, but as we started to adopt Vite we quickly ran into a handful of additional complications as well. All of these problems required us to either change our code, change our packaging approach, or override Vite’s default behavior.
Vite’s default handling of stylesheets
Vite’s default behavior is to work off of HTML files. You give it the HTML files that make up your pages and it scans for stylesheets, module scripts, images, and more. It then either handles those files JIT when in development mode, or produces optimized HTML files and bundles when in production mode.
One side effect of this behavior is that Vite tries to inject any stylesheets it comes across into the corresponding HTML page for you. This breaks how Lit handles stylesheets and the custom templating we use to inject them ourselves. The solution is to append ?inline to the end of each stylesheet path: e.g. import styles from './top-button.less?inline'. This tells Vite to skip inserting the stylesheet into the page and to instead inline it as a string in the bundle.
Not quite ESM compliant packages
Reddit’s frontend packages had long been marked with the required ”type”: “module” configuration in their package.json files to designate them as ESM packages. However, due to quirks in our Rollup build configuration, we never fully adopted the ESM spec for these packages. Specifically, our packages were missing “export maps”, which are defined via the exports property in each package’s package.json. This became extremely evident when Vite dumped thousands of “Unresolved module” errors the first time we tried to start it up in dev mode.
In order to fix this, we wrote a codemod that scanned the entire codebase for import statements referencing packages that are part of the monorepo’s yarn workspace, built the necessary export map entries, and then wrote them to the appropriate package.json files. This solved the majority of the errors with the remaining few being fixed manually.
Javascript code before and after
Cryptic error messages
After rolling out export maps for all of our packages, we quickly ran into a problem that is pretty common in medium to large organizations: communication and knowledge sharing. Up to this point, all of the devs working on the frontend had never had to deal with defining export map entries, and our previous build process allowed any package subpath to be imported without any extra work. This almost immediately led to reports of module resolution errors, with Typescript reporting that it was unable to find a module at the paths developers were trying to import from. Unfortunately, the error reported by the version of Typescript that we’re currently on doesn’t mention export maps at all, so these errors looked like misconfigured tsconfig.json issues for anyone not in the know.
To address this problem, we quickly implemented a new linter rule that checked whether the path being imported from a package is defined in the export map for the package. If not, this rule would provide a more useful error message to the developer along with instructions on how to resolve the configuration issue. Developers stopped reporting problems related to export maps, and we were able to move on to our next challenge.
“Publishable” packages
Our initial approach to publishing packages from our monorepo relied on generating build output to a dist folder that other packages would then import from: e.g. import { MyThing } from ‘@reddit/some-lib/dist’. This approach allowed us to use these packages in a consistent manner both within our monorepo as well as within any downstream apps relying on them. While this worked well for us in an incremental Rollup world, it quickly became apparent that it was limiting the amount of improvement we could get from Vite. It also meant we had to continue running a bunch of tsc processes in watch mode outside of Vite itself.
To solve this problem, we adopted an ESM feature called “export conditions”. Export conditions allow you to define different module resolution patterns for the import paths defined in a package’s export map. The resolution pattern to use can then be specified at build time, with a default export condition acting as the fallback if one isn’t specified by the build process. In our case, we configured the default export condition to point to the dist files and defined a new source export condition that would point to the actual source files. In our monorepo we tell our builds to use the source condition while downstream consumers fallback on the default condition.
Legacy systems that don’t support export conditions
Leveraging export conditions allowed us to support our internal needs (referencing source files for Vite) and external needs (referencing dist files for downstream apps and libraries) for any project using a build system that supported them. However, we quickly identified several internal projects that were on build tools that didn’t support the concept of export conditions because the versions being used were so old. We briefly evaluated the effort of upgrading the tooling in these projects but the scope of the work was too large and many of these projects were in the process of being replaced, meaning any work to update them wouldn’t provide much value.
In order to support these older projects, we needed to ensure that the module resolution rules that older versions of Node relied on were pointing to the correct dist output for our published packages. This meant creating root index.ts “barrel files” in each published package and updating the main and types properties in the corresponding package.json. These changes, combined with the previously configured default export condition work we did, meant that our packages were set up to work correctly with any JS bundler technology actively in use by Reddit projects today. We also added several new lint rules to enforce the various patterns we had implemented for any package with a build script that relied upon our internal standardized build tooling.
Framework integration
Reddit’s frontend relies on an in-house framework, and that framework depends on an asset manifest file that’s produced by a custom Rollup plugin after the final bundle is written to the disk. Vite, however, does not build everything up front when run in development mode and thus does not write a bundle to disk, which means we also have no way of generating the asset manifest. Without going into details about how our framework works, the lack of an asset manifest meant that adopting Vite required having our framework internally shim one for development environments.
Fortunately we were able to identify some heuristics around package naming and our chunking strategy that allowed us to automatically shim ~99% of the asset manifest, with the remaining ~1% being manually shimmed. This has proven pretty resilient for us and should work until we’re able to adopt Vite for production builds and re-work our asset loading and chunking strategy to be more Vite-friendly.
Vite isn’t perfect
At this point we were able to roll Vite out to all frontend developers behind an environment variable flag. Developers were able to opt-in when they started up their development environment and we began to get feedback on what worked and what didn’t. This led to a few minor and easy fixes in our shim logic. More importantly, it led to the discovery of a major internal package maintained by our Developer Platform team that just wouldn’t resolve properly. After some research we discovered that Vite’s dependency optimization process wasn’t playing nice with a dependency of the package in question. We were able to opt that dependency out of the optimization process via Vite’s config file, which ultimately fixed the issue.
Typechecking woes
The last major hurdle we faced was how to re-enable some level of typechecking when using Vite. Our old Rollup process would do typechecking on each incremental build, but Vite uses ESBuild which doesn’t do it at all. We still don’t have a long-term solution in place for this problem, but we do have some ideas of ways to address it. Specifically, we want to add an additional service to Snoodev, our k8s based development environment, that will do typechecking in a separate process. This separate process would be informative for the developer and would act as a build gate in our CI process. In the meantime we’re relying on the built-in typechecking support in our developers’ editors and running our legacy rollup build in CI as a build gate. So far this has surprisingly been less painful than we anticipated, but we still have plans to improve this workflow.
Result: Mission Accomplished!
So after all of this, where did we land? We ended up crushing our goal! Additionally, the timings below don’t capture the 1-2 minutes of tsc build time we no longer spend when switching branches and running yarn install (these builds were triggered by a postinstall hook). On top of the raw time savings, we have significantly reduced the complexity of our dev runtime by eliminating a bunch of file watchers and out-of-band builds. Frontend developers no longer need to care about whether a package is “publishable” when determining how to import modules from it (i.e. whether to import source files or dist files).
| Build Type | Rollup Build Time | Goal | Vite Build Time |
|---|---|---|---|
| Initial Client Build | ~118 seconds | Less than 10 seconds | Less than 1 second |
| Incremental Client Build | ~40 seconds | Less than 10 seconds | Less than 1 second |
We also took some time to capture some metrics around how much time we’re collectively saving developers by the switch to Vite. Below is a screenshot of the time savings from the week of 05/05/2024 - 05/11/2024:
Adding these two numbers up means we saved a total of 4.22 days worth of build time over the course of a week. These numbers are actually under-reporting as well because, while working on this project, we also discovered and fixed several issues with our development environment configuration that were causing us to do full rebuilds instead of incremental builds for a large number of file changes. We don’t have a good way of capturing how many builds were converted, but each file change that was converted from a full build to an incremental build represents an additional ~78 seconds of time savings beyond what is already being captured by our current metrics.
In addition to the objective data we collected, we also received a lot of subjective data after our launch. Reddit has an internal development Slack channel where engineers across all product teams share feedback, questions, patterns, and advice. The feedback we received in this channel was overwhelmingly positive, and the number of complaints about build issues and build times significantly reduced. Combining this data with the raw numbers from above, it’s clear to us that this was time well spent. It’s also clear to us that our project was an overwhelming success, and internally our team feels like we’re set up nicely for additional improvements in the future.
Do projects like this sound interesting to you? Do you like working on tools and libraries that increase developer velocity and allow product teams to deliver cool and performant features? If so, you may be interested to know that my team (Web Platform) is hiring! Looking for something a little different? We have you covered! Reddit is hiring for a bunch of other positions as well, so take a look at our careers page and see if anything stands out to you!
14:45 UTC
A Day in the Life of a Reddit Tech Executive Assistant
Written by Mackenzie Greene
Hello from behind the curtain
I’m Mackenzie, and for the last five years, I’ve had the distinct pleasure of being the Executive Assistant (EA) to Reddit’s CTO, Chris Slowe, and many of his VPs along the way. Growing alongside Chris, the Tech Organization, the EA team, and Reddit as a whole has been an exciting, challenging, and immensely rewarding journey.
I say “hello from behind the curtain” because that’s where we EAs typically get our work done. While Reddit’s executives are presenting on stage, sitting at the head of a conference room table, or speaking on an earnings call, their EAs are working furiously behind the curtain to make everything click. So what goes on behind the curtain? It’s impossible for me to explain one single ‘day in the life’, for no two days are the same. My role is a whirlwind dance that involves juggling people, places, things, time, tasks, schedules, and agendas. It’s chaos. It’s mayhem. But, it’s beautiful. Each day brings new challenges and opportunities, and I wouldn’t have it any other way.
Every day MUST begin with coffee
Wherever I am in the world, I cannot kick off my workday without my morning coffee. For me, coffee is not just about the caffeine boost - it’s about centering myself mentally and preparing for the day ahead. Whether I'm grabbing a cappuccino at the Reddit office, brewing a pot in my kitchen, or sipping a latte from the mountains, I’ll always make room for a fresh cup of ‘jo before wor
Then it’s off to the races
I open my laptop, pull out my notebook and nose dive into the digital chaos: sifting through emails, Slack messages, and calendar notifications. I chat with fellow EAs, check in with Executives, and ensure no fires need extinguishing from the night before. I often compare my role to that of an air traffic controller, but instead of planes, it’s meetings, deadlines, messages, reminders, and presentations that need landing. It’s all about keeping everything on track and ensuring that nothing crashes.
Cat Herding
Free time is scarce for any executive, especially for the CTO of a freshly public company. My day-to-day consists of working behind the scenes to ensure that every hour of Chris’s day is used efficiently - hopefully, to make his life and the lives of his almost 1200 direct and indirect reports easier. Monday mornings, I kick off the week with Chris and his Chief of Staff, Lisa, in what we call the ‘Tech Cat Herders’ Meeting. Here, we run through the week's agenda and scheme for what's ahead. I ensure that Chris and his VP’s are prepared and know what to expect with their meetings for the day and the week. This often means communicating with cross functional (XFN) partners to jointly prepare an agenda, creating slides for All-Hands meetings, or gathering the notes and action items from emails. However, no matter how prepared we are, there are always changes! Reddit is a dynamic, fast-paced environment with shifting deadlines, competing priorities, eager employees, and seemingly infinite projects running in parallel. For Chris, and for me by proxy, this means constant change - further underscoring the importance of always being on my toes.
In between the chaos
While cat-herding makes up a significant portion of my day, project-based work (beyond schedule and calendar management) is quickly becoming one of my favorite parts of my role. Reddit’s mission is to bring community and belonging to everyone in the world, and I try to apply this mission to my work within the Product and Tech organization. I am a people-person at my core, and thankfully, Reddit has recognized this and encouraged me to pursue side-projects to help foster a sense of community and engagement within the organization.
One such example is the Reddit Engineering Mentorship Panel. I saw an opportunity to encourage and create conversation around mentorship within the team, so I created (and MC’d!) an Engineering Mentorship Panel. I assembled a diverse group of panelists whom I encouraged to discuss specific and unique forms of mentorship, and share challenges and success stories alike. Adding value through initiatives like this is deeply fulfilling to me. It's about more than just organizing events—it's about nurturing an environment where individuals can learn from each other, grow together, and feel a sense of belonging. This is just one example of a project that Reddit allows me to lean into my passion for community-building to drive meaningful engagement and development opportunities for my team.
EOD
As the day winds down, I do a final sweep of emails and tasks to ensure nothing has slipped through the cracks. I set up the agenda for the next day, ensuring that everything is in place for another round of organized chaos. I banter a bit with the EA team, sharing stories about mishaps behind the curtain.
There you have it—a tiny glimpse into the beautifully chaotic life of an Executive Assistant at Reddit. It’s a role that demands adaptability, precision, and a good sense of humor (remember I am working amongst the finest trolls). Being an Executive Assistant isn’t just about managing schedules and screening calls. It’s about being the behind-the-scenes partner who keeps everything running smoothly. It’s a mix of strategy, diplomacy and a little magic. And yes, sometimes it is herding cats, but I wouldn’t trade it for anything
It’s impossible for Chris to be in every place at once, therefore I have to clone him.
16:59 UTC
Post Guidance and Community Safety with Phil Aquilina | Building Reddit Ep. 19
Post Guidance and Community Safety with Phil Aquilina
Hello Reddit!
I’m happy to announce the nineteenth episode of the Building Reddit podcast. In today’s episode, I interviewed Staff Engineer Phil Aquilina about his work with the new Post Guidance feature and the Community Automations platform that it’s built on. We also cover some of his history at Reddit (spoiler: He’s an OG) and how he got into software engineering.
You can listen on all major podcast platforms: Apple Podcasts, Spotify, Google Podcasts, and more!
Reddit is a big place and the safety of our users is one of our highest priorities. Scaling that safety is a constant focus, and we’ve built and evolved many different tools to enable that, used by Reddit employees and by community moderators.
In this episode, you’ll hear from Phil Aquilina, a Staff Engineer on the Community Safety team. His team recently had a big win with the release of the Post Guidance feature, which is built on top of the Community Automations platform that he designed. He’s also been at Reddit for a while, so we’ll dive into his tenure at Reddit, why he’s still excited about coming to work, and how his work is making Reddit safer for everyone.
Check out all the open positions at Reddit on our careers site: https://www.redditinc.com/careers
21:43 UTC
Introducing a Global Retrieval Ranking Model in the Ads Funnel
Written by: Simon Kim, Matthew Dornfeld, and Tingting Zhang.
Context
In this blog post, we will explore the Ads Retrieval team’s journey to introduce the global retrieval ranking (also known as the First Pass Ranker) in the Ads Funnel, with the goal of improving marketplace performance and reducing infrastructure expenses.
Global Auction Trimmer in Marketplace
Reddit is a vast online community with millions of active users engaged in various interest-based groups. Since launching its ad auction system, Reddit has aimed to enhance ad performance and help advertisers efficiently reach the right users, optimizing budget utilization. This is done by passing more campaigns through the system and selecting optimal ad candidates based on advertisers' targeting criteria.
With the increasing number of ads from organic advertiser growth, initiatives to increase candidate submissions, and the growing complexity of heavy ranking models, it has become challenging to scale prediction model serving without incurring significant costs. The global auction trimmer, the candidate selection process is essential for efficiently managing system costs and seizing business opportunities by:
- Enhancing advertiser and marketplace results by selecting high-quality candidate ads at scale, reducing the pool from millions to thousands.
- Maintaining infrastructure performance stability and cost efficiency.
- Improving user experience and ensuring high ad quality.
Model Challenge
The Ads Retrieval team has been experimenting with various ML-based embedding models and utility functions over the past 1.5 years. Initially, the team utilized traditional NLP methods to learn latent representations of ads, such as word2vec and doc2vec. Later, they transitioned to a more complex Two-Tower Sparse Network.
When using the traditional embedding models, we observed an improvement in ad quality, but it was not as significant as expected. Moreover, these models were not sufficient to enhance advertiser and marketplace results or improve user experience and ensure high ad quality. Consequently, we decided to move to the Two-Tower Sparse Network.
However, we discovered that building a traditional Two-Tower Sparse Network required creating multiple models for different campaign objective types. This approach would lead to having multiple user embeddings for each campaign objective type, substantially increasing our infrastructure costs to serve them.
The traditional embedding models and the traditional Two-Tower Sparse Network
Our Solution: Multi-task two-tower sparse network model
To overcome this problem, we decided to use the Multi-tasks two tower sparse network for the following reasons.
- Ad-Specific Learning: The ad tower’s multi-task setup allows for the optimization of different campaign objectives (clicks, video views, conversion etc) simultaneously. This ensures that the ad embeddings are well-tuned for various campaign objective types, enhancing overall performance.
- Task-Specific Outputs: By having separate output layers for different ad objective types, the model can learn task-specific representations while still benefiting from shared lower-level features.
- Enhanced Matching: By learning a single user embedding and multiple ad embeddings (for different campaign objective types), the model can better match users with the most relevant ads for each campaign objective type, improving the overall user experience.
- Efficiency in Online Inference
- Single User Embedding: Using a single user embedding across multiple ad embeddings reduces computational complexity during online inference. This makes the system more efficient and capable of handling high traffic with minimal latency.
- Dynamic Ad Ranking: The model can dynamically rank ads for different campaign objective types in real-time, providing a highly responsive and adaptive ad serving system.
You can see the Multi-tasks learning two tower model architecture in the below image.
Multi-tasks learning two tower model architecture
System Architecture
The global trimmer is deployed in the Adserver shard with an online embedding delivery service. This enables the sourcing of more candidates further upstream in the auction funnel, addressing one of the biggest bottlenecks: the data and CPU-intensive heavy ranker model used in the Ad Inference Server. The user-ad two-tower sparse network model is updated daily. User embeddings are retrieved every time a request is made to the ad selector service, which determines which ads to show on Reddit. While embeddings are generated online, we cache them for 24 hours. Ad embeddings are updated approximately every five minutes.
Model Training Pipeline
We developed a model training pipeline with clearly defined steps, leveraging our in-house Ad TTSN engine. The user-ad muti-task two tower sparse network (MTL-TTSN) model is retained by several gigabytes of user engagement, ad interactions, and their contextual information. We implemented this pipeline on the Kubeflow platform.
Model Serving
After training, the user and ad MTL-TTSN models consist of distinct user and ad towers. For deployment, these towers are split and deployed separately to dedicated Gazette model servers.
Embedding Delivery Service
The Embedding Service is capable of dynamically serving all embeddings for the user and ad models. It functions as a proxy for the Gazette Inference Service (GIS), the platform hosting Reddit's ML models. This service is crucial as it centralizes the caching and versioning of embeddings retrieved from GIS, ensuring efficient management and retrieval.
Model Logging and Monitoring
After a model goes live, we meticulously monitor its performance to confirm it benefits the marketplace. We record every request and auction participant, as well as hundreds of additional metadata fields, such as the specific model used and the inference score provided to the user. These billions of daily events are sent to our data warehouse, enabling us to analyze both model metrics and the business performance of each model. Our dashboards provide a way to continuously track a model’s performance during experiments.
Conclusion and What’s Next
We are still in the early stages of our journey. In the coming months, we will enhance our global trimmer sophistication by incorporating dynamic trimming to select the top K ads, advanced exploration logic, allowing more upstream candidates to flow in and model improvements. We will share more blog posts about these projects and use cases in the future.
Acknowledgments and Team: The authors would like to thank teammates from Ads Retrieval team including Nastaran Ghadar, Samantha Han, Ryan Lakritz, François Meunier, Artemis Nika, Gilad Tsur, Sylvia Wu, and Anish Balaji as well as our cross-functional partners: Kayla Lee, Benjamin Rebertus, James Lubowsky, Sahil Taneja, Marat Sharifullin, Yin Zhang, Clement Wong, Ashley Dudek, Jack Niu, Zack Keim, Aaron Shin, Mauro Napoli, Trey Lawrence, and Josh Cherry.
Last but not least, we greatly appreciate the strong support from the leadership: Xiaorui Gan, Roelof van Zwol, and Hristo Stefanov.
22:23 UTC
Instant Comment Loading on Android & iOS
Written by Ranit Saha (u/rThisIsTheWay) and Kelly Hutchison (u/MoarKelBell)
Reddit has always been the best place to foster deep conversations about any topic on the planet. In the second half of 2023, we embarked on a journey to enable our iOS and Android users to jump into conversations on Reddit more easily and more quickly! Our overall plan to achieve this goal included:
- Modernizing our Feeds UI and re-imagining the user’s experience of navigating to the comments of a post from the feeds
- Significantly improve the way we fetch comments such that from a user’s perspective, conversation threads (comments) for any given post appear instantly, as soon as they tap on the post in the feed.
This blog post specifically delves into the second point above and the engineering journey to make comments load instantly.
Observability and defining success criteria
The first step was to monitor our existing server-side latency and client-side latency metrics and find opportunities to improve our overall understanding of latency from a UX perspective. The user’s journey to view comments needed to be tracked from the client code, given the iOS and Android clients perform a number of steps outside of just backend calls:
- UI transition and navigation to the comments page when a user taps on a post in their feed
- Trigger the backend request to fetch comments after landing on the comments page
- Receive and parse the response, ingest and keep track of pagination as well as other metadata, and finally render the comments in the UI.
We defined a timer that starts when a user taps on any post in their Reddit feed, and stops when the first comment is rendered on screen. We call this the “comments time to interact” (TTI) metric. With this new raw timing data, we ran a data analysis to compute the p90 (90th percentile) TTI for each user and then averaged these values to get a daily chart by platform. We ended up with our baseline as ~2.3s for iOS and ~2.6s for Android:
Comment tree construction 101
The API for requesting a comment tree allows clients to specify max count and max depth parameters. Max count limits the total number of comments in the tree, while max depth limits how deeply nested a child comment can be in order to be part of the returned tree. We limit the nesting build depth to 10 to limit the computational cost and make it easier to render from a mobile platform UX perspective. Nested children beyond 10 depth are displayed as a separate smaller tree when a user taps on the “More replies” button.
The raw comment tree data for a given ‘sort’ value (i.e., Best sort, New sort) has scores associated with each comment. We maintain a heap of comments by their scores and start building the comments ’tree’ by selecting the comment at the top (which has the highest score) and adding all of its children (if any) back into the heap, as candidates. We continue popping from the heap as long as the requested count threshold is not reached.
Pseudo Code Flow:
- Fetch raw comment tree with scores
- Select all parent (root) comments and push them into a heap (sorted by their score)
- Loop the requested count of comments
- Read from the heap and add comment to the final tree under their respective parent (if it's not a root)
- If the comment fetched from the heap has children, add those children back into the heap.
- If a comment fetched from the heap is of depth > requested_depth (or 10, whichever is greater), and wrap them under the “More replies” cursor (for that parent).
- Loop through remaining comments in the heap, if any
- Read from the heap and group them by their parent comments and create respective “load more” cursors
- Add these “load more” cursors to the final tree
- Return the final tree
Example:
A post has 4 comments: ‘A’, ‘a’, ‘B’, ‘b’ (‘a’ is the child of ‘A’, ‘b’ of ‘B’). Their respective scores are: { A=100, B=90, b=80, a=70 }.If we want to generate a tree to display 4 comments, the insertion order is [A, B, b, a].
We build the tree by:
- First consider candidates [A, B] because they're top level
- Insert ‘A’ because it has the highest score, add ‘a’ as a candidate into the heap
- Insert ‘B’ because it has the highest score, add ‘b’ as a candidate into the heap
- Insert ‘b’ because it has the highest score
- Insert ‘a’ because it has the highest score
Scenario A: max_comments_count = 4
Because we nest child comments under their parents the displayed tree would be:
A
-a
B
-b
Scenario b: max_comments_count = 3
If we were working with a max_count parameter of ‘3’, then comment ‘b’ would not be added to the final tree and instead would still be left as a candidate when we get to the end of the ranking algorithm. In the place of ‘b’, we would insert a ‘load_more’ cursor like this:
A
-a
B
- load_more(children of B)
With this method of constructing trees, we can easily ‘pre-compute’ trees (made up of just comment-ids) of different sizes and store them in caches. To ensure a cache hit, the client apps request comment trees with the same max count and max depth parameters as the pre-computed trees in the cache, so we avoid having to dynamically build a tree on demand. The pre-computed trees can also be asynchronously re-built on user action events (like new comments, sticky comments and voting), such that the cached versions are not stale. The tradeoff here is the frequency of rebuilds can get out of control on popular posts, where voting events can spike in frequency. We use sampling and cooldown period algorithms to control the number of rebuilds.
Now let's take a look into the high-level backend architecture that is responsible for building, serving and caching comment trees:
- Our comments service has Kafka consumers using various engagement signals (i.e., upvote, downvotes, timestamp, etc…) to asynchronously build ‘trees’ of comment-ids based on the different sort options. They also store the raw complete tree (with all comments) to facilitate a new tree build on demand, if required.
- When a comment tree for a post is requested for one of the predefined tree sizes, we simply look up the tree from the cache, hydrate it with actual comments and return back the result. If the request is outside the predefined size list, a new tree is constructed dynamically based on the given count and depth.
- The GraphQL layer is our aggregation layer responsible for resolving all other metadata and returning the results to the clients.
- Comment tree construction 101
Client Optimizations
Now that we have described how comment trees are built, hopefully it’s clear that the resultant comment tree output depends completely on the requested max comment count and depth parameters.
Splitting Comments query
In a system free of tradeoffs, we would serve full comment trees with all child comments expanded. Realistically though, doing that would come at the cost of a larger latency to build and serve that tree. In order to balance this tradeoff and show user’s comments as soon as possible, the clients make two requests to build the comment tree UI:
- First request with a requested max comment count=8 and depth=10
- Second request with a requested max comment count=200 and depth=10
The 8 comments returned from the first call can be shown to the user as soon as they are available. Once the second request for 200 comments finishes (note: these 200 comments include the 8 comments already fetched), the clients merge the two trees and update the UI with as little visual disruption as possible. This way, users can start reading the top 8 comments while the rest load asynchronously.
Even with an initial smaller 8-count comment fetch request, the average TTI latency was still >1000ms due to time taken by the transition animation for navigating to the post from the feed, plus comment UI rendering time. The team brainstormed ways to reduce the comments TTI even further and came up with the following approaches:
- Faster screen transition: Make the feed transition animation faster.
- Prefetching comments: Move the lower-latency 8-count comment tree request up the call stack, such that we can prefetch comments for a given post while the user is browsing their feed (Home, Popular, Subreddit). This way when they click on the post, we already have the first 8 comments ready to display and we just need to do the latter 200-count comment tree fetch. In order to avoid prefetching for every post (and overloading the backend services), we could introduce a delay timer that would only prefetch comments if the post was on screen for a few seconds.
- Reducing response size: Optimize the amount of information requested in the smaller 8-count fetch. We identified that we definitely need the comment data, vote counts and moderation details, but wondered if we really need the post/author flair and awards data right away. We explored the idea of waiting to request these supplementary metadata until later in the larger 200-count fetch.
Here's a basic flow of the diagram:
This ensures that Redditors get to see and interact with the initial set of comments as soon as the cached 8-count comment tree is rendered on screen. While we observed a significant reduction in the comment TTI, it comes with a couple of drawbacks:
- Increased Server Load - We increased the backend load significantly. Even a few seconds of delay to prefetch comments on feed yielded an average increase of 40k req/s in total (combining both iOS/Android platforms). This will increase proportionally with our user growth.
- Visual flickering while merging comments - The largest tradeoff though is that now we have to consolidate the result of the first 8-count call with the second 200-count call once both of them complete. We learned that comment trees with different counts will be built with a different number of expanded child comments. So when the 200-count fetch completes, the user will suddenly see a bunch of child comments expanding automatically. This leads to a jarring UX, and to prevent this, we made changes to ensure the number of uncollapsed child comments are the same for both the 8-count fetch and 200-count fetch.
Backend Optimizations
While comment prefetching and the other described optimizations were being implemented in the iOS and Android apps, the backend team in parallel took a hard look at the backend architecture. A few changes were made to improve performance and reduce latency, helping us achieve our overall goals of getting the comments viewing TTI to < 1000ms:
- Migrated to gRPC from Thrift (read our previous blog post on this).
- Made sure that the max comment count and depth parameters sent by the clients were added to the ‘static predefined list’ from which comment trees are precomputed and cached.
- Optimized the hydration of comment trees by moving them into the comments-go svc layer from the graphQL layer. The comments-go svc is a smaller golang microservice with better efficiency in parallelizing tasks like hydration of data structures compared to our older python based monolith.
- Implemented a new ‘pruning’ logic that will support the ‘merge’ of the 8-count and 200-count comment trees without any UX changes.
- Optimized the backend cache expiry for pre-computed comment trees based on the post age, such that we maximize our pre-computed trees cache hit rate as much as possible.
The current architecture and a flexible prefetch strategy of a smaller comment tree also sets us up nicely to test a variety of latency-heavy features (like intelligent translations and sorting algorithms) without proportionally affecting the TTI latency.
Outcomes
So what does the end result look like now that we have released our UX modernization and ultra-fast comment loading changes?
- Global average p90 TTI latency improved by 60.91% for iOS, 59.4% for Android
- ~30% reduction in failure rate when loading the post detail page from feeds
- ~10% reduction in failure rates on Android comment loads
- ~4% increase in comments viewed and other comment related engagements
We continue to collect metrics on all relevant signals and monitor them to tweak/improve the collective comment viewing experience. So far, we can confidently say that Redditors are enjoying faster access to comments and enjoying diving into fierce debates and reddit-y discussions!
If optimizing mobile clients sounds exciting, check out our open positions on Reddit’s career site.
21:18 UTC
Day in a Life of a Principal Security Engineer
a securimancer working to keep Reddit safe and secure
Written by u/securimancer
Greetings fine humans. I’m here today writing a “Day in a Life” blog post because someone asked me to. I cannot imagine this is interesting, but Redditors tend to surprise me so let’s do this.
Morning Routine
Like many of us, mornings are when I take care of all the dependent lifeforms under my command. Get in an hour or so of video games (Unicorn Overlord currently) for my mental health. Feed the coterie of beasts (including the children), make coffee for the wife and me, prep the kids for school. Catch up on Colbert (my news needs comedy otherwise darkness consumes), check out what’s been happening on Medium and Reddit, and read a few of my favorite cybersecurity / engineering mail lists. Crack open the ol’ calendar and see what my ratio of “get shit done” to “help other people get shit done” is in store for my day. All roughly before 8am. And the beauty of working for a Bay Area company (if we can call it that, we’re so remote friendly) is that I normally have a precious few hours before people in SF wake up to get things done.
Daily Tasks
Each morning has a brief reflection of what I need to get done that day. I’m a big fan of the Eisenhower Method to figure out what I actually need to prioritize in my day. It’s exceedingly rare that I get a majority of my day focused on work that I’ve initiated, so prioritizing activities from code review and pull request feedback to architectural systems design reviews to pair programming requests from the team to random break/fix fires that pop up, all of that gets organized so I feel like I’m (at least trying) to do the most impactful work for the day. Reddit has a few systems to help drive queues of work: Jira for planned work and “big rock” items that we’re trying to accomplish for that quarter, Harold (an in-house developed shame mechanism) for code review and deployment, and Launch Control (Reddit’s flavor of Google’s LaunchCal) for architecture design reviews. Plenty of potential dopamine hits as “things to get done.”
Meetings
It’s exceedingly rare that I have meetings that could have been an email (and if I do, they’re almost always vendor meetings). A lot of what my meetings tend to focus on are around conflict resolutions across teams as we try to achieve different goals or drive consensus to resolve problems that come up on various programs teams are trying to deliver. Working on Security, you can often get perceived as the “Department of No”, but in every meeting I work hard to make sure that isn’t the case. It starts with getting a shared context of what is the problem at hand, understanding the outcomes that we need to drive toward and inputs into the problem (timelines, humans, trade offs), and deciding how we move forward. Meetings are a terrible way to convey decisions as they are only as good as the individuals that remember them, so lots of these meetings are centered around decision docs or technical design reviews. Capturing your rationale for a decision not only helps make sure you understand the problem (if you can’t write about it, it’s hard to think about it), but also helps capture the whys and rationale behind those decisions for future you and other product and engineering staff.
There’s also meetings that I live for, those that are building up humans. We have biweekly SPACE (Security, Privacy, and Compliance Engineering) brown bags where we talk about new things we’ve shipped or some training topic that upskills all of us. We have biweekly threat modeling meetings where we pick a topic/scenario and go through a threat modeling exercise live, which helps build the muscle memory of how to do technical diagramming, and helps build a shared context of how the system works, what our risk appetite is, and how various team members think about the problem providing multiple viewpoints to the discussion (honestly the most valuable component). As a Principal Engineer, I’m keenly aware of my humanity and the fact that I do not scale in my efforts alone: training and building up future PEs is how I scale myself (at least until cloning becomes more readily available).
Ubiquity
One of my super powers is being everything everywhere all at once, or so I’ve been told by my fellow Snoos. I’ve been told that I have an uncanny knack to be in so many Slack channels and part of so many threads of discussion that it’s “inhuman”. Being a damn fine security engineer is hard because not only do you have to have the understanding and context of the thing you’re trying to secure, but also know how to actually secure the thing. This is nigh impossible if you don’t know what’s going on in your business (and we’re still “small enough” size-wise that this is still possible for one human), so I’ve got Slack keyword alerts, channel organization, and a giant 49” ultrawide monitor that has a dedicated Slack tiled window to keep me plugged in and accessible. I also have developed over many years my response to pings from Slack: “Can I solve this problem, if not who can? Is this something I should solve or can I delegate? Can this be answered async with good quality, or is a larger block of dedicated time required to solve? Is this thread too long and needs a different approach?” This workflow is second nature to me and helps me move around the org. I’ve also been here almost 5 years and, as I’m in Security and have to know everything about everything to secure anything (which I don’t, but I am a master of Googling, learning, and listening), I’ve been exposed to pretty much everything in our engineering sphere. With that knowledge comes great power of helping connect teams together that wouldn’t have connected otherwise.
Do Security Stuffs
Occasionally I actually get to do “security” things. These past two quarters it’s been launching Reddit’s “unified access control” solution leveraging Cloudflare Zero Trust, moving us off old crusty Nginx OAuth proxies onto a modern system that has such groundbreaking things like <sarcasm> caching and logs </sarscasm>, among other things. But really, it’s the planning, designing, and execution of a complex technical migration with only a handful of engineers. I oversee security across the entire business so that requires opining on web app security, k8s / AWS / GCP security, IAM concepts, observability, mobile app dev, CI/CD security, and all the design patterns that are included in this smörgåsbord of technology. Keeping all this in my head is why I can’t remember names and faces and my wife has to tell me multiple times where I’m supposed to be and when. But the thing that keeps me going is always the “building”, seeing things get stood up at Reddit that I know are sound and secure. It’s not denying people’s requests or crapping all over a developer for picking a design they didn’t know had a serious security design flaw. We’re not a bank (either in terms of money we get to throw at security, or tolerance for security friction), we get to make risk tradeoff decisions based on Reddit’s risk tolerance (which is high except where it comes to privacy or financial exchanges) and listen to our business as we try to find ways to improve ads serving and improve our users’ experience. So I view myself like any other software engineer, I just happen to know a lot about security. And I guess not just security, I know a lot about our safety systems, our networking environment, and our Kubernetes architecture. It just comes with the Security space, that inquisitive mind of “how does this thing work?” and wanting to be competent when you talk about it and try to secure it.
Not everything is 0s and 1s, however. A lot of security is process, paperwork, and persistence. Designing workflow approval processes for how an IAM flow should look like. Reviewing IT corporate policies for accuracy and applicability. Crafting responses to potential advertisers’ IT teams on “how secure is Reddit, really”. Writing documentation for how an engineering system works and how other engineers should interact with it. Updating runbooks with steps on how others should respond to an incident or page. Building Grafana dashboards to quantify and visualize how a tooling rollout is working. Providing consulting on product features like authentication / authorization business logic across services. Interviewing, not only for my own team but also within other engineering and cross-functional areas of the business.
End of Day Routine
Eventually, I run out of time in the day as I’m beckoned away from my dark, cave-like, Diet Coke strewn office by the promise of dinner. Wrapping up document review, (hopefully) crossing things off my to-do list, and closing out Slack threads for the day, I try to pack everything up and not carry it with me after work. It’s challenging being an almost completely remote company with a heavy presence in the West Coast, as pings and notifications come in as dinner and kids’ bedtime happens. But I know not everything can be finished in a day, some things will slip, and there will always be more work tomorrow. Which is juxtaposed occasionally with bouts of imposter syndrome, even for someone as senior and tenured as I am. Happens to all of us.
After-hours work is restricted to on-call duty and pet projects. You don’t want to know how many on-call queues I’m secondary escalation on. Or how many Single Point of Securimancers services that I still own (looking at you, Reddit onion service). And pet projects are typically things that I’ve got desires to do: prototyping security solutions we want to look into, messing with my k8s homelab, doing routine upgrades. Nothing clears the mind like watching semver numbers go up (until you find the undocumented change that breaks everything).
Future Outlook
And finally, what's on the horizon for our little SPACE team? We’re still a small team coming out of IPO, and our greatest super power is networking and influencing our engineering peers. We got our ISO 27001 and SOC2 Type 2 last year and continue to ever increase scope and complexity of public accreditation. We’re close partners with our Infrastructure and IT teams to modernize our tech and continue to evolve our capabilities in host and network security, data loss prevention, and security observability. We’ve got two wonderful interns from YearUp that started and are going to be with us this summer, and we continue to focus on improving our team composition (more women and diversity, more junior folks and less singleton seniors). All of this work takes effort by this PE.
So there you have it, a “day in a life” of a u/securimancer. If you made it this far, congratulations on your achievement. Got any questions or want to share your own experiences? Drop 'em in the comments below!
20:20 UTC