/r/RedditEng
Welcome to the Reddit Tech Blog. We'll be talking about how we contribute to Reddit's mission to bring community and belonging to everyone. Read about how we build stuff people love and what it's like to be part of our team.
If you're interested in working with us, go to our careers page: https://www.redditinc.com/careers
Reddit Engineering Tech Blog
/r/RedditEng
Written by Dom Valencia
I'm Dom Valenciana, a Senior Software Engineer at the heart of Reddit's Advertiser Reporting. Today, I pull back the curtain on a development so unique it might just redefine how you view advertising tech. Amidst the bustling world of digital ads, we at Reddit have crafted our own programming language and modern web-based IDE, specifically designed to supercharge our "Custom Columns" feature. While it might not be your go-to for crafting the next chatbot, sleek website, or indie game, our creation stands proud as a Turing-complete marvel. Accompanied by a bespoke IDE complete with all the trimmings—syntax highlighting, autocomplete, type checking.
Join me as we chart the course from the spark of inspiration to the pinnacle of innovation, unveiling the magic behind Reddit's latest technological leap.
At the beginning of our bi-annual company-wide Hackathon, a moment when great ideas often come to light, my project manager shared a concept with me that sparked our next big project. She suggested enhancing our platform to allow advertisers to perform basic calculations on their ad performance data directly within our product. She observed that many of our users were downloading this data, only to input it into Excel for further analysis using custom mathematical formulas. By integrating this capability into our product, we could significantly streamline their workflow.
This idea laid the groundwork for what we now call Custom Columns. If you're already familiar with using formulas in Excel, then you'll understand the essence of Custom Columns. This feature is a part of our core offering, which includes Tables and CSVs displaying advertising data. It responds to a clear need from our users: the ability to conduct the same kind of calculations they do in Excel, but seamlessly within our platform.
As soon as I laid eyes on the mock-ups, I was captivated by the concept. It quickly became apparent that, perhaps without fully realizing it, the product and design teams had laid down a challenge that was both incredibly ambitious and, by conventional standards, quite unrealistic for a project meant to be completed within a week. But this daunting prospect was precisely what I relished. Undertaking seemingly insurmountable projects during hackweeks aligns perfectly with my personal preference for how to invest my time in these intensive, creative bursts.
Understandably, within the limited timeframe of the hackathon, we only managed to develop a basic proof of concept. However, this initial prototype was sufficient to spark significant interest in further developing the project.
Building an interpreter or compiler is a classic challenge in computer science, with a well-documented history of academic problem-solving. My inspiration for our project at Reddit comes from two influential resources:
Writing An Interpreter In Go by Thorsten Ball
I'll only skim the surface of the compiler and interpreter concepts—not to sidestep their complexity, but to illuminate the real crux of our discussion and the true focal point of this blog: the journey and innovation behind the IDE.
In the spirit of beginning with the basics, I utilized my prior experience crafting a Lexer and Parser to navigate the foundational stages of building our IDE.
We identified key functionalities essential to our IDE:
The standard route in compiling involves starting with the Lexer, which tokenizes input, followed by the Parser, which constructs an Abstract Syntax Tree (AST). This AST then guides the Interpreter in executing the code.
A critical aspect of this project was to ensure that these complex processes were seamlessly integrated with the user’s browser experience. The challenge was to enable real-time code input and instant feedback—bridging the intricate workings of Lexer and Parser with the user interface.
With plenty of sources on the topic and the details of the linter squared away the biggest looming question was: How do you build a Browser-Based IDE? Go ahead, I'll give you time to google it. As of May 2024, when this document was written, there is no documentation on how to build such a thing. This was the unfortunate reality I faced when I was tasked with building this feature. The hope was that this problem had already been solved and that I could simply plug into an existing library, follow a tutorial, or read a book. It's a common problem, right?
After spending hours searching through Google and scrolling past the first ten pages of results, I found myself exhausted. My search primarily turned up Stack Overflow discussions and blog posts detailing the creation of basic text editors that featured syntax highlighting for popular programming languages such as Python, JavaScript, and C++. Unfortunately, all I encountered were dead ends or solutions that lacked completeness. Faced with this situation, it became clear that the only viable path forward was to develop this feature entirely from scratch.
The initial approach I considered was to use a basic <textarea></textarea>
HTML element and attach an event listener to capture its content every time it changed. This content would then be processed by the Lexer and Parser. This method would suffice for rudimentary linting and type checking.
However, the <textarea>
element inherently lacks the capability for syntax highlighting or autocomplete. In fact, it offers no features for manipulating the text within it, leaving us with a simple, plain text box devoid of any color or interactive functionality.
So Textbox + String Manipulation is out.
The subsequent approach I explored, which led to a detailed proof of concept, involved utilizing the contenteditable attribute to make any element editable, a common foundation for many What You See Is What You Get (WYSIWYG) editors. Initially, this seemed like a viable solution for basic syntax highlighting. However, the implementation proved to be complex and problematic.
As users typed, the system needed to dynamically update the HTML of the text input to display syntax highlighting (e.g., colors) and error indications (e.g., red squiggly lines). This process became problematic with contenteditable elements, as both my code and the browser attempted to modify the text simultaneously. Moreover, user inputs were captured as HTML, not plain text, necessitating a parser to convert HTML back into plain text—a task that is not straightforward. Challenges such as accurately identifying the cursor's position within the recursive HTML structure, or excluding non-essential elements like a delete button from the parsed text, added to the complexity.
Additionally, this method required conceptualizing the text as an array of tokens rather than a continuous string. For example, to highlight the number 123 in blue to indicate a numeric token, it would be encapsulated in HTML like <span class="number">123</span>, with each word and symbol represented as a separate HTML element. This introduced an added layer of complexity, including issues like recalculating the text when a user deletes part of a token or managing user selections spanning multiple tokens.
So ContentEditable + HTML Parsing is out.
For months, I struggled with a problem, searching for solutions but finding none satisfying. Eventually, I stepped back to reassess, choosing to work backwards from the goal in smaller steps.
With the Linter set up, I focused on creating an intermediary layer connecting them to the Browser. This layer, I named TextNodes, would be a character array with metadata, interacted with via keyboard inputs.
This approach reversed my initial belief about data flow direction, from HTML Textbox to JavaScript structure to the opposite.
Leveraging array manipulation, I crafted a custom textbox where each TextNode lived as a <span>, allowing precise control over text and style. A fake cursor, also a <span>, provided a visual cue for text insertion and navigation.
A overly simplified version of this solution would look like this:
This was precisely the breakthrough I needed! My task now simplified to rendering and manipulating a single array of characters, then presenting it to the user.
At this point, you might be wondering, "How does creating a custom text box solve the problem? It sounds like a lot of effort just to simulate a text box." The approach of utilizing an array to generate <span> elements on the screen might seem straightforward, but the real power of this method lies in the nuanced communication it facilitates between the browser and the parsing process.
Here's a clearer breakdown: by employing an array of TextNodes as our fundamental data structure, we establish a direct connection with the more sophisticated structures produced by the Lexer and Parser. This setup allows us to create a cascading series of references—from TextNodes to Tokens, and from Tokens to AST (Abstract Syntax Tree) Nodes. In practice, this means when a user enters a character into our custom text box, we can first update the TextNodes array. This change then cascades to the Tokens array and subsequently to the AST Nodes array. Each update at one level triggers updates across the others, allowing information to flow seamlessly back and forth between the different layers of data representation. This interconnected system enables dynamic and immediate reflection of changes across all levels, from the user's input to the underlying abstract syntax structure.
When we pair this with the ability to render the TextNodes array on the screen in real time, we can immediately show the user the results of the Lexer and Parser. This means that we can provide syntax highlighting, autocomplete, linting, and type checking in real time.
Let's take a look at a diagram of how the textbox will work in practice:
After the user's keystroke we update the TextNodes and recalculate the Tokens and AST via the Lexer and Parser. We make sure to referentially link the TextNodes to the Tokens and AST Nodes. Then we re-render the Textbox using the updated TextNodes. Since each TextNode has a reference to the Token it represents, we can apply syntax highlighting, autocomplete, linting, and type checking to the TextNodes individually. We can also reference what part of the AST the TextNode is associated with to determine if it's part of a valid expression.
What began as a Hackathon spark—integrating calculation features directly within Reddit's platform—morphed into the Custom Columns project, challenging and thrilling in equal measure. From a nascent prototype to a fully fleshed-out product, the evolution was both a personal and professional triumph.
So here we are, at the journey's end but also at the beginning of a new way advertisers will interact with data. This isn't just about what we've built; it’s about de-mystifying tooling that even engineers feel is magic. Until the next breakthrough—happy coding.
Written by Sanjay Kairam (Staff Scientist - Machine Learning/Community)
Every day, thousands of people around the world start new communities on Reddit. Have you ever wondered what’s special about the founders who create those communities that take off from the very beginning?
Working with Jeremy Foote from Purdue University, we surveyed 951 community founders just days after they had created their new communities. We wanted to understand their motivations, goals, and community-building plans. Based on differences in these community attitudes, we then built statistical models to predict how much their newly-created communities would grow over the first 28 days.
This research will appear in May at CHI 2024, but we wanted to share some of our findings with you first, to help you kickstart your communities on Reddit.
What fuels a founder?
Passion for a specific topic is what drives most community founders on Reddit, and it’s also what drives communities that have the most successful early trajectories. 63% of founders that we surveyed created their community out of topical interest, followed by 39% who created their community to exchange information, and 37% who wanted to connect with others. Founders who are motivated by a specific topic create engaging spaces that attract more unique visitors, contributors, and subscribers over the first 28 days.
Different strokes for different folks.
Every founder has their own vision of success for their community, and their communities tend to succeed along those terms. Our survey asked founders to rank various measures for how they would evaluate the success of their communities. Some measures focused on quantity (e.g. a large number of contributors) and others focused on quality (e.g. high-quality information about the topic). We found that founders varied broadly in terms of which measures they preferred. Quality-oriented founders attracted more early contributors while quantity-oriented founders attracted more early visitors. In other words, founders’ goals translate into differences in the communities they build.
Strategic moves for community growth.
The types of community-building strategies that founders have, both within and outside of Reddit, have a measurable impact on the early success of their communities. Founders who had specific plans to raise awareness about their community attracted 273% more visitors in the first 28 days, than those without these plans. They also attracted 75% more contributors and 189% more subscribers. Founders who had specific plans to welcome newcomers or encourage contributions also had measurably more contributors after 28 days. For inspiration, you can learn more here about specific strategies that mods have used to successfully grow their communities.
The diversity of communities across Reddit comes from the diversity of the founders of these communities, who each bring their own backgrounds, motivations, and goals to these spaces. At Reddit, my role is connected to understanding and modeling this diversity and working with design, community, and product teams on developing tools that support every founder on their journey.
If you’ve thought about creating a community, there’s no better time than now! Just remember: make the topic and purpose of your community clear, have a clear vision of success, and take the initiative to raise awareness of your community both on and off Reddit. We can’t wait to welcome your new community as part of Reddit’s diverse, international ecosystem.
P.S. We have some “starting community” guides on https://redditforcommunity.com/ that have super helpful tips for how to start and grow your Reddit community.
P.P.S. If doing this type of research sounds exciting, check out our open positions on Reddit’s career site.
Written by Nick Fohs - CorpTech Systems & Infra Manager.
Snoo & a Yubikey with a sign that says \"Yubikey acquired!\"
Following the Security Incident we experienced in February of 2023, Reddit’s Corporate Technology and Security teams took a series of steps to better secure our internal infrastructure and business systems.
One of the most straightforward changes that we made was to implement WebAuthn based security keys as the mechanism by which our employees use Multi Factor Authentication (MFA) to log into internal systems. In this case, we worked with Yubico to source and ship YubiKeys to all workers at Reddit.
WebAuthn based MFA is a phishing resistant implementation of Public Key Cryptography that allows various websites to identify a user based on a one time registration of keypair. Or, it allows each device to register with a website in a way that will only allow you through if the same device presents itself again.
Why is this better than other options? One time passcodes, authenticator push notifications, and SMS codes can all generally be used on other computers or by other people, and are not limited to the device that’s trying to log in.
We elected to send 2x YubiKey 5C NFC to everyone to ensure that we could cover the most variety of devices, and facilitate login from mobile phones. We were focused on getting everyone at least one key to rely on, and one to act as a backup in case of loss or damage. We don’t limit folks from adding the WebAuthn security key of their choice if they already had one, and enabled people to expense a different form factor if they preferred.
Frankly, we continue to evaluate the key choice decision and may change this for new hires in the future. In the context of a rapid global rollout, we wanted to be sure that everyone had a key that would work with as many devices as possible, and a backup in case of failure to minimize downtime if someone lost their main key.
As our laptop fleet is 95% Mac, we also encouraged the registration of Touch ID as an additional WebAuthn Factor. We found that the combination of these two together is easiest for daily productivity, and ensures that the device people use regularly can still authenticate if they are away from their key.
At the time of our rollout, most of the Touch ID based registrations for our identity platforms were based on Browser-specific pairings (mostly in Chrome). While the user experience is generally great, the registration was bound to Chrome’s cookies, and would leave the user locked out if they needed to clear cookies. Pairing a YubiKey was the easiest way to ensure they had a persistent factor enrolled that could be used across whatever device they needed to log in on.
At the core, the challenge with a large-scale hardware rollout is a logistical one. Reddit has remained a highly distributed workforce, and people are working from 50 different countries.
We began with the simple step of collecting all shipping addresses. Starting with Google Forms and App Script, we were able to use Yubi Enterprise Delivery APIs to perform data validation and directly file the shipment. Yubico does have integration into multiple ticketing and service management platforms, and even example ordering websites that can be deployed quickly. We opted for Google Forms for speed, trust, and familiarity to our users
From there, shipment, notification, and delivery were handled by Yubico to its supported countries. For those countries with workers not on the list, we used our existing logistics providers to help us ship keys directly.
The major change in WebAuthn and Security Keys has been the introduction and widespread adoption of Passkeys. Passkeys are a definite step forward in eliminating the shortcomings of passwords, and improving security overall. In the Enterprise though, there are still hurdles to relying only on Passkeys as the only form of authentication.
Our teams have been very active over the past year shipping a bunch of process, technology, and security improvements to better secure our internal teams. We’re going to try and continue sharing as much as we can as we reach major milestones.
If you want to learn more, come hang out with our Security Teams at SnooSec in NYC on July 15th. You can check out the open positions on our Corporate Technology or Security Teams at Reddit.
Written by Vikram Aravamudhan, Staff Software Engineer.
tldr;
- We share the telemetry behind Reddit's Home Feed or just any other feed.
- Home rewrite project faced some hurdles with regression on topline metrics.
- Data wizards figured that 0.15% load error manifested as 5% less posts viewed.
- Little Things Matter, sometimes!
This is Part 2 in the series. You can read Part 1 here - Rewriting Home Feed on Android & iOS.
We launched a Home Feed rewrite experiment across Android and iOS platforms. Over several months, we closely monitored key performance indicators to assess the impact of our changes.
We encountered some challenges, particularly regression on a few top-line metrics. This prompted a deep dive into our front-end telemetry. By refining our instrumentation, our goal was to gather insights into feed usability and user behavior patterns.
Within this article, we shed light on such telemetry. Also, we share experiment-specific observability that helped us solve the regression.
Core non-interactive eventing on Feeds
================================================
The following events are the signals we monitor to ensure the health and performance of all feeds in Web, Android and iOS apps.
Home screen (and many other screens) records both successful and failed feed fetches, and captures the following metadata to analyze feed loading behaviors.
Events
feed-load-success
feed-load-fail
Additional Metadata
load_type
feed_size
correlation_id
error_reason
Each time a post appears on the screen, an event is logged. In the context of a feed rewrite, this guardrail metric was monitored to ensure users maintain a consistent scrolling behavior and encounter a consistent number of posts within the feed.
Events
post-view
Additional Metadata
experiment_variant
- The variant of the rewrite experiment.correlation_id
To ensure users have engaged with a post rather than just speed-scrolling, an event is recorded after a post has been on the screen for at least 2 seconds.
Events
post-consume
Additional Metadata
correlation_id
A large number of interactions can occur within a post, including tapping anywhere within its area, upvoting, reading comments, sharing, hiding, etc. All these interactions are recorded in a variety of events. Most prominent ones are listed below.
Events
post-click
post-vote
Additional Metadata
click_location
- The tap area that the user interacted with. This is essential to understand what part of the post works and the users are interested in.Reddit posts feature a variety of media content, ranging from static text to animated GIFs and videos. These videos may be hosted either on Reddit or on third-party services. By tracking the performance of the video player in a feed, the integrity of the feed rewrite was evaluated.
Events
videoplayer-start
videoplayer-switch-bitrate
videoplayer-served
videoplayer-watch_[X]_percent
========================================
In addition to monitoring the volume of analytics events, we set up supplemental observability in Grafana. This helped us compare the backend health of the two endpoints under experimentation.
In the new feeds architecture, we opted to change the way image quality was picked. Rather than the client requesting a specific thumbnail size or asking for all available sizes, we let the server drive the thumbnail quality best suited for the device.
Network Requests from the apps include display specifications, which are used to compute the optimal image quality for different use cases. Device Pixel Ratio (DPR) and Screen Width serve as core components in this computation.
Events (in Grafana)
image_response_size_bytes
(b/w variants)Additional Metadata
experiment_variant
During the experimentation phase, we observed a decrease in Posts Viewed. This discrepancy indicated that the experiment group was not scrolling to the same extent as the control group. More on this later.
To validate our hypothesis, we introduced observability on Request Per Second (RPS) by variant. This provided an overview of the volume of posts fetched by each device, helping us identify any potential frontend rendering issues.
Events (in Grafana)
rps
(b/w variants)error_rate
(b/w variants)posts_in_response
(b/w variants)Additional Metadata
experiment_variant
========================================
From a basic dashboard comparing the volume of aforementioned telemetry to a comprehensive analysis, the team explored numerous correlations between these metrics.
These were some of the questions that needed to be addressed.
Q. Are users seeing the same amount of posts on screen in Control and Treatment?
Signals validated: Feed Load Success & Error Rate, Post Views per Feed Load
Q. Are feed load behaviors consistent between Control and Treatment groups?
Signals validated: Feed Load By Load Type, Feed Fails By Load Type, RPS By Page Number
Q. Are Text, Images, Polls, Video, GIFs, Crossposts being seen properly?
Signals validated: Post Views By Post Type, Post Views By Post Type
Q. Do feed errors happen the first time they open or as they scroll?
Signals validated: Feed Fails By Feed Size
========================================
During the experimentation phase, we observed a decrease in Posts Viewed. This discrepancy indicated that the experiment group was not scrolling to the same extent as the control group.
Feed Error rate increased from 0.3% to 0.6%, but caused 5% decline in Posts viewed This became a “General Availability” blocker. With the help of data wizards from our Data Science group, the problem was isolated to an error that had a mere impact of 0.15% in the overall error rate. By segmenting this population, the altered user behavior was clear.
The downstream effects of a failing Feed Load we noticed were:
Stepping into this investigation, the facts we knew:
Once we fixed this miniscule error, the experiment unsurprisingly recovered to its intended glory.
LITTLE THINGS MATTER!!!
I just wanted to post a message of thanks to all of the Engineers (and friends-of-engineering) who have posted here over the last couple of years, striving to provide an inside view of what it's like to work at Reddit (and what it is, exactly, that we're trying to do here)
https://i.redd.it/prgmgosyequc1.gif
I also want to thank the (now) 10k subscribers for being here. Hopefully you're enjoying it too!
And while I'm standing at this mic, what do you want to hear more about?
Written by Erin Esco.
For the past few years, we have been developing a next-generation web app internally referred to as “Shreddit”, a complete rebuild of the web experience intended to provide better stability and performance to users. When we found ourselves able to support traffic on this new app, we wanted to run the migrations as A/B tests to ensure both the platform and user experience changes did not negatively impact users.
Legacy web application user interface
Shreddit (our new web application) user interface
The initial experiment set-up to migrate traffic from the old app (“legacy” to represent a few legacy web apps) to the new app (Shreddit) was as follows:
A sequence diagram of the initial routing logic for cross-app experiments.
When a user made a request, Fastly would hash the request’s URL and convert it to a number (N) between 0 and 99. That number was used to determine if the user landed on the legacy web app or Shreddit. Fastly forwarded along a header to the web app to tell it to log an event that indicated the user was exposed to the experiment and bucketed.
This flow worked, but presented a few challenges:
- Data analysis was manual. Because the experiment set-up did not use the SDKs offered from our experiments team, data needed to be analyzed manually.
- Event reliability varied across apps. The web apps had varying uptime and different timings for event triggers, for example:
a. Legacy web app availability is 99%
b. Shreddit (new web app) availability is 99.5%
This meant that when bucketing in experiments we would see a 0.5% sample ratio mismatch which would make our experiment analysis unreliable.
- Did not support experiments that needed access to user information. We could not run an experiment exclusively for or without mods.
As Shreddit matured, it reached a point where there were enough features requiring experimentation that it was worth investing in a new service to leverage the experiments SDK to avoid manual data analysis.
Let’s go over the original life cycle of a request to a web app at Reddit in order to better understand the proposed architecture.
A diagram of the different services/entities a request encounters in its original life cycle.
User requests pass through Fastly then to nginx which makes a request for authentication data that gets attached and forwarded along to the web app.
The goal was to create a way to allow cross-app experiments to:
To achieve this, we devised a high-level plan to build a reverse proxy service (referred to hereafter as the “routing service”) to intercept requests and handle the following:
Envoy is a high-performance proxy that offers a rich configuration surface for routing logic and customization through extensions. It has gained increasing adoption at Reddit for these reasons, along with having a large active community for support.
The diagram below shows where we envisioned Envoy would sit in the overall request life cycle.
A high-level diagram of where we saw the new reverse proxy service sitting.
These pieces above are responsible for different conceptual aspects of the design (experimentation, authentication, etc).
The service’s responsibility is to bucket users in experiments, fire expose events, and send them to the appropriate app. This requires access to the experiments SDK, a sidecar that keeps experiment data up to date, and a sidecar for publishing events.
We chose to use an External Processing Filter to house the usage of the experiments SDK and ultimately the decision making of where a request will go. While the external processor is responsible for deciding where a request will land, it needs to pass the information to the Envoy router to ensure it sends the request to the right place.
The relationship between the external processing filter and Envoy’s route matching looks like this:
A diagram of the flow of a request with respect to experiment decisions.
Once this overall flow was designed and we handled abstracting away some of the connections between these pieces, we needed to consider how to enable frontend developers to easily add experiments. Notably, the service is largely written in Go and YAML, the former of which is not in the day to day work of a frontend engineer at Reddit. Engineers needed to be able to easily add:
For an engineer to add an experiment to the routing service, they need to make two changes:
External Processor (Go Service)
Developers add an entry to our experiments map where they define their experiment name and a function that takes a request as an argument and returns back whether a given request is eligible for that experiment. For example, an experiment targeting logged in users visiting their settings page, would check if the user was logged in and navigating to the settings page.
Entries to Envoy’s route_config
Once developers have defined an experiment and what requests are eligible for it, they must also define what variant corresponds to what web app. For example, control might go to Web App A and your enabled variant might go to Web App B.
The external processor handles translating experiment names and eligibility logic into a decision represented by headers that it appends to the request. These headers describe the name and variant of the experiment in a predictable way that developers can interface with in Envoy’s route_config
to say “if this experiment name and variant, send to this web app”.
This config (and the headers added by the external processor) is ultimately what enables Envoy to translate experiment decisions to routing decisions.
Prior to launch, we integrated a few types of testing as part of our workflow and deploy pipeline.
For the external processor, we added unit tests that would check against business logic for experiment eligibility. Developers can describe what a request looks like (path, headers, etc.) and assert that it is or is not eligible for an experiment.
For Envoy, we built an internal tool on top of the Route table check tool that verified the route that our config matched was the expected value. With this tool, we can confirm that requests landed where we expect and are augmented with the appropriate headers.
Our first experiment was an A/A test that utilized all the exposure logic and all the pieces of our new service, but the experiment control and variant were the same web app. We used this A/A experiment to put our service to the test and ensure our observability gave us a full picture of the health of the service. We also used our first true A/B test to confirm we would avoid the sample ratio mismatch that plagued cross-app experiments before this service existed.
There were a number of things we instrumented to ensure we could measure that the service met our expectations for stability, observability, and meeting our initial requirements.
Experiment Decisions
We tracked when a request was eligible for an experiment, what variant the experiments SDK chose for that request, and any issues with experiment decisions. In addition, we verified exposure events and validated the reported data used in experiment analysis.
Measuring Packet Loss
We wanted to be sure that when we chose to send a request to a web app, it actually landed there. Using metrics provided by Envoy and adding a few of our own, we were able to compare Envoy’s intent of where it wanted to send requests against where they actually landed.
With these metrics, we could see a high-level overview of what experiment decisions our external processing service was making, where Envoy was sending the requests, and where those requests were landing.
Zooming out even more, we could see the number of requests that Fastly destined for the routing service, landed in the nginx layer before the routing service, landed in the routing service, and landed in a web app from the routing service.
Following our A/A test, we made the service generally available internally to developers. Developers have utilized it to run over a dozen experiments that have routed billions of requests. Through a culmination of many minds and tweaks, we have a living service that routes requests based on experiments and the final architecture can be found below.
Written by Emily Mucken on behalf of Reddit’s Women Eng Employee Resource Group (ERG)
We are a community of women Snoos (employees) who are working in engineering roles here at Reddit!
The goal of our group is to foster a greater sense of community & belonging with each other and our allies through events, camaraderie, and upskilling.
Here’s a little more about us:
Most of our Women Eng Snoos are located in the US & Canada, but we also have members in Spain, the UK and the Netherlands! Most of our engineering roles are 100% remote, allowing us the freedom and flexibility to work from a location that suits our life and needs best.
Women in engineering here at Reddit partner with tech leaders to host internal education and development events (recent highlights were a Design Docs class, and a Code Review class hosted by internal experts on these topics).
Reddit offers our Snoos a professional development stipend to use towards upskilling and adding knowledge in areas we are curious about.
We have weekly (optional!) virtual & IRL hangouts with each other to stay connected.
The vibe is real-talk, supportive… and fun!
We love having a safe space to vent to peers who “get it”.
In addition to being part of Women Eng, many of our members belong to other communities here inside of Reddit:
In our group, you’ll find: kid moms, cat moms, dog moms, plant moms, musicians, artists, scientists, athletes, puzzle-lovers, fashionistas, speakers, writers and podcasters and more!
We are each unique, but united by a passion for promoting, supporting and advancing our talented women in engineering here at Reddit.
We are … building Reddit!
We have women in engineering roles of all levels and distributed across all orgs:
If you’re interested in what it’s like to be an engineer and a trans woman at Reddit, check out our most recent Building Reddit podcast episode featuring Lonni Ingram!
Building Reddit Ep. 18: Front-End Craftsmanship with Lonni Ingram
Hello Reddit!
I’m happy to announce the eighteenth episode of the Building Reddit podcast. In today’s episode, I interviewed Staff Front-End Engineer Lonni Ingram about how she works with Reddit’s web experience. We dive into many of the site features you already use, including the new Shreddit stack and the text editor.
There may or may not also be some very useful cooking tips in this episode, so I hope you enjoy it! Let me know in the comments.
You can listen on all major podcast platforms: Apple Podcasts, Spotify, Google Podcasts, and more!
If you’ve visited Reddit with a web browser in the past few months, then you likely landed on our new front-end experience, internally named Shreddit. This new implementation took years to finish and the effort of many engineers, but the end result is a faster and cleaner experience that is easier than ever to use.
One of the engineers who works on that project, Lonni Ingram, joins the podcast in this episode. She’s worked on several different aspects of Reddit’s web Front-end, from the text editor to the post composer, in her role as a Staff Front-End Engineer. In this discussion she shares more about how front-end development works at reddit, some of the toughest bugs she’s encountered, and what she’s excited about on the web.
Check out all the open positions at Reddit on our careers site: https://www.redditinc.com/careers
Written by Vikram Aravamudhan
ℹ️tldr;
We have rewritten Home, Popular, News, Watch feeds on our mobile apps for a better user experience. We got several engineering wins.
Android uses Jetpack Compose, MVVM and server-driven components. iOS uses home-grown SliceKit, MVVM and server-driven components.
Happy users. Happy devs. 🌈
---------------------------------------------
This is Part 1 in the “Rewriting Home Feed” series. You can find Part 2 in next week's post.
In mid-2022, we started working on a new tech stack for the Home and Popular feeds in Reddit’s Android and iOS apps. We shared about the new Feed architecture earlier. We suggest reading the following blogs written by Merve and Alexey.
Re-imagining Reddit’s Post Units on Android : r/RedditEng - Merve explains how we modularized the feed components that make up different post units and achieved reusability.
Improving video playback with ExoPlayer : r/RedditEng - Alexey shares several optimizations we did for video performance in feeds. A must read if your app has ExoPlayer.
As of this writing, we are happy and proud to announce the rollout of the newest Home Feed (and Popular, News, Watch & Latest Feed) to our global Android and iOS Redditors 🎉. Starting as an experiment mid-2023, it led us into a path with a myriad of learnings and investigations that fine tuned the feed for the best user experience. This project helped us move the needle on several engineering metrics.
Prior to this project’s inception, we knew we wanted to make improvements to the Home screen. Time To Interact (TTI), the metric we use to measure how long the Home Feed takes to render from the splash screen, was not ideal. The response payloads while loading feeds were large. Any new feature addition to the feed took the team an average 2 x 2-week-sprints. The screen instrumentation needed much love. As the pain points kept increasing, the team huddled and jotted down (engineering) metrics we ought to move before it was too late.
A good design document should cover the non-goals and make sure the team doesn’t get distracted. Amidst the appetite for a longer list of improvements mentioned above, the team settled on the following four success metrics, in no particular order.
Home TTI = App Initialization Time (Code) + Home Feed Page 1 (Response Latency + UI Render)
We measure this from the time the splash screen opens, to the time we finish rendering the first view of the Home screen. We wanted to improve the responsiveness of the Home presentation layer and GQL queries.
Goals:
Non-Goals:
Over the course of time, our GQL response sizes became heavier and there was no record of the Fields [to] UI Component mapping. At the same time, our p90 values in non-US markets started becoming a priority in Android.
Goals:
Non-Goals:
Addition of any new feature to an existing feed was not quick and took the team an average of 1-2 sprints. The problem was exacerbated by not having a wide variety of reusable components in the codebase.
There are various ways to measure Developer Productivity in each organization. At the top, we wanted to measure New Development Velocity, Lead time for changes and the Developer satisfaction - all of it, only when you are adding new features to one of the (Home, Popular, etc.) feeds on the Reddit platform.
Goals:
Non-Goals:
UI Snapshot test helps to make sure you catch unexpected changes in your UI. A test case renders a UI component and compares it with a pre-recorded snapshot file. If the test fails, the change is unexpected. The developers can then update the reference file if the change is intended. Reddit’s Android & iOS codebase had a lot of ground to cover in terms of UI snapshot test coverage.
Plan:
The Home experiment ran for 8 months. Over the course, we hit immediate wins on some of the Core Metrics. On other regressed metrics, we went into different investigations, brainstormed many hypotheses and eventually closed the loose ends.
Look out for Part 2 of this “Rewriting Home Feed” series explaining how we instrumented the Home Feed to help measure user behavior and close our investigations.
Across both platforms, the TTI wins were great. This improvement means, we are able to surface the first Home feed content in front of the user 10-12% quicker and users will see Home screen 200ms-300ms faster.
Image 1: iOS TTI improvement of 10-12% between our Control (1800 ms) and Test (1590 ms)
Image 2: Android TTI improvement of 10-12% between our Control (2130 ms) and Test (1870 ms)
We experimented with different page sizes, trimmed the response payload with necessary fields for the first render and noticed a decent reduction in the response size.
Image 3: First page requests for home screen with 50% savings in gzipped response (20kb ▶️10kb)
We identified upstream paths that were slow, optimized fields for speed, and provided graceful degradation for some of the less stable upstream paths. The following graph shows the overall savings on the global user base. We noticed higher savings in our emerging markets (IN, BR, PL, MX).
Image 4: (Region: US) First page requests for Home screen with 200ms-300ms savings in latency
Image 5: (Region: India) First page requests with (1000ms-2000ms) savings in latency
Once we got the basics of the foundation, the pace of new feed development changed for the better. While the more complicated Home Feed was under construction, we were able to rewrite a lot of other feeds in record time.
During the course of rewrite, we sought constant feedback from all the developers involved in feed migrations and got a pulse check around the following signals. All answers trended in the right direction.
Few other signals that our developers gave us feedback were also trending in the positive direction.
The previous feed architecture had a monolith codebase and had to be modified by someone working on any feed. To make it easy for all teams to build upon the foundation, on Android we adopted the following model:
Image 6: Android Feeds Modules
So, any new feed was to expect a plug-and-play approach and write only the implementation code. This sped up the dev effort. To understand how we did this on iOS, refer Evolving Reddit’s Feed Architecture : r/RedditEng
Image 7: Android Feed High-level Architecture
By writing smaller slices of UI components, we were able to supplement each with a snapshot test on both platforms. We have approximately 75 individual slices in Android and iOS that can be stitched in different ways to make a single feed item.
We have close to 100% coverage for:
We asked the individual teams to contribute snapshots whenever a new slice is added to the slice repository. Teams were able to catch the failures during CI builds and make appropriate fixes during the PR review process.
Continuing on the above engineering wins, teams are migrating more screens in the app to the new feed architecture. This ensures we’ll be delivering new screens in less time, feeds that load faster and perform better on Redditor’s devices.
Happy Users. Happy Devs 🌈
Thanks to the hard work of countless number of people in the Engineering org, who collaborated and helped build this new foundation for Reddit Feeds.
Special thanks to our blog reviewers Matt Ewing, Scott MacGregor, Rushil Shah.
Written by Stas Kravets
We've tackled the challenges of using Python at scale, particularly the lack of true multithreading and memory leaks in third-party libraries, by introducing Monoceros, a Go tool that launches multiple concurrent Python workers in a single pod, monitors their states, and configures an Envoy Proxy to route traffic across them. This enables us to achieve better resource utilization, manage the worker processes, and control the traffic on the pod.
In doing so, we've learned a lot about configuring Kubernetes probes properly and working well with Monoceros and Envoy. Specifically, this required caution when implementing "deep" probes that check for the availability of databases and other services, as they can cause cascading failures and lengthy recovery times.
Historically, Python has been one of Reddit's most commonly used languages. Our monolith was written in Python, and many of the microservices we currently operate are also coded in Python. However, we have had a notable shift towards adopting Golang in recent years. For example, we are migrating GraphQL and federated subgraphs to Golang. Despite these changes, a significant portion of our traffic still relies on Python, and the old GraphQL Python service must behave well.
To maintain consistency and simplify the support of services in production, Reddit has developed and actively employs the Baseplate framework. This framework ensures that we don't reinvent the wheel each time we create a new backend, making services look similar and facilitating their understanding.
For a backend engineer, the real fun typically begins as we scale. This presents an opportunity (or, for the pessimists, a necessity) to put theoretical knowledge into action. The straightforward approach, "It is a slow service; let's spend some money to buy more computing power," has its limits. It is time to think about how we can scale the API so it is fast and reliable while remaining cost-efficient.
At this juncture, engineers often find themselves pondering questions like, "How can I handle hundreds of thousands of requests per second with tens of thousands of Python workers?"
Python is generally single-threaded, so there is a high risk of wasting resources unless you use some asynchronous processing. Placing one process per pod will require a lot of pods, which might have another bad consequence - increased deployment times, more cardinality for metrics, and so on. Running multiple workers per pod is way more cost-efficient if you can find the right balance between resource utilization and contention.
In the past, one approach we employed was Einhorn, which proved effective but is not actively developed anymore. Over time, we also learned that our service became a noisy neighbor on restarts, slowing down other services sharing the nodes with us. We also found that the latency of our processes degrades over time, most likely because of some leaks in the libraries we use.
We noticed that the request latency slowly grew on days when we did not re-deploy it. But, it got better immediately after the deployment. Smells like a resource leak! In another case, we identified a connection leak in one of our 3rd-party dependencies. This leak was not a big problem during business hours when deployments were always happening, resetting the service. However, it became an issue at night. While waiting for the fixes, we needed to implement the service's periodical restart to keep it fast and healthy.
Another goal we aimed for was to balance the traffic between the worker processes in the pod in a more controlled manner. Einhorn, by way of SO_REUSEPORT
, only uses random connection balancing, meaning connections may be distributed across processes in an unbalanced manner. A proper load balancer would allow us to experiment with different balancing algorithms. To achieve this, we opted to use Envoy Proxy, positioned in front of the service workers.
When packing the pod with GraphQL processes, we observed that GraphQL became a noisy neighbor during deployments. During initialization, the worker requires much more CPU than normal functioning. Once all necessary connections are initialized, the CPU utilization goes down to its average level. The other pods running on the same node are affected proportionally by the number of GQL workers we start. That means we cannot start them all at once but should do it in a more controlled manner.
To address these challenges, we introduced Monoceros.
Monoceros is a Go tool that performs the following tasks:
While Monoceros proved exceptionally effective, over time, our deployments became more noisy with error messages in the logs. They also produced heightened spikes of HTTP 5xx errors triggering alerts in our clients. This prompted us to reevaluate our approach.
Because the 5xx spikes could only happen when we were not ready to serve the traffic, the next step was to check the configuration of Kubernetes probes.
Let's delve into the realm of Kubernetes probes consisting of three key types:
A straightforward method to configure these probes involves creating three or fewer endpoints. The Liveness Probe can return a 200 OK every time it's invoked. The Readiness Probe can be similar to the Liveness Probe but should return a 503 when the service shuts down. This ensures the probe fails, and Kubernetes refrains from sending new requests to the pod undergoing a restart or shutdown. On the other hand, the Startup Probe might involve a simple waiting period before completion.
An intriguing debate surrounds whether these probes should be "shallow" (checking only the target service) or "deep" (verifying the availability of dependencies like databases, cache, etc.) While there's no universal solution, caution is advised with "deep" probes. They can lead to cascading failures and extended recovery times.
Consider a scenario where the liveness check incorporates database connectivity, and the database experiences downtime. The pods get restarted, and auto-scaling reduces the deployment size over time. When the database is restored, all traffic returns, but with only a few pods running, managing the sudden influx becomes a challenge. This underscores the need for thoughtful consideration when implementing "deep" probes to avoid potential pitfalls and ensure robust system resilience.
These are the considerations for configuring probes we incorporated with the introduction of Envoy and Monoceros. When dealing with a single process per service pod, management is straightforward: the process oversees all threads/greenlets and maintains a unified view of its state. However, the scenario changes when multiple processes are involved.
Our initial configuration followed this approach:
Connection from Ingress via Envoy to Python workers with the configuration of the health probes
Looks reasonable, but where are all those 503s coming from?
Spikes of 5xx when the pod state is Not Ready
It was discovered that during startup when we sequentially launched all N Python workers, they weren't ready to handle the traffic immediately. Initialization and the establishment of connections to dependencies took a few seconds. Consequently, while the initial worker might have been ready when the last one started, some later workers were not. This led to probabilistic failures depending on the worker selected by the Envoy for a given request. If an already "ready" worker was chosen, everything worked smoothly; otherwise, we encountered a 503 error.
Ensuring all workers are ready during startup can be a nuanced challenge. A fixed delay in the startup probe might be an option, but it raises concerns about adaptability to changes in the number of workers and the potential for unnecessary delays during optimized faster deployments.
Enter the Health Check Filter feature of Envoy, offering a practical solution. By leveraging this feature, Envoy can monitor the health of multiple worker processes and return a "healthy" status when a specified percentage of them are reported as such. In Monoceros, we've configured this filter to assess the health status of our workers, utilizing the "aggregated" endpoint exposed by Envoy for the Kubernetes startup probe. This approach provides a precise and up-to-date indication of the health of all (or most) workers, and addresses the challenge of dynamic worker counts.
We've also employed the same endpoint for the Readiness probe but with different timeouts and thresholds. When assessing errors at the ingress, the issues we were encountering simply disappeared, underscoring the effectiveness of this approach.
Improvement of 5xx rate once the changes are introduced
Take note of the chart at the bottom, which illustrates that valid 503s returned during the readiness check when the pod shuts down.
Another lesson we learned was to eliminate checking the database connectivity in our probes. This check, which looked completely harmless, when multiplied by many workers, overloaded our database. When the pod starts during the deployment, it goes to the database to check if it is available. If too many pods do it simultaneously, the database becomes slow and can return an error. That means it is unavailable, so the deployment kills the pod and starts another one, worsening the problem.
Changing the probes concept from “everything should be in place, or I will not go out of the bed” to “If you want 200, give me my dependencies, but otherwise, I am fine” served us better.
Exercising caution when adjusting probes is paramount. Such modifications have the potential to lead to significant service downtime, and the repercussions may not become evident immediately after deployment. Instead, they might manifest at unexpected times, such as on a Saturday morning when the alignment of your data centers with the stars in the distant galaxy changes, influencing network connectivity in unpredictable ways.
Nonetheless, despite the potential risks, fine-tuning your probes can be instrumental in reducing the occurrence of 5xx errors. It's an opportunity worth exploring, provided you take the necessary precautions to mitigate unforeseen consequences.
You can start using Monoceros for your projects, too. It is open-sourced under the Apache License 2.0 and can be downloaded here.
Written by Ian Leitch
Today we are happy to announce the open-sourcing of one of our iOS testing tools, CodableRPC. CodableRPC is a general-purpose RPC client & server implementation that uses Swift’s Codable for serialization, enabling you to write idiomatic and type-safe procedure calls.
While a general-purpose RPC implementation, we’ve been using CodableRPC as a vital component of our iOS UI testing infrastructure. In this article, we will take a closer look at why RPC is useful in a UI testing context, and some of the ways we use CodableRPC.
Apple’s UI testing framework enables you to write high-level tests that query the UI elements visible on the screen and perform actions on them, such as asserting their state or performing gestures like tapping and swiping. This approach forces you to write tests that behave similarly to how a user would interact with your app while leaving the logic that powers the UI as an opaque box that cannot be opened. This is an intentional restriction, as a good test should in general only verify the contract expressed by a public interface, whether it be a UI, API, or single function.
But of course, there are always exceptions, and being able to inspect the app’s internal state, or trigger actions not exposed by the UI can enable some very powerful test scenarios. Unlike unit tests, UI tests run in a separate process from the target app, meaning we cannot directly access the state that resides within the app. This is where RPC comes into play. With the server running in the app, and the client in the test, we can now implement custom functionality in the app that can be called remotely from the test.
Now let’s take a look at some of the ways we’re using CodableRPC, and some potential future uses too.
We’ve made a significant reduction in app launch time over the past couple of years, and we’ve implemented regression tests to ensure our hard-earned gains don’t slip away. You’re likely imagining a test that benchmarks the app's launch time and compares it against a baseline. That’s a perfectly valid assumption, and it’s how we initially tried to tackle performance regression testing, but in the end, we ended up taking a different approach. To understand why, let’s look at some of the drawbacks of benchmarking:
We wanted our regression tests to run as pre-merge checks on our pull requests. This meant they needed to be fast, ideally completing in around 15 minutes or less (including build time). But we also wanted to cover all possible experiment scenarios. These requirements made benchmarking impractical, at least not without spending huge amounts of money on hardware and engineering time.
Instead, we chose to focus on preventing the kinds of actions that we know are likely to cause a performance regression. Loading dependencies, creating view controllers, rendering views, reading from disk, and performing network requests are all things we can detect. Our regression tests therefore launch the app once for each set of experiment variants and use CodableRPC to inspect the actions performed by the app. The test then compares the results with a hardcoded list of allowed actions.
Every solution has trade-offs, and you’d be right to point out that this approach won’t prevent regressions caused by actions that aren’t explicitly tested for. However, we’ve found these cases to be very rare. We are currently in the process of rearchitecting the app launch process, which will further prevent engineers from introducing accidental performance regressions, but we’ll leave that for a future article.
UI tests can be used as either local functional tests or end-to-end tests. With local functional testing, the focus is to validate that a given feature functions the same without depending on the state of remote systems. To isolate our functional tests, we developed an in-house solution for stubbing network requests and restoring the app state on launch. These mechanisms ensure our tests function consistently in scenarios where remote system outages may impact developer productivity, such as in pre-merge pull request checks. We use CodableRPC to signal the app to dump its state to disk when a test is running in “record” mode.
As a user navigates the app, they trigger analytics events that are important for understanding the health and performance of our product surfaces. We use UI tests to validate that these events are emitted correctly. We don’t expose the details of these events in the UI, so we use CodableRPC to query the app for all emitted events and validate the results in the test.
How the app manages memory has become a big focus for us over the past 6 months, and we’ve fixed a huge number of memory leaks. To prevent regressions, we’ve implemented some UI tests that exercise common product surfaces to monitor memory growth and detect leaks. We are using CodableRPC to retrieve the memory footprint of the app before and after navigating through a feature to compare the memory change. We also use it to emit signposts from the app, allowing us to easily mark test iterations for memory leak analysis.
At Reddit, we strive to perform as many tests as possible at pre-merge time, as this directly connects a test failure with the cause. However, a common problem teams face when developing UI tests is their long runtime. Our UI test suites have grown to cover all areas of the app, yet that means they can take a significant amount of time to run, far too long for a pre-merge check. We manage this by running a subset of high-priority tests as pre-merge checks, and the remainder on a nightly basis. If we could reduce the runtime of our tests, we could run more of them as pre-merge checks.
One way in which CodableRPC can help reduce runtime is by skipping common UI flows with a programmatic action. For example, if tests need to authenticate before the main steps of the test can execute, an RPC call could be used to perform the authentication programmatically, saving the time it takes to type and tap through the authentication flow. Of course, we recommend you retain one test that performs the full authentication flow without any RPC trickery.
Another aspect of UI testing that leads to long runtimes is the need to re-launch the app, typically once per test. This is a step that’s very hard to optimize, but we can avoid it entirely by using an RPC call to completely tear down the app UI and state and restore it to a clean state. For example, instead of logging out, and relaunching the app to reset state, an RPC call could deallocate the entire view controller stack, reset UserDefaults, remove on-disk files, or any other cleanup actions.
Many apps are not initially developed with the ability to perform such a comprehensive tear-down, as it requires careful coordination between the dependency injection system, view controller state, and internal storage systems. We have a project planned for 2024 to rearchitect how the app handles account switching, which will solve many of the issues currently blocking us from implementing such an RPC call.
We have taken a look at some of the ways that an RPC mechanism can complement your UI tests, and even unlock new testing possibilities. At Reddit, RPC has become a crucial component supporting some of our most important testing investments. We hope you find CodableRPC useful, and that this article has given you some ideas for how you can use RPC to level up your own test suites.
If working on a high-traffic iOS app sounds like something you’re interested in, check out the open positions on our careers site. We’re hiring!
Written by Scott Reisor
I’m Scott and I work in Developer Experience at Reddit. Our teams maintain the libraries and tooling that support many platforms of development: backend, mobile, and web.
The source code for all this development is currently spread across more than 2000 git repositories. Some of these repos are small microservice repos maintained by a single team, while others, like our mobile apps, are larger mono-repos that multiple teams build together. It may sound absurd to have more repositories than we do engineers, but segmenting our code like this comes with some big benefits:
Of course, there are always downsides to any approach. Today I’m going to share some of the ways we wrangle this mass of repos, in particular how we used Sourcegraph to manage the complexity.
To start, it can be a challenge to search for code across 2000+ repos. Our repository host provides some basic search capabilities, but it doesn’t do a great job of surfacing relevant results. If I know where to start looking, I can clone the repo and search it locally with tools like grep (or ripgrep for those of culture). But at Reddit I can also open up Sourcegraph.
Sourcegraph is a tool we host internally that provides an intelligent search for our decentralized code base with powerful regex and filtering support. We have it set up to index code from all our 2000 repositories (plus some public repos we depend on). All of our developers have access to Sourcegraph’s web UI to search and browse our codebase.
As an example, let’s say I’m building a new HTTP backend service and want to inject some middleware to parse custom headers rather than implementing that in each endpoint handler. We have libraries that support these common use cases, and if I look up the middleware package on our internal Godoc service, I can find a Wrap funcion that sounds like what I need to inject middleware. Unfortunately, these docs don’t currently have useful examples on how Wrap is actually used.
I can turn to Sourcegraph to see how other people have used the Wrap function in their latest code. A simple query for middleware.Wrap returns plain text matches across all of Reddit’s code base in milliseconds. This is just a very basic search, but Sourcegraph has an extensive query syntax that allows you to fine-tune results and combine filters in powerful ways.
These first few results are from within our httpbp framework, which is probably a good example of how it’s used. If we click into one of the results, we can read the full context of the usage in an IDE-like file browser.
And by IDE-like, I really mean it. If I hover over symbols in the file, I’ll see tooltips with docs and the ability to jump to other references:
This is super powerful, and allows developers to do a lot of code inspection and discovery without cloning repos locally. The browser is ideal for our mobile developers in particular. When comparing implementations across our iOS and Android platforms, mobile developers don’t need to have both Xcode and Android Studio setup to get IDE-like file browsing, just the tool for the platform they’re actively developing. It’s also amazing when you’re responding to an incident while on-call. Being able to hunt through code like this is a huge help when debugging.
Some of this IDE-like functionality does depend on an additional precise code index to work, which, unfortunately, Soucegraph does not generate automatically. We have CI setup to generate these indexes on some of our larger/more impactful repositories, but it does mean these features aren’t currently available across our entire codebase.
At Reddit scale, we are always working on strategic migrations and maturing our infrastructure. This means we need an accurate picture of what our codebase looks like at any point in time. Sourcegraph aids us here with their Code Insights features, helping us visualize migrations and dependencies, code smells and adoption patterns.
Straight searching can certainly be helpful here. It’s great for designing new API abstractions or checking that you don’t repeat yourself with duplicate libraries. But sometimes you need a higher level overview of how your libraries are put to use. Without all our code available locally, it’s difficult to run custom scripting to get these sorts of usage analytics.
Sourcegraph’s ability to aggregate queries makes it easy to audit where certain libraries are being used. If, say, I want to track the adoption of the v2 version of our httpbp framework, I can query for all repos that import the new package. Here the select:repo aggregation causes a single result to be returned for each repo that matches the query:
This gives me a simple list of all the repos currently referencing the new library, and the result count at the top gives me a quick summary of adoption. Results like this aren’t always best suited for a UI, so my team often runs these kinds of queries with the Sourcegraph CLI which allows us to parse results out of a JSON formatted response.
While these aggregations can be great for a snapshot of the current usage, they really get powerful when leveraged as part of Code Insights. This is a feature of Sourcegraph that lets you build dashboards with graphs that track changes over time. Sourcegraph will take a query and run it against the history of your codebase. For example, that query above looks like this for over the past 12 months, illustrating healthy adoption of the v2 library:
This kind of insight has been hugely beneficial in tracking the success of certain projects. Our Android team has been tracking the adoption of new GraphQL APIs while our Web UI team has been tracking the adoption of our Design System (RPL). Adding new code doesn’t necessarily mean progress if we’re not cleaning up the old code. That’s why we like to track adoption alongside removal where possible. We love to see graphs with Xs like this in our dashboards, representing modernization along with legacy tech-debt cleanup.
Code Insights are just a part of how we track these migrations at Reddit. We have metrics in Grafana and event data in BigQuery that also help track not just source code, but what’s actually running in prod. Unfortunately Sourcegraph doesn’t provide a way to mix these other data sources in its dashboards. It’d be great if we could embed these graphs in our Grafana dashboards or within Confluence documents.
One of the biggest challenges of any multi-repo setup is coordinating updates across the entire codebase. It’s certainly nice as library maintainers to be able to release changes without needing to update everything everywhere all at once, but if not all at once, then when? Our developers enjoy the flexibility to adopt new versions at their own pace, but if old versions languish for too long it can become a support burden on our team.
To help with simple dependency updates, many teams leverage Renovate to automatically open pull requests with new package versions. This is generally pretty great! Most of the time teams get small PRs that don’t require any additional effort on their part, and they can happily keep up with the latest versions of our libraries. Sometimes, however, a breaking API change gets pushed out that requires manual intervention to resolve. This can range anywhere from annoying to a crippling time sink. It’s these situations that we look towards Sourcegraph’s Batch Changes.
Batch Changes allow us to write scripts that run against some (or all) of our repos to make automated changes to code. These changes are defined in a metadata file that sets the spec for how changes are applied and the pull request description that repo owners will see when the change comes in. We currently need to rely on the Sourcegraph CLI to actually run the spec, which will download code and run the script locally. This can take some time to run, but once it’s done we can preview changes in the UI before opening pull requests against the matching repos. The preview gives us a chance to modify and rerun the batch before the changes are in front of repo owners.
The above shows a Batch Change that’s actively in progress. Our Release Infrastructure team has been going through the process of moving deployments off of Spinnaker, our legacy deployment tool. The changeset attempts to convert existing Spinnaker config to instead use our new Drone deployment pipelines. This batch matched over 100 repos and we’ve so far opened 70 pull requests, which we’re able to track with a handy burndown chart.
Sourcegraph can’t coerce our developers into merging these changes, teams are ultimately still responsible for their own codebases, but the burndown gives us a quick overview of how the change is being adopted. Sourcegraph does give us the ability to bulk-add comments on the open pull requests to give repo owners a nudge. If there ends up being some stragglers after the change has been out for a bit, the burndown gives us insight to escalate with those repo owners more directly.
Wrangling 2000+ repos has its challenges, but Sourcegraph has helped to make it way easier for us to manage. Code Search gives all of our developers the power to quickly scour across our entire codebase and browse results in an IDE-like web UI. Code Insights gives our platform teams a high level overview of their strategic migrations. And Batch Changes provide a powerful mechanism to enact these migrations with minimal effort on individual repo owners.
There’s yet more juice for us to squeeze out of Sourcegraph. We look forward to updating our deployment with executors which should allow us to run Batch Changes right from the UI and automate more of our precise code indexing. I also expect my team will also find some good usages for code monitoring in the near future as we deprecate some APIs.
Thanks for reading!
Hello Reddit!
I’m happy to announce the seventeenth episode of the Building Reddit podcast. With the new year, I wanted to catch up with our CTO, Chris Slowe, and find out what is coming up this year. We invited two members of his team to join as well: Tyler Otto, VP of Data Science & Safety, and Matt Snelham, VP of Infrastructure. The conversation touches on a lot of recent changes in infrastructure, safety, and AI at Reddit.
We’re trying this new roundtable format, so I hope you enjoy it! Let me know in the comments.
You can listen on all major podcast platforms: Apple Podcasts, Spotify, Google Podcasts, and more!
Building Reddit Ep. 17: What’s Next for Reddit Tech
From whichever perspective you look at it, Reddit is always evolving and growing. Users post and comment about current events or whatever they’re into lately, and Reddit employees improve infrastructure, fix bugs, and deploy new features. Any one player in this ecosystem would probably have trouble seeing the complete picture.
In this episode, you’ll get a better understanding of the tech side of this equation with this very special roundtable discussion with three of the people best positioned to share where Reddit has been and where it’s going. The roundtable features Reddit’s Chief Technology Officer and Founding Engineer, Chris Slowe, VP of Data Science and Safety, Tyler Otto, and VP of Infrastructure, Matt Snelham.
In this discussion, they’ll share what they’re most proud of at Reddit, how they are keeping users safe against new threats, and what they want to accomplish in 2024.
Check out all the open positions at Reddit on our careers site: https://www.redditinc.com/careers
Written by Aleksandr Plentsov, Alessandro Tiberi, and Daniel Peters.
One of Reddit’s most distinguishing features as a platform is its abundance of rich user-generated content, which creates both significant opportunities and challenges.
On one hand, content safety is a major consideration: users may want to opt out of seeing some content types, and brands may have preferences about what kind of content their ads are shown next to. You can learn more about solving this problem for adult and violent content from our previous blog post.
On the other hand, we can leverage this content to solve one of the most fundamental problems in the realm of advertising: irrelevant ads. Making ads relevant is crucial for both sides of our ecosystem - users prefer seeing ads that are relevant to their interests, and advertisers want ads to be served to audiences that are likely to be interested in their offerings
Relevance can be described as the proximity between an ad and the user intent (what the user wants right now or is interested in in general). Optimizing relevance requires us to understand both. This is where content understanding comes into play - first, we get the meaning of the content (posts and ads), then we can infer user intent from the context - immediate (what content do they interact with right now) and from history (what did the user interact with previously).
It’s worth mentioning that over the years the diversity of content types has increased - videos and images have become more prominent. Nevertheless, we will only focus on the text here. Let’s have a look at the simplified view of the text content understanding pipeline we have in Reddit Ads. In this post, we will discuss some components in more detail.
Ads Content Understanding Pipeline
While we need to understand content, not all content is equally important for advertising purposes. Brands usually want to sell something, and what we need to extract is what kind of advertisable things could be relevant to the content.
One high-level way to categorize content is the IAB context taxonomy standard, widely used in the advertising industry and well understood by the ad community. It provides a hierarchical way to say what some content is about: from “Hobbies & Interests >> Arts and Crafts >> Painting” to “Style & Fashion >> Men's Fashion >> Men's Clothing >> Men's Underwear and Sleepwear.”
IAB can be enough to categorize content broadly, but it is too coarse to be the only signal for some applications, e.g. ensuring ad relevance. We want to understand not only what kinds of discussions people have on Reddit, but what specific companies, brands, and products they talk about.
This is where the Knowledge Graph (KG) comes to the rescue. What exactly is it? A knowledge graph is a graph (collection of nodes and edges) representing entities, their properties, and relationships.
An entity is a thing that is discussed or referenced on Reddit. Entities can be of different types: brands, companies, sports clubs and music bands, people, and many more. For example, Minecraft, California, Harry Potter, and Google are all considered entities.
A relationship is a link between two entities that allows us to generalize and transfer information between entities: for instance, this way we can link Dumbledore and Voldemort to the Harry Potter franchise, which belongs to the Entertainment and Literature categories.
In our case, this graph is maintained by a combination of manual curation, automated suggestions, and powerful tools. You can see an example of a node with its properties and relationships in the diagram below.
Harry Potter KG node and its relationships
The good thing about KG is that it gives us exactly what we need - an inventory of high-precision advertisable content.
The general idea is as follows: take some piece of text and try to find the KG entities that are mentioned inside it. Problems arise upon polysemy. A simple example is “Apple”, which can refer either to the famous brand or a fruit. We train special classification models to disambiguate KG titles and apply them when parsing the text. Training sets are generated based on the idea that we can distinguish between different meanings of a given title variation using the context in which it appears - surrounding words and the overall topic of discussion (hello, IAB categories!).
So, if Apple is mentioned in the discussion of electronics, or together with “iPhone” we can be reasonably confident that the mention is referring to the brand and not to a fruit.
The IAB Taxonomy can be quite handy in some situations - in particular, when a post does not mention any entities explicitly, or when we want to understand if it discusses topics that could be sensitive for user and/or advertiser (e.g. Alcohol). To overcome this we use custom multi-label classifiers to detect the IAB categories of content based on features of the text.
IAB categories and KG entities are quite useful individually, but when combined they provide a full understanding of a post/ad. To synthesize these signals we attribute KG entities to IAB categories based on the relationships of the knowledge graph, including the relationships of the IAB hierarchy. Finally, we also associate categories based on the subreddit of the post or the advertiser of an ad. Integrating together all of these signals gives a full picture of what a post/ad is actually about.
Now that we have annotated text content with the KG entities associated with it, there are several Ads Funnel stages that can benefit from contextual signals. Some of them are retrieval (see the dedicated post), targeting, and CTR prediction.
Let’s take our CTR prediction model as an example for the rest of the post. You can learn more about the task in our previous post, but in general, given the user and the ad we want to predict click probability, and currently we employ a DNN model for this purpose. To introduce KG signals into that model, we use representations of both user and ad in the same embedding space.
First, we train a word2vec-like model on the tagged version of our post corpus. This way we get domain-aware representations for both regular tokens and KG entities as well.
Then we can compute Ad / Post embeddings by pooling embeddings of the KG entities associated with it. One common strategy is to apply tf-idf weighting, which will dampen the importance of the most frequent entities.
The embedding for a given ad A is given by
Embedding formula a given ad (A)
where:
To obtain user representations, we can pool embeddings of the content they recently interacted with: visited posts, clicked ads, etc.
In the described approach, there are multiple hyperparameters to tune: KG embeddings model, post-level pooling, and user-level pooling. While it is possible to tune them by evaluating the downstream applications (CTR model metrics), it proves to be a pretty slow process as we’ll need to compute multiple new sets of features, train and evaluate models.
A crucial optimization we did was introducing the offline framework standardizing the evaluation of user and content embeddings. Its main idea is relatively simple: given user and ad embeddings for some set of ad impressions, you can measure how good the similarity between them is for the prediction of the click events. The upside is that it’s much faster than evaluating the downstream model while proving to be correlated with those metrics.
The last thing we want to cover here is how exactly we use these embeddings in the model. When we first introduced KG signal in the CTR prediction model, we stored precomputed ad/user embeddings in the online feature store and then used these raw embeddings directly as features for the model.
User/Ad Embeddings in the CTR prediction DNN - v1
This approach had a few drawbacks:
Addressing these issues, we switched to another approach where we
User/Ad Embeddings in the CTR prediction DNN - v2
We were able to cover here only some highlights of what has already been done in the Ads Content Understanding. A lot of cool stuff was left overboard: business experience applications, targeting improvements, ensuring brand safety beyond, and so on. So stay tuned!
In the meantime, check out our open roles! We have a few Machine Learning Engineer roles open in our Ads org.
Hey everyone!
We're excited to announce that this week is Snoosweek, our internal hack-a-thon! This means that our team will be taking some time to hack on new ideas, explore projects outside of their usual work, collaborate together with the goal of making Reddit better, and learn new skills in the process.
We'll be back next week with our regularly scheduled programming.
-The r/redditeng team
Written by Jianyi Yi.
Today, Reddit hosts billions of posts containing various forms of media content, including images, videos, gifs, and embedded third-party media. As Reddit continues to evolve into a more media-oriented platform, users are uploading media content at an accelerating pace. This poses the challenge of effectively managing, analyzing, and auditing our rapidly expanding media assets library.
Media metadata provides additional context, organization, and searchability for the media content. There are two main types of media metadata on Reddit. The first type is media data on the post model. For example, when rendering a video post we need the video thumbnails, playback URLs, bitrates, and various resolutions. The second type consists of metadata directly associated with the lifecycle of the media asset itself, such as processing state, encoding information, S3 file location, etc. This article mostly focuses on the first type of media data on the post model.
Metadata example for a cat image
Although media metadata exists within Reddit's database systems, it is distributed across multiple systems, resulting in inconsistent storage formats and varying query patterns for different asset types. For example, media data used for traditional image and video posts is stored alongside other post data, whereas media data related to chats and other types of posts is stored in an entirely different database..
Additionally, we lack proper mechanisms for auditing changes, analyzing content, and categorizing metadata. Currently, retrieving information about a specific asset—such as its existence, size, upload date, access permissions, available transcode artifacts, and encoding properties—requires querying the corresponding S3 bucket. In some cases, this even involves downloading the underlying asset(s), which is impractical and sometimes not feasible, especially when metadata needs to be served in real-time.
The challenges mentioned above have motivated us to create a unified system for managing media metadata within Reddit. Below are the high-level system requirements for our database:
After evaluating several database systems available to Reddit, we opted for AWS Aurora Postgres. The decision came down to choosing between Postgres and Cassandra, both of which can meet our requirements. However, Postgres emerged as the preferred choice for incident response scenarios due to the challenges associated with ad-hoc queries for debugging in Cassandra, and the potential risk of some data not being denormalized and unsearchable.
Here's a simplified overview of our media metadata storage system: we have a service interfacing with the database, handling reads and writes through service-level APIs. After successfully migrating data from our other database systems in 2023, the media metadata store now houses and serves all the media data for all posts on Reddit.
System overview for the media metadata store
While setting up a new Postgres database is straightforward, the real challenge lies in transferring several terabytes of data from one database to another, all while ensuring the system continues to behave correctly with over 100k reads and hundreds of writes per second at the same time.
Imagine the consequences if the new database has the wrong media metadata for many posts. When we transition to the media metadata store as the source of truth, the outcome could be catastrophic!
We handled the migration in the following stages before designating the new metadata store as the source of truth:
There are several scenarios where data differences may arise between the new database and the source:
We addressed this challenge by setting up a Kafka consumer to listen to a stream of data change events from the source database. The consumer then performs data validation with the media metadata store. If any data inconsistencies are detected, the consumer reports the differences to another data table in the database. This allows engineers to query and analyze the data issues.
System overview for data migration
We heavily optimized the media metadata store for reads. At 100k requests per second, the media metadata store achieved an impressive read latency of 2.6 ms at p50, 4.7 ms at p90, and 17 ms at p99. It is generally more available and 50% faster than our previous data system serving the same media metadata. All this is done without needing a read-through cache!
At the current pace of media content creation, we estimate that the size of media metadata will reach roughly 50 TB by the year 2030. To address this scalability challenge, we have implemented table partitioning in Postgres. Below is an example of table partitioning using a partition management extension for Postgres called pg_partman:
SELECT partman.create_parent(
p_parent_table => 'public.media_post_attributes',
p_control => 'post_id', // partition on the post_id column
p_type => 'native', // use postgres’s built-in partition
p_interval => '90000000', // 1 partition for every 90000000 ids
p_premake => 30 // create 30 partitions in advance
);
Then we used a pg_cron scheduler to run the above SQL statements periodically to create new partitions when the number of spare partitions falls below 30.
SELECT cron.schedule('@weekly', $$CALL partman.run_maintenance_proc()$$);
We opted to implement range-based partitioning for the partition key post_id
instead of hash-based partitioning. Given that post_id increases monotonically with time, range-based partitioning allows us to partition the table by distinct time periods. This approach offers several important advantages:
Firstly, most read operations target posts created within a recent time period. This characteristic allows the Postgres engine to cache the indexes of the most recent partitions in its shared buffer pool, thereby minimizing disk I/O. With a small number of hot partitions, the hot working set remains in memory, enhancing query performance.
Secondly, many read requests involve batch queries on multiple post IDs from the same time period. As a result, we are more likely to retrieve all the required data from a single partition rather than multiple partitions, further optimizing query execution.
Another important performance optimization we did is to serve reads from a denormalized JSONB field. Below is an example illustrating all the metadata fields required for displaying an image post on Reddit. It's worth noting that certain fields may vary for different media types such as videos or embedded third-party media content.
By storing all the media metadata fields required to render a post within a serialized JSONB format, we effectively transformed the table into a NoSQL-like key-value pair. This approach allows us to efficiently fetch all the fields together using a single key. Furthermore, it eliminates the need for joins and vastly simplifies the querying logic, especially when the data fields vary across different media types.
We will continue the data migration process on the second type of metadata, which is the metadata associated with the lifecycle of media assets themselves.
We remain committed to enhancing our media infrastructure to meet evolving needs and challenges. Our journey of optimization continues as we strive to further refine and improve the management of media assets and associated metadata.
If this work sounds interesting to you, check out our careers page to see our open roles!
Written by: Sotiris Nanopoulos and Shadi Altarsha
Welcome to our exploration of shutting down applications in Kubernetes. Throughout our discussion, we'll be honing in on the shutdown process of Envoy, shedding light on the hurdles and emphasizing the critical need for a smooth application shutdown running in Kubernetes.
Envoy pods sending/receiving requests to/from upstreams
Contrary to pod creation, where Kubernetes patiently waits for Kubelet to report the new IP address before initiating the propagation of the new endpoint, deleting a pod involves the simultaneous removal of the endpoint and the Kubelet's termination tasks, unfolding in parallel.
This parallel execution introduces a potential for race conditions, where the pod's processes may have completely exited, but the endpoint entry is still in use among various components. This could cause a fair amount of race conditions where the pod’s process could be completely exited but the endpoint entry is being used among the components.
Timeline of the events that occur when a pod gets deleted in Kubernetes
In a perfect world, Kubernetes would gracefully wait for all components subscribing to Endpoint object updates to remove the endpoint entry before proceeding with pod deletion. However, Kubernetes operates differently. Instead, it promptly sends a SIGTERM signal to the pod.
The pod, being mindful of this signal, can handle the shutdown gracefully. This involves actions like waiting longer before closing processes, processing incoming requests, closing existing connections, cleaning up resources (such as databases), and then exiting the process.
By default, Kubernetes waits for 30 seconds (modifiable using terminationGracePeriodSeconds) before issuing a SIGKILL signal, forcing the pod to exit.
Additionally, Kubernetes provides a set of Pod Lifecycle hooks, including the preStop hook. Leveraging this hook allows for executing commands like sleep 15, prompting the process to wait 15 seconds before exiting. Configuring this hook involves details, including its interaction with terminationGracePeriodSeconds, which won't be covered here for brevity."
Envoy handles SIGTERM by shutting down immediately without waiting for connections in flight to terminate or by shutting down the listener first. Instead, it offers an admin “endpoint /healthcheck/fail” which does the following things:
Importantly, calling this endpoint does not:
Envoy expects that there is a discovery service performing a health check on the /ready endpoint. When the health checks start failing the system should eject Envoy from the list of active endpoints thus making the incoming traffic go to zero. After a while, Envoy will have 0 traffic since it communicates with the existing connection holders to go away and the service discovery system ejects it. Then it is safe to shut down with a SIGTERM
A scenario where we have an application deployed in a Kubernetes cluster hosted on AWS. This application serves public internet traffic, with Envoy acting as the ingress, Contour as the Ingress Controller, and an AWS Network Load Balancer (NLB) facilitating external connectivity.
Demonstrating how the public traffic is reaching the application via the NLB & Envoy
As we are trying to scale the Envoy cluster in front of the application to allow more traffic, we noticed that the Envoy deployment wasn’t hitless and our clients started receiving 503 errors which indicates that the backend wasn’t available for their requests. This is the major indicator of a non-proper shutdown process.
A graph that shows how the client is getting 503s because of a non-hitless shutdown
The NLB, AWS target group, and Envoy Architecture
We have the following architecture:
Contour deploys the shutdown manager, as a sidecar container, which is called by k8s a preStop hook and is responsible for blocking shutdown until Envoy has zero active connections. The first thing we were suspicious of was if this program worked as expected. Debugging preStop hooks is challenging because they don’t produce logs unless they fail. So even though Contour logs the number of active connections you can’t find that log line anywhere. To overcome this issue we had to rely on two things:
Using this we can verify that when Envoy shuts down the number of active connections is 0. This is great because Contour is doing the correct thing but not so great because this would have been an easy fix.
For readers who have trust issues, like the authors of this post, there is another way to verify empirically that the shutdown from K8's perspective is hitless. Port-forward the k8s service running Envoy and use a load generator to apply persistent load. While you apply the load kill a pod or two and ensure you get no 5xx responses.
After finishing step 1 we know that the issue must be in the way the NLB is deregistering Envoy from its targets. At this point, we have a pretty clear sign of where the issue is but it is still quite challenging to figure out why the issue is happening. NLBs are great for performance and scaling but as L4 load balancers they have only TCP observability and opinionated defaults.
The first thing that we notice is that our implementation of NLBs by default does TCP health checks on the serving port. This doesn’t work for Envoy. As mentioned in the Background section Envoy does not close the serving port until it receives a SIGTERM and as a result, our NLB is never ejecting Envoy that is shutting down from the healthy nodes in the target group. To fix this we need to change a couple of things:
This fixes the health checks and now Envoy is correctly ejected from the Target group when the prestop hook is executed.
However, even with this change, we continued to see errors in deployment.
When Envoy executes the preStop hook and starts the pod termination process the pod is marked as not ready and k8s ejects it from the Endpoint Object. Because Envoy is deployed as a Nodeport service, Contour sets the ExternalTrafficPolicy to local. This means that if there is not a pod ready on the node, the request fails with either a connection failure or a TCP reset. This was a really hard point to grasp for the authors as it is a bit inconsistent between the traditional k8s networking. Pods that are marked as not ready are generally reachable (you can port-forward to a not-ready pod and send traffic to it fine). But with Kubeproxy-based routing for local external traffic policy that is false.
Because we have a 1-1 mapping between pods and nodes in our setup we can make some assumptions here that can help with this issue. In particular:
The final piece of the puzzle is TCP keep alive and the NLB deregistration delay. While Contour/Envoy waits for active connections to go to 0 there are still idle connections that need to be timed out and also the NLB needs to deregister the target. Both of these can take quite a bit of time (up to 5.5 mins). During this time Envoy might still get the occasional request so we should be waiting during shutdown. Achieving this is not hard but it makes the deployment a bit slower. In particular, we have to:
In summary, the journey highlights that a well-orchestrated shutdown is not just a best practice but a necessity. Understanding how Kubernetes executes these processes is crucial for navigating complexities, preventing errors, and maintaining system integrity, ensuring the stability and reliability of applications in the Kubernetes ecosystem.
Written by Abinodh Thomas, Senior Software Engineer.
Trust in automated testing is a fragile treasure, hard to gain and easy to lose. As developers, the expectation we have when writing automated tests is pretty simple: alert me when there’s a problem, and assure me when all is well. However, this trust is often challenged by the existence of flaky tests– unpredictable tests with inconsistent results.
In a previous post, we delved into the UI Testing Strategy and Tooling here at Reddit and highlighted our journey of integrating automated tests in the app over the past two years. To date, our iOS project boasts over 20,000 unit/snapshot tests and 2500 UI tests. However, as our test suite expanded, so did the prevalence of test flakiness, threatening the integrity of our development process. This blog post will explore our journey towards developing an automated service we call the Flaky Test Quarantine Service (FTQS) designed to tackle flaky tests head-on, ensuring that our test coverage remains reliable and efficient.
What are flaky tests, and why are they bad news?
These key issues can adversely affect test automation frameworks, effectively becoming their Achilles’ heel.
Now that we understand why flaky tests are such bad news, what’s the solution?
The Solution!
Our initial approach was to configure our test runner to retry failing tests up to 3 times. The idea being that legit bugs would cause consistent test failure(s) and alert the PR author. Whereas flaky tests will pass on retry and prevent CI rebuilds. This strategy was effective in immediately improving perceived CI stability. However, it didn't address the core problem - we had many flaky tests, but no way of knowing which ones were flaky and how often.We then attempted to manually disable these flaky tests in the test classes as we received user reports. But with the sheer volume of automated tests in our project, it was evident that this manual approach was neither sustainable nor scalable. So, we embarked on a journey to create an automated service to identify and rectify flaky tests in the project.
In the upcoming sections, I will outline the key milestones that are necessary to bring this automated service to life, and share some insights into how we successfully implemented it in our iOS project. You’ll see a blend of general principles and specific examples, offering a comprehensive guide on how you too can embark on this journey towards more reliable tests in your projects. So, let’s get started!
As flaky tests often don’t directly block developers, it is hard to understand their true impact from word of mouth. For every developer who voices their frustration about flaky tests, there might be nine others who encounter the same issue but don't speak up, particularly if a subsequent test retry yields a successful result. This means that, without proper monitoring, flaky tests can gradually lead to significant challenges we’ve discussed before. Robust observability helps us nip the problem in the bud before it reaches a tipping point of disruption. A centralized Test Metrics Database that keeps track of each test execution makes it easier to gauge how flaky the tests are, especially if there is a significant number of tests in your codebase.
There are some CI systems that automatically logs this kind of data, so you can probably ignore this step if the service you use offers this. However, if it doesn’t, I recommend collecting the following information for each test case:
A small snippet into the Test Metrics Database
This data should be consistently captured and fed into the Test Metrics Database after every test run. In scenarios where multiple projects/platforms share the same database, adding an additional repository field is advisable as well. There are various methods to export this data; one straightforward approach is to write a script that runs this export step once the test run completes in the CI pipeline. For example, on iOS, we can find repository/commit related information using terminal commands or CI environment variables, while other information about each test case can be parsed from the .xcresult file using tools like xcresultparser. Additionally, if you use a service like BrowserStack to run tests using real devices like we do, you can utilize their API to retrieve information about the test run as well.
With our test tracking mechanism in place for each test case, the next step is to sift through this data to pinpoint flaky tests. Now the crucial question becomes: what criteria should we use to classify a test as flaky?
Here are some identification strategies we considered:
Based on the criteria above, we developed SQL queries to extract this information from the Test Metrics Database. These queries also support including a specific timeframe (like the last 3 days) to help filter out any test cases that might have been fixed already.
To further streamline this process, instead of directly querying the Test Metrics Database, we’re considering setting up another database containing the list of flaky tests in the project. A new column can be added in this database to mark test cases as flaky. Automatically updating this database, based on scheduled analysis of the Test Metrics Database can help dynamically track status of each test case by marking or unmarking them as flaky as needed.
At this point, we had access to a list of test cases in the project that are problematic. In other words, we were equipped with a list of actionable items that will not only enhance the quality of test code but also improve the developers’ quality of life once resolved.
In addressing the flakiness of our test cases, we’re guided by two objectives:
Short Term Objective
To achieve the short-term objective, there are a couple of strategies. One approach we adopted at Reddit was to temporarily exclude tests that are marked as flaky from subsequent CI runs. This means that until the issues are resolved, these tests are effectively skipped. Utilizing the bazel build system we use for the iOS project, we manage this by listing the tests which were identified as flaky in the build config file of the UI test targets and mark them to be skipped. A benefit to doing this is ensuring that we do not duplicate efforts for test cases that were acted on already. Additionally, when FTQS commits these changes and raises a pull request, the teams owning these modules and test cases are added as reviewers, notifying them that one or more test cases belonging to a feature they are responsible for is being skipped.
Pull Request created by FTQS that quarantines flaky tests
However, before going further, I do want to emphasize the trade-offs of this short term solution. While it can lead to immediate improvements in CI stability and reduction in infrastructure costs, temporarily disabling tests also means losing some code and test coverage. This could motivate the test owners to prioritize fixes faster, but the coverage gap remains as a consideration. If this approach seems too drastic, other strategies can be considered, such as continuing to run the tests in CI but disregarding its output, increasing the re-run count upon test failure, or even ignoring this objective entirely. Each of these alternative strategies comes with its own drawbacks, so it's crucial to thoroughly assess the number of flaky tests in your project and the extent to which test flakiness is adversely impacting your team's workflow before making a decision.
Long Term Objective
To achieve the long-term objective, we ensure that each flaky test is systematically tracked and addressed by creating JIRA tasks and assigning those tasks to the test owners. At Reddit, our shift-left approach to automation means that the test ownership is delegated to the feature teams. To help the developer debug the test flakiness, the ticket includes information such as details about recent test runs, guidelines for troubleshooting and fixing flakiness, etc.
Jira ticket automatically created by FTQS indicating that a test case is flaky
There can be a number of reasons why tests are flaky, and we might do a deep dive into them in another post, but common themes we have noticed include:
While these are just a few examples, analyzing tests with these considerations in mind can uncover many opportunities for improvement, laying the groundwork for more reliable and robust testing practices.
After taking action to rectify flaky tests, the next crucial step is evaluating the effectiveness of these efforts. If observability around test runs already exists, this becomes pretty easy. In this section, let’s explore some charts and dashboards that help monitor the impact.
Firstly, we need to track the direct impact on the occurrence of flaky tests in the codebase; for that, we can track:
Ideally, as a result of our rectification efforts, we should see a downward trend in these metrics. This can be further improved by analyzing the ratio of flaky test runs to total test runs to get more accurate insights.
Next, we’ll need to figure out the impact on developer productivity. Charting the following information can give us insights into that:
Ideally, as the number of flaky tests reduce, there should be a noticeable decrease in both metrics, reflecting fewer instances of developers needing to rerun CI workflows.
In addition to the metrics above, it is also important to monitor the management of tickets created for fixing flaky tests by setting up these charts:
These charts could provide insights into how test owners are handling the reported flaky tests. FTQS adds a custom label to every Jira ticket it creates, so we were able to visualize this information using a Jira dashboard.
While some impacts like the overall improvement in test code quality and developer productivity might be less quantifiable, they should become evident over time as flaky tests are addressed in the codebase.
At Reddit, in the iOS project, we saw significant improvements in test stability and CI performance. Comparing the 6-month window before and after implementing FTQS, we saw:
P90 successful build time over time
Initially, FTQS was only quarantining flaky unit and snapshot tests, but after extending it to our UI tests recently, we noticed a 9.75% week-over-week improvement in test stability.
Nightly UI Test Pass Rate over Time
The influence of flaky tests varies greatly depending on the specifics of each codebase, so it is crucial to continually refine the queries and strategies used to identify them. The goal is to strike the right balance between maintaining CI/test stability and ensuring timely resolution of these problematic tests.
While FTQS has been proven quite effective here at Reddit, it still remains a reactive solution. We are currently exploring more proactive approaches like running the newly added test cases multiple times in the PR stage in addition to FTQS. This practice aims to identify potential flakiness earlier in the development lifecycle to prevent these issues from affecting other branches once merged.
We’re also currently in the process of developing a Test Orchestration Service. A key feature we’re considering for this service is dynamically determining which tests to exclude from runs, and feed them to the test runner instead of the runner trying to identify flaky tests based on build config files. While this method would be much quicker, we are still exploring ways to ensure that the test owners are promptly notified when any of the tests they own turns out to be flaky.
As we wrap up, it's clear that confronting flaky tests with an automated solution has been a game changer for our development workflow. This initiative has not only reduced the manual overhead, but also significantly improved the stability of our CI/CD pipelines. However, this journey doesn’t end here, we’re excited to further innovate and share our learnings, contributing to a more resilient and robust testing ecosystem.
If this work sounds interesting to you, check out our careers page to see our open roles.
by Alex Caulfield on behalf of the Eng Mentorship Leads
I’m Alex, an engineer working on internal safety tools here at Reddit. I’ve been here for over two years, working remotely and enjoying the collaboration I get within the safety department. To help foster connections outside of my department in a remote world, I worked with other engineers to plan and run a mentorship program pilot within engineering. Now that the pilot is complete, we want to share our process for planning and executing the pilot, and what we’re looking to do next for engineering mentorship at Reddit.
As our engineering teams at Reddit become more distributed, it has become more difficult to find that community and belonging across our different teams and orgs. In different employee groups, like technical guilds for frontend, backend, and mobile engineering, as well as employee resource groups (ERGs), like Wom-Eng, we heard Snoos wanted more opportunities to find other engineers at Reddit with similar domain knowledge to help them with their career development.
In 2023, a few engineers looked to foster our engineering community by connecting Snoos across different organizations who were aligned on certain interests, like learning or teaching Go, Kubernetes, or Jetpack Compose, or part of certain groups within Reddit, such as technical guilds or ERGs. To do this, we developed an engineering mentorship pilot program to encourage relationships between different ICs across the engineering org and help people upskill. The mentorship leads group looked to gather interested engineers, match them based on their stated preferences, and provide resources to help build strong connections between the mentor and mentee matches.
Since this was our first attempt at building a program from the ground up, we wanted to make sure our group of 5 leads (ranging from IC1, Software Engineer, to IC5, Staff Engineer, on our IC career ladder) were able to support all participants in the program. We looped in members of our CTO’s staff to help us format a proposal of what the program would look like, including going over the objectives of the pilot and details of how it would be implemented.
During the pilot proposal, we determined that we would pick 10 mentors and 10 mentees for our initial pilot. This would allow us to be hands-on with each of the pairings to answer questions, confirm the fit, and gather feedback for future iterations of the program. We also determined we would run the pilot for 3 months, giving enough time for mentors and mentees to develop a strong relationship and give us feedback on the format of the program, while allowing us to take those learnings and build it into a larger program going forward.
We took this proposal to our CTO, Chris Slowe, and got feedback and sign-off to move forward, along with ongoing support from him and his team. For this pilot, we specifically targeted ICs who wanted to stay technical so we could ensure that the matches were the right fit for the career growth people wanted to cultivate.
We then sent out an initial survey to gauge interest in the program. To pick the matches, we gathered preferences around:
After receiving around 100 responses and looking at the preferences of the responders, we sent out our initial matches, resulting in 8 pairings that participated in the initial 3 month pilot. The participants included:
During the program, we encouraged our pairings to meet multiple times a month and continued to check in with participants for feedback on what materials we could provide. We provided a document walking through different topics to talk about during the 3 months of the program. These topics included conversation starters, ways to share interests, and questions to help hone in on focus areas for their time working together. As the engineers progressed through the program, we received feedback that providing an explicit goal setting framework would be helpful, and in the future we would like to include training sessions for mentees on goal setting. This would allow the mentor/mentee relationships to have stronger focus areas and improve accountability across their sessions.
Halfway through the pilot, we scheduled a roundtable discussion with all the mentors participating. The dedicated time was intended for the mentors to meet each other and share their experiences working with their mentees. Based on feedback, this was a great space for mentors to share what had been working and support each other as they worked with their mentees. We will continue to develop the role of the mentors and explore areas in which they can be helpful to their mentees. In the future, we want to encourage mentors to think of themselves as coaches when they don’t have direct experience with the mentee’s situation - just asking the right questions or considering how you would do something given your perspective can be helpful.
Overall, we consider the pilot a success. After the conclusion of the pilot, we sent out a survey to gather feedback and find areas we could improve on for the next iteration. From this survey, we learned that:
We are particularly excited about the unanimous feedback from our mentees, as 100% expressed that they felt at ease posing questions to their mentor – questions that they might hesitate to ask their managers. Furthermore, all mentees indicated that their mentor played a pivotal role in boosting their confidence and professional growth.
We believe, and know that Reddit does too, that connecting engineers across the company can only make our engineering org stronger and make us more unified in our mission to bring community and belonging to everyone in the world.
As we begin 2024, we are looking to expand our engineering mentorship program with the lessons from the pilot. With this, we are going to grow our program pool and spend more time providing resources to cultivate the relationships between mentors and mentees. New resources include better goal setting frameworks, mentor training, and new question banks to target growth areas for the mentee.
As the program grows, we hope to continue to foster community and belonging within Reddit’s engineering org by including more members (engineering managers, data scientists, product managers), giving early career engineers opportunities to mentor, and continuing to create a space for engineers to grow in their career.
If being part of the Reddit engineering org sounds exciting to you, please take a look at our openings on our careers page.
Hello Reddit!
I’m happy to announce the sixteenth episode of the Building Reddit podcast. With my work at Reddit, I don’t interact directly with our Machine Learning tech at all, so I’ve built up a lot of curiosity about how we do things here. I was excited to finally learn more and get all my questions answered with this episode!
In this episode I spoke with Reddit’s Senior Manager of ML Content & Platform, Rosa Català. She’s driven the design and development of the Unified Machine Learning Platform at Reddit and focused on an ML tech first approach. She dove into fascinating topics like how to build a platform that is future-proof, where ML tech is going in the future, and what makes Reddit so unique in the ML space.
This is a great episode, so I hope you enjoy it! Let me know in the comments.
You can listen on all major podcast platforms: Apple Podcasts, Spotify, Google Podcasts, and more!
Building Reddit Ep. 16: Unifying All The ML Platforms with Rosa Català
Machine Learning plays a role in most every computer application in use these days. Beneath the shine of generative AI applications, there’s a whole other side to ML that includes the tools and infrastructure that allow it to handle Reddit-scale traffic. Taking something as complex as the machine learning lifecycle and scaling it to tens or hundreds of thousands of requests per second is no easy feat.
Rosa Català is the Senior Director of ML Content & Platform at Reddit. She has driven the design and implementation of a Unified Machine Learning platform that powers everything from feed recommendations to spam detection. In this episode, she explains how the platform was developed at Reddit, how ML is being used to improve Reddit for users, and her vision for where ML is going next.
Check out all the open positions at Reddit on our careers site: https://www.redditinc.com/careers
Written by Alexey Bykov (Senior Software Engineer & Google Developer Expert for Android)
Video has become an important part of our lives and is now commonly integrated into various mobile applications.
Reddit is no exception. We have over 10 different video surfaces:
https://i.redd.it/kc1rrtulenfc1.gif
In this article, I will share practical tips, supported by production data, on how to improve playback from different perspectives and effectively use ExoPlayer
in your Android app.
This article will be beneficial if you are an Android Engineer and familiar with the basics of the ExoPlayer and Media 3.
There are several popular ways to deliver Video on Demand (VOD) from the server to the client.
Binary
The simplest way is to get a binary file, like an mp4
, and play it on the client. It works great and works on all devices.
However, there are drawbacks: For instance, the binary approach doesn't automatically adapt to changes in the network and only provides one bitrate and resolution. This may be not ideal for longer videos, as there may not be enough bandwidth to download the video quickly.
Adaptive
To tackle the bandwidth drawback with binary delivery, there's another way — adaptive protocols, like HLS
developed by Apple and DASH
by MPEG.
Instead of directly getting the video and audio segments, these protocols work by getting a manifest file first. This manifest file has various segments for each bitrate, along with separate tracks for audio and video.
After the manifest file is downloaded, the protocol’s implementation will choose the best video quality based on your device's available bandwidth. It's smart enough to adapt the video quality on the fly, depending on your network's condition. This is especially useful for longer videos.
It’s not perfect, however. For example, to start playing the video in DASH
, it may take at least 3 round trips, which involve fetching the manifest, audio segments, and video segments.
This may increase the chance of a network error.
On the other hand, in HLS
, it may take 4 round trips, including fetching the master manifest, manifest, audio segments, and video segments.
Reddit experience
Historically, we have used DASH
for all video content for Android and HLS
for all video content for Web and iOS. However, about 75% of our video content is less than 45 seconds long.
For short videos, we hypothesize that it is not necessary to be switching bitrate during the playbacks.
To verify our theory, we conducted an experiment where we served certain videos in MP4
format instead of DASH
, with different duration limitations.
We observed that a 45-second
limitation showed the most pragmatic result:
decreased by 5.5%
decreased by 2.5%
increased by 1.7%
Based on these findings, we've made the decision to serve all videos that are under 45 seconds in pure MP4
format. For longer videos, we'll continue to serve them in adaptive streamable format.
The concept of prefetching involves fetching content before it is displayed and showing it from the cache when the user reaches it.However, first we need to implement caching, which may not be straightforward.
Let's review the potential problems we may encounter with this.
ExternalDir isn’t available everywhere
In Android, we have two options for caching: internal cache or external cache. For most apps, using internalDir
is a practical choice, unless you need to cache very large video files. In that case, externalDir
may be a better option.
It's important to note that the system may clean up the internalDir
if your application reaches a certain quota, while the external cache is only cleaned up if your application is deleted (if it's stored under the app folder).
At Reddit, we initially attempted to cache video in the externalDir
, but later switched to the internalDir
to avoid compatibility issues on devices that do not have it, such as OPPO.
SimpleCache may clean other files
If you take a look at the implementation of SimpleCache
, you'll notice that it's not as simple as its name suggests.
So, SimpleCache
could potentially remove other cache files unless there is a specific dedicated folder that may affect other app logic, be careful with this.
By the way, I spent a lot of time studying the implementation, but I missed those lines. Thanks to Maxim Kachinkin for bringing them to my attention.
SimpleCache hits disk on the constructor
We encountered a lot of ANRs (Application Not Responding) while SimpleCache was being created. Diving into the implementation, I realized it was hitting disk in constructor:
So make sure to create this instance on a background thread to avoid this.
URL uses as a cache-key
This is by default. However, if your URL is different due to signing signature or additional parameters, make sure to provide a custom cache key factory for the data source. This will help increase cache-hit and optimize performance.
Eviction should be explicitly enabled
Eviction is a pretty nifty strategy to prevent cached data from piling up and causing trouble. Lots of libraries, like Glide
, actually use it under the hood. If video content is not the main focus of your app, SimpleCache
also allows for easy implementation in just one line:
Prefetching options
Well. You have 5 prefetching options to choose from: DownloadManager, DownloadHelper
, DashUtil
, DashDownloader
, and HlsDownloader
.
In my opinion, the easiest way to accomplish this is by using DownloadManager
. You can integrate it with ExoPlayer
, and it uses the same SimpleCache
instance to work:
It's also really customizable: for instance, it lets you pause, resume, and remove downloads, which can be really handy when users scroll too quickly and ongoing download processes are no longer necessary. It also provides a bunch of options for threading and parallelization.
For prefetching adaptive streams, you can also use DownloadManager
in combination with DownloadHelper that simplifies that job.
Unfortunately, one disadvantage is that there is currently no option to preload a specific amount of video content (e.g., around 500kb), as mentioned in this discussion.
Reddit experience
We tried out different options, including prefetching only the next video, prefetching 2 next videos in parallel or one after the other, and only for short video content (mp4).
After evaluating these prefetching approaches, we discovered that implementing a prefetching feature for only the next video yielded the most practical outcome.
didn’t change
increased by 1.9%
decreased by 0.872%
didn’t change
To further improve our experiment, we want to consider the users’ internet connection strength as a factor for prefetching. We conducted a multi-variant experiment with various bandwidth options, starting from 2 mbps up to 20 mbps.
Unfortunately, this experiment wasn't successful. For example, with a speed of 2 mbps:
decreased by 0.9%
decreased by 1.1%
increased by 3%
In the future, we also plan to experiment with this further and determine if it would be more beneficial to partially prefetch N videos in parallel.
Load control is a mechanism that allows for managing downloads. In simple terms, it addresses the following questions:
And a cool thing is that we can customize this behavior!
bufferForPlaybackMs, default: 2500
Refers to the amount of video content that should be loaded before the first frame is rendered or playback is interrupted by the user (e.g., pause/seek).
bufferForPlaybackAfterRebufferMs, default: 5000
Refers to the amount of data that should be loaded after playback is interrupted due to network changes or bitrate switch
minBuffer & maxBuffer, default: 50000
During playback, ExoPlayer
buffers media data until it reaches maxBufferMs
. It then pauses loading until the buffer decreases to the minBufferMs
, after which it resumes loading.
You may notice that by default, these values are set to the same value. However, in earlier versions of ExoPlayer
, these values were different. Different buffer configuration value could lead to increased rebuffering when the network is unstable.
By setting these values to the same value, the buffer is consistently filled up. (This technique is called Drip-Feeding).
If you want to dig deeper, there are very good articles about buffers:
Reddit experience
Since most of our videos are short, we noticed that the default buffer values were a bit too lengthy. So, we thought it would be a good idea to try out some different values and see how they work for us.
We found that setting bufferForPlaybackMs and bufferForPlaybackAfterRebufferMs = 1 000
, and minBuffer and maxBuffer = 20,000
, gave us the most pragmatic results:
increased by 2.7%
increased by 4.4%
decreased by 11.9%
decreased by 17.7%
decreased by 4.8%
increased by 1.5%
So far this experiment has been one of the most impactful that we ever conducted from all video experiments.
Improving video quality can be challenging because higher quality often leads to slower download speeds, so it’s important to find a proper balance in order to optimize the viewing experience.
To select the appropriate video bitrate and ensure optimal video quality based on the network, ExoPlayer
uses BandwidthMeter
.
It calculates the network bandwidth required for downloading segments and selects appropriate audio and video tracks based on that for subsequent videos.
Reddit experience
At some point, we noticed that although users have good network bandwidth, we don't always serve the best video quality.
The first issue we identified was that prefetching doesn't contribute to overall network bandwidth in BandwidthMeter, as DataSource
in DownloadManager
doesn’t know anything about it. The fix is to include prefetching when considering the overall bandwidth.
And conducted experiment to confirm on production, which yielded the following result:
increased by 1.4%
increased by 0.5%
decreased by 0.5%
increased by 0.3%
(Which is a trade-off)It is worth mentioning that the current BandwidthMeter is still not perfect in calculating the proper video bitrate. In media 1.0.1, an ExperimentalBandwidthMeter has been added, which will eventually replace the old one that should improve the state of things.
Additionally, by default, BandwidthMeter
uses hardcoded values which are different depending on network type and country. It may be not relevant for the current network and in general could be not accurate. For instance, it considers Great Britain 3G faster than 4G.
We haven’t experimented with this yet, but one way to address this would be to remember the latest network bandwidth and setting it up when application starts:
There are also a few customizations available in AdaptiveTrackSelection.Factory
to manage when to switch between better and worse quality: minDurationForQualityIncreaseMs
(default value: 15 000) and minDurationForQualityDecreaseMs
(default value: 25000) that may help with this.
Choosing a bitrate for MP4 Content
If videos are not the primary focus of your application and you only use them, for instance, to showcase updates or year reviews, sticking with an average bitrate may be pragmatic.
At Reddit, when we first transitioned short videos to mp4, we began sending the current bandwidth to receive the next video batch.
However, this solution is not very precise as bandwidth may fluctuate more frequently. We decided to improve it this way:
The main difference between this implementation (second diagram) and adaptive bitrate (DASH/HLS) is that we do not need to prefetch the manifest first (as we obtain it when fetching the video batch), reducing the chances of network errors. Also, the bitrate will remain constant during playback.
When we were experimenting with this approach, we initially relied on approximate bitrates for each video and audio, which was not precise. As a result, the metrics did not move in the right direction:
increased by 9.70%
increased by 12.9%
decreased by 2%
In the future, we will experiment with exact video and audio bitrates, as well as with thresholds, to achieve a good balance between download time and quality.
At some point, we noticed a spike of 4001 playback error, which indicates that the decoder is not available. This problem appeared on almost every android vendor.
Each device has limitations in terms of available decoders and this issue may occur, for instance, when another app has not released the decoder properly.
While we may not be able to mitigate the decoder issue 100%, ExoPlayer
provides an opportunity to switch to a software decoder if a primary one isn't available:
Although this solution is not ideal, as falling back to software decoder can perform slower than hardware decoder, it is better than not being able to play the video. Enabling the fallback option during experimentation resulted in a 0.9% decrease
in playback errors.
To reduce such cases, ExoPlayer
uses the audio manager and can request focus on your behalf. However, you need to explicitly do so:
Another thing that could help is to use only one instance of ExoPlayer
per app. Initially, this may seem like a simple solution. However, if you have videos in feeds, manually managing thumbnails and last frames can be challenging. Additionally, if you want to reuse already initialized decoders, you need to avoid calling stop()
and call prepare()
with new video on top of current playback.
On the other hand, synchronizing multiple instances of ExoPlayer
is also a complex task and may result in audio bleeding issues as well.
At Reddit, we reuse video players when navigating between surfaces. However, when scrolling, we currently create a new instance for each video playback, which adds unnecessary overhead.
We are currently considering two options: a fixed player pool based on the availability of decoders, or using a single instance. Once we conduct the experiment, we will write a new blog post to share our findings.
We have two choices: TextureView
or SurfaceView
. While TextureView
is a regular view that is integrated into the view hierarchy, SurfaceView
has a different rendering mechanism. It draws in a separate window directly to the GPU, while TextureView
renders to the application window and needs to be synchronized with the GPU, which may create overhead in terms of performance and battery consumption.
However, if you have a lot of animations with video, keep in mind that prior to Android N, SurfaceView
had issues in synchronizing animations.
ExoPlayer
also provides default controls (play/pause/seekbar) and allows you to choose where to render video.
Reddit experience
Historically, we’ve been using TextureView
to render videos. However, we are planning to switch to SurfaceView
for better efficiency.
Currently, we are migrating our features to Jetpack Compose
and have created composable wrappers for videos. One issue we face is that, since most of our main feeds are already in Compose, we need to constantly reinflate videos, which can take up to 30ms
according to traces, causing frame drops.
To address this, Jetpack Compose 1.4
introduced a ViewPool
where you need to override callbacks:
However, we decided to implement our own ViewPool
to potentially reuse inflated views across different screens and have more control in the future, like pre-initializing them before displaying the first video:
This implementation resulting in the following benefits:
increased by 1.7%
increased by 0.3%
increased by 1.4%
1ms
, improved x30
24ms
, improved x1.5
Additionally, since default ExoPlayer
controls are implemented by using old-fashioned views, I’d recommend always implementing your own controls to avoid unnecessary inflation.
There are wrappers for SurfaceView
is already available in Jetpack Compose 1.6: AndroidExternalSurface and AndroidEmbeddedExternalSurface.
One of the key things to keep in mind when working with videos is the importance of analytics and regularly conducting A/B testing with various improvements.
This not only helps us identify positive changes, but also enables us to catch any regression issues.
If you just started to working with videos, consider to have at least next events:
ExoPlayer
also provides an AnalyticsListener which can help with that.
Additionally, I must say that working with videos has been quite a challenging experience for me. But hey, don't worry if things don't go exactly as planned for you too — it's completely normal.
In fact, it's meant to be like this.
If working with videos were a song, it would be "Trouble" by Cage the Elephant.
Thanks for reading. If you want to connect and discuss this further, please feel free to DM me on Reddit. Also props to my past colleague Jameson Williams, who had direct contributions to some of the improvements mentioned here.
Thanks to the following folks for helping me review this — Irene Yeh, Merve Karaman, Farkhad Khatamov, Matt Ewing, and Tony Lenzi.
Written by Pratik Lotia (Senior Security Engineer) and Spencer Koch (Principal Security Engineer).
At Reddit, our amazing development teams are routinely building and testing new applications to provide quality feature improvements to our users. Our infrastructure and security teams ensure we provide a stable, reliable and a secure environment to our developers. Several of these applications require the use of a HTTP frontend whether for short term feature testing or longer term infrastructure applications. While we have offices in various parts of the world, we’re a remote-friendly organization with a considerable number of our Snoos working from home. This means that the frontend applications need to be accessible for all Snoos over the public internet while enforcing role-based access control and preventing unauthorized access at the same time. Given we have hundreds of web facing internal-use applications, providing a secure yet convenient, scalable and maintainable method for authenticating and authorizing access to such applications is an integral part of our dev-friendly vision.
Common open-source and COTS software tools often come with a well-tested auth integration which makes supporting authN (authentication) relatively easy. However, supporting access control for internally developed applications can easily become challenging. A common pattern is to let developers implement an auth plugin/library into each of their applications. This comes with the additional overhead of library per language maintenance and OAuth client ID creation/distribution per app, which makes decentralization of auth management unscalable. Furthermore, this impacts developer velocity as adding/troubleshooting access plugins can significantly increase time to develop an application, let alone the overhead for our security teams to verify the new workflows.
Another common pattern is to use per application sidecars where the access control workflows are offloaded to a separate and isolated process. While this enables developers to use well-tested sidecars provided by security teams instead of developing their own, the overhead of compute resources and care/feeding of a fleet of sidecars along with onboarding each sidecar to our SSO provider is still tedious and time consuming. Thus, protecting hundreds of such internal endpoints can easily become a continuous job prone to implementation errors and domino-effect outages for well-meaning changes.
Our current legacy architecture consists of a public ELB backed by a singleton Nginx proxy integrated with the oauth2-proxy plugin using Google Auth. This was setup long before we standardized on using Okta for all authN use cases. At the time of the implementation, supporting AuthZ via Google Groups wasn’t trivial enough due to so we resorted to hardcoding groups of allowed emails per service in our configuration management repository (Puppet). The overhead of onboarding and offboarding such groups was negligible and served us fine as our user base was less than 300 employees.. As we started growing in the last three years, it started impacting developer velocity. We also weren’t upgrading Nginx and oauth2-proxy as diligently as we should. We could have invested in addressing the tech debt, but instead we chose to rearchitect this in a k8s-first world.
In this blog post, we will take a look at how Reddit approached implementing modern access control by exposing internal web applications via a web-proxy with SSO integration. This proxy is a public facing endpoint which uses a cloud provider supported load balancer to route traffic to an internal service which is responsible for performing the access control checks and then routing traffic to the respective application/microservice based on the hostnames.
Envoy Proxy: A proxy service using Envoy proxy acts as a gateway or an entry point for accessing all internal services. Envoy’s native oauth2_filter works as a first line of defense to authX Reddit personnel before any supported services are accessed. It understands Okta claim rules and can be configured to perform authZ validation.
ELB: A public facing ELB orchestrated using k8s service configuration to handle TLS termination using Reddit’s TLS/SSL certificates which will forward all traffic to the Envoy proxy service directly.
Oauth2-proxy: K8s implementation of oauth2-proxy to manage secure communication with OIDC provider (Okta) for handling authentication and authorization. Okta blog post reference.
Snoo: Reddit employees and contingent workers, commonly referred to as ‘clients’ in this blog.
Internal Apps: HTTP applications (both ephemeral and long-lived) used to support both development team’s feature testing applications as well as internal infrastructure tools.
This architecture drew heavily from JP Morgan’s approach (blog post here). A key difference here is that Reddit’s internal applications do not have an external authorization framework, and rely instead on upstream services to provide the authZ validation.
Using a web proxy not only enables us to avoid assignment of a single (and costly) public IP address per endpoint but also significantly reduces our attack surface.
allowed_group
list. The client’s Okta group details are retrieved using the scopes obtained from auth_token (JWT) parameter in the callback from Okta to the oauth2-proxy.Application onboarding: When an app/service owner wants to make an internal service accessible via the proxy, the following steps are taken:
virtualhost
in the Envoy proxy configuration defined as Infrastructure as Code (IaC), though the Envoy config is quite lengthy and may be difficult for developers to grok what changes are required. Note that allowed Okta groups can be defined in this object. This step can be skipped if no group restriction is required.On the Okta side, one needs to add a new Application
of type OpenID Connect
and set the allowed grant types as both Client Credentials
and Authorization Code
. For each upstream, a callback URL is required to be added in the Okta Application configuration. There are plenty of examples on how to set up Okta so we are not going to cover that here. This configuration will generate the following information:
There are several resources on the web such as Tetrate’s blog and Ambassador’s blog which provide a step-by-step guide to setting up Envoy including logging, metrics and other observability aspects. However, they don’t cover the authorization (RBAC) aspect (some do cover the authN part).
Below is a code snippet which includes the authZ configuration. The "@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute
is the important bit here for RBAC which defines allowed Okta groups per upstream application.
node:
id: oauth2_proxy_id
cluster: oauth2_proxy_cluster
static_resources:
listeners:
- name: listener_oauth2
address:
socket_address:
address: 0.0.0.0
port_value: 8888
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: AUTO
stat_prefix: pl_intranet_ng_ingress_http
route_config:
name: local_route
virtual_hosts:
- name: upstream-app1
domains:
- "pl-hello-snoo-service.example.com"
routes:
- match:
prefix: "/"
route:
cluster: upstream-service
typed_per_filter_config:
"envoy.filters.http.rbac":
"@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute
rbac:
rules:
action: ALLOW
policies:
"perroute-authzgrouprules":
permissions:
- any: true
principals:
- metadata:
filter: envoy.filters.http.jwt_authn
path:
- key: payload
- key: groups
value:
list_match:
one_of:
string_match:
exact: pl-okta-auth-group
http_filters:
- name: envoy.filters.http.oauth2
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.oauth2.v3.OAuth2
config:
token_endpoint:
cluster: oauth
uri: "https://<okta domain name>/oauth2/auseeeeeefffffff123/v1/token"
timeout: 5s
authorization_endpoint: "https://<okta domain name>/oauth2/auseeeeeefffffff123/v1/authorize"
redirect_uri: "%REQ(x-forwarded-proto)%://%REQ(:authority)%/callback"
redirect_path_matcher:
path:
exact: /callback
signout_path:
path:
exact: /signout
forward_bearer_token: true
credentials:
client_id: <myClientIdFromOkta>
token_secret:
# these secrets are injected to the Envoy deployment via k8s/vault secret
name: token
sds_config:
path: "/etc/envoy/token-secret.yaml"
hmac_secret:
name: hmac
sds_config:
path: "/etc/envoy/hmac-secret.yaml"
auth_scopes:
- openid
- email
- groups
- name: envoy.filters.http.jwt_authn
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.jwt_authn.v3.JwtAuthentication
providers:
provider1:
payload_in_metadata: payload
from_cookies:
- IdToken
issuer: "https://<okta domain name>/oauth2/auseeeeeefffffff123"
remote_jwks:
http_uri:
uri: "https://<okta domain name>/oauth2/auseeeeeefffffff123/v1/keys"
cluster: oauth
timeout: 10s
cache_duration: 300s
rules:
- match:
prefix: /
requires:
provider_name: provider1
- name: envoy.filters.http.rbac
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC
rules:
action: ALLOW
audit_logging_options:
audit_condition: ON_DENY_AND_ALLOW
policies:
"authzgrouprules":
permissions:
- any: true
principals:
- any: true
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: "/dev/stdout"
typed_json_format:
"@timestamp": "%START_TIME%"
client.address: "%DOWNSTREAM_REMOTE_ADDRESS%"
envoy.route.name: "%ROUTE_NAME%"
envoy.upstream.cluster: "%UPSTREAM_CLUSTER%"
host.hostname: "%HOSTNAME%"
http.request.body.bytes: "%BYTES_RECEIVED%"
http.request.headers.accept: "%REQ(ACCEPT)%"
http.request.headers.authority: "%REQ(:AUTHORITY)%"
http.request.method: "%REQ(:METHOD)%"
service.name: "envoy"
downstreamsan: "%DOWNSTREAM_LOCAL_URI_SAN%"
downstreampeersan: "%DOWNSTREAM_PEER_URI_SAN%"
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
common_tls_context:
tls_certificates:
- certificate_chain: {filename: "/etc/envoy/cert.pem"}
private_key: {filename: "/etc/envoy/key.pem"}
clusters:
- name: upstream-service
connect_timeout: 2s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: upstream-service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: pl-hello-snoo-service
port_value: 4200
- name: oauth
connect_timeout: 2s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: oauth
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: <okta domain name>
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: <okta domain name>
# Envoy does not verify remote certificates by default, uncomment below lines when testing TLS
#common_tls_context:
#validation_context:
#match_subject_alt_names:
#- exact: "*.example.com"
#trusted_ca:
#filename: /etc/ssl/certs/ca-certificates.crt
This initial setup seemed to check most of our boxes. This moved our cumbersome Nginx templated config in Puppet to our new standard of using Envoy proxy but a considerable blast radius still existed as it relied on a single Envoy configuration file which would be routinely updated by developers when adding new upstreams. It provided a k8s path for Developers to ship new internal sites, albeit in a complicated config. We could use Okta as the OAuth2 provider, instead of proxying through Google. It used native integrations (albeit a relatively new one, that at the time of research was still tagged as beta
). We could enforce uniform coverage of oauth_filter on sites by using a dedicated Envoy and linting k8s manifests for the appropriate config.
In this setup, we were packaging the Envoy proxy, a standalone service, to run as a k8s service which has its own ops burden. Because of this, our Infra Transport team wanted to use Contour, an open-source k8s ingress controller for Envoy proxy. This enables adding dynamic updates to the Envoy configuration in cloud native way, such that adding new upstream applications does not require updating the baseline Envoy proxy configuration. Using Contour, adding new upstreams is simply a matter of adding a new k8s CRD object which does not impact other upstreams in the event of any misconfiguration. This ensures that the blast radius is limited. More importantly, Contour’s o11y aspect worked better with reddit’s established o11y practices.
However, Contour lacked support for (1) Envoy’s native Oauth2 integration as well as (2) authZ configuration. This meant we had to add some complexity to our original setup in order to achieve our reliability goals.
Contour Ingress Controller: A ingress controller service which manages Envoy proxy setup using k8s-compatible configuration files
Contour is only a manager/controller
. Under the hood, this setup still uses the Envoy proxy to handle the client traffic. A similar k8s enabled ELB is requested via a LoadBalancer service from Contour. Unlike the raw Envoy proxy which has a native Oauth2 integration, Contour requires setting up and managing an external auth (ExtAuthz) service to verify access requests. Adding native Oauth2 support to Contour is a considerable level of effort. This has been an unresolved issue since 2020.Contour does not support AuthZ and adding this is not on their roadmap yet. Writing these support features and contributing upstream to the Contour project was considered as future work with support from Reddit’s Infrastructure Transport team.
The ExtAuthz service can still use oauth2-proxy to manage auth with Okta via a combination of the Marshal service
and Oauth2-Proxy
forms the ExtAuthz service which in turn communicates with Okta to verify access requests.Unlike the raw Envoy proxy which supports both gRPC and HTTP for communication with ExtAuthz, Contour’s implementation supports only gRPC traffic. Secondly, the Oauth2-Proxy only supports auth requests over HTTP. Adding gRPC support is a high effort task as it would require design-heavy refactoring of the code.Due to the above reasons, we require an intermediary service to translate gRPC traffic to HTTP traffic (and then back). Open source projects such as grpc-gateway allow translating HTTP to gRPC (and then vice versa) but not the other way around.
Due to these reasons, a Marshal service
is used to provide protocol translation service for forwarding traffic from contour to oauth2-proxy. This service:
authorization context
defined in Contour’s HTTPProxy upstream object as the list of Okta groups allowed to access a particular upstream. The auth context parameter will be forwarded as an http header (allowed_groups
) to enable oauth2-proxy to accept. This is a hacky way to do RBAC. The less preferred alternative is to use a k8s configmap to define an allow-list of emails (hard-coded).The oauth2-proxy manages the auth verification tasks by managing the communication with Okta. Based on these verifications, the oauth2-proxy sends either a success or a failure response back to the Marshal service which in turn translates and sends it to the Envoy proxy service.
Application Onboarding: When an app/service owner wants to make a service accessible via the new intranet proxy, the following steps are taken:
As described earlier, the two major concerns with this approach are:
We were faced with implementing, operationalizing and maintaining another service (Marshal service) to perform this. Adding multiple complex workflows and using a hacky method to do RBAC would open the door to implementation vulnerabilities, let alone the overhead of managing multiple services (contour, oauth2-proxy, marshal service). Until the ecosystem matures to a state where gRPC is the norm and Contour adopts some of the features present in Envoy, this pattern isn’t feasible for someone wanting to do authZ (works great for authN though!).
At the same time we were investigating modernizing our proxy, we were also going down the path of zero-trust architecture with Cloudflare for managing Snoo network access based on device and human identities. This presented us with an opportunity to use Cloudflare’s Application concept for managing Snoo access to internal applications as well.
In this design, we continue to leverage our existing internal Nginx ingress architecture in Kubernetes, and eliminate our singleton Nginx performing authN. We can define an Application via Terraform and align the access via Okta groups, and utilizing Cloudflare tunnels we can route that traffic directly to the nginx ingress endpoint. This focuses the authX decisions to Cloudflare with an increased observability angle (seeing how the execution decisions are made).
As mentioned earlier, our apps do not have a core authorization framework. They do understand defined custom HTTP headers to process downstream business logic. In the new world, we leverage the Cloudflare JWT to determine userid and also pass any additional claims that might be handled within the application logic. Any traffic without a valid JWT can be discarded by Nginx ingress via k8s annotations, as seen below.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: intranet-site
annotations:
nginx.com/jwt-key: "<k8s secret with JWT keys loaded from Cloudflare>"
nginx.com/jwt-token: "$http_cf_access_jwt_assertion"
nginx.com/jwt-login-url: "http://403-backend.namespace.svc.cluster.local"
Because we have a specific IngressClass that our intranet sites utilize, we can enforce a Kyverno policy to require these annotations so we don’t inadvertently expose a site, in addition to restricting this ELB from having internet access since all network traffic must pass through the Cloudflare tunnel.
Cloudflare provides overlapping keys as the key is rotated every 6 weeks (or sooner on demand). Utilizing a k8s cronjob and reloader, you can easily update the secret and restart the nginx pods to take the new values.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: cloudflare-jwt-public-key-rotation
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
spec:
restartPolicy: OnFailure
serviceAccountName: <your service account>
containers:
- name: kubectl
image: bitnami/kubectl:<your k8s version>
command:
- "/bin/sh"
- "-c"
- |
CLOUDFLARE_PUBLIC_KEYS_URL=https://<team>.cloudflareaccess.com/cdn-cgi/access/certs
kubectl delete secret cloudflare-jwk || true
kubectl create secret generic cloudflare-jwk --type=nginx.org/jwk \
--from-literal=jwk="`curl $CLOUDFLARE_PUBLIC_KEYS_URL`"
In closing, we wanted to provide the remaining weaknesses based on our threat model of the new architecture. There are two main points we have here:
Thanks for reading this far about our identity aware proxy journey we took at Reddit. There’s a lot of copypasta on the internet and half-baked ways to achieve the outcome of authenticating and authorizing traffic to sites, and we hope this blog post is useful for showing our logic and documenting our trials and tribulations of trying to find a modern solution for IAP. The ecosystem is ever evolving and new features are getting added to open source, and we believe a fundamental way for engineers and developers learning about open source solutions to problems is via word of mouth and blog posts like this one. And finally, our Security team is growing and hiring so check out reddit jobs for openings.
Written by Doug Turnbull
In an earlier post, we shared how Reddit's search relevance team has been working to bring Learning to Rank - ML for search relevance ranking - to optimize Reddit’s post search. We saw in that post some background for LTR, that, indeed, LTR can only be as good as the training data, and how Reddit was gathering our initial training data.
In this post we’ll dive into a different kind of challenge: feature engineering.
In case you missed it, the TL; DR on Learning to Rank (LTR). LTR applies machine learning to relevance ranking. Relevance ranking sorts search results by a scoring function. Given some features x1, x2, … xn we might create a simple, linear scoring function, where we weigh each feature with weights w1, w2, … wn as follows:
S(x1, x2, … xn) = w1*x1 + w2*x2 + … wn*xn
We want to use machine learning to learn optimal weights (w1..wn) for our features x1..xn.
Of course, there are many such “scoring functions” that need not be linear. Including deep learning and gradient boosting forms. But that’s a topic for another day. For now, you can imagine a linear model like the one above.
Today’s topic, though, is feature engineering.
Features is Learning to Rank, tend to come in three flavors:
The first two features come relatively easy with standard ML tooling. You can imagine a classifier or just dumb python code to tell us the facts listed above. The document features presume we’ve indexed those facts about a post. So aside from the overhead of indexing that data, from an ML perspective, it’s not anything new.
Where things get tricky is with query-dependent features. At Reddit, we use Solr. As such, we construct our query-dependent features as Solr queries. For example, to get the BM25 score of a post title, you might imagine a templated query such as:
post_title($keywords)
And, indeed, using Solr’s Learning to Rank plugin, we can ask Solr to score and retrieve sets of features on the top N results.
As snipped, from Solr’s documentation, you can see how we create a set of features, including query-dependent (ie parameterized), query-only, or document only features:
You can get all this from a standard Solr LTR tutorial - such as this great one.
However, what you may not get, are these painful lessons learned while doing feature engineering for Learning to Rank.
As mentioned, many of our features are query dependent. Statistics like BM25 (as we give above in our example).
Unfortunately for us, with BM25 stats, our tiny development samples don’t actually mirror BM25 scores in production. Tiny samples of production won’t be able to compute lexical scores accurately. Why? Because, under the hood, BM25 is fancy version of TF * IDF (term frequency * inverse document frequency). That last stat - IDF - corresponds to 1 / document frequency.
Why does that matter? Think about what happens when you search for “Luke Skywalker” - skywalker occurs rarely - it has a low document frequency and thus high IDF, it's more specific, so it's more important. Luke, however, occurs in many contexts. It's rather generic.
Our tiny sample doesn't actually capture the true “specificity” or “specialness” of a term like “skywalker”. It’s just a set of documents that match a query. In fact, because we’re focused on the queries we want to work with, document frequency might be badly skewed. It might look something like:
This presents quite a tricky problem when experimenting with features we want to put into production!
Luckily, we can make it rank exactly like production if we take one important step: we patch the global term statistics used in the test index’s search engine scoring. BM25, for example, uses the document frequency - how many documents match the term in the corpus relative to the total docCount. We just have to lie to our production Solr and say “actually this terms document frequency is 45 bajillion” and not “5” as you might think.
To do this, we use a Managed Stats Plugin for our development Solr instances. For every query in our training set (the only accurate stats we care about) we can extract stats from production using the terms component or from various function queries.
Getting a response like
Then we can format it into a CSV for our local Solr, keeping this to the side as part of our sample:
Now we can experiment locally with all the features in the world we’d want, and expect scoring that accurately matches prod!
One important lesson learned when developing the model - you should add the lovingly, hand-crafted ranking features in the manually tuned retrieval solution.
In our last article we discussed the importance of negative sampling of our training data. With negative sampling, we take a little training data from obvious non-matches. If you think about this, you’ll realize that what we’ve done is tell the ranking model a little bit about how first-pass retrieval ought to work. This may be counterintuitive - as Learning to Rank reranks the first pass retriever.
But it’s important. If we don’t do this, we can really make a mess of things when we rerank.
The model needs to know to not just arbitrarily shuffle results based on something like a title match. But instead, to compute a ranking score that incorporates important levels of the original retrieval ranking PLUS mild tweaks with these other features.
Another way of thinking about it - the base, rough retrieval ranking still should represent 80% of the “oomph” in the score. The role of LTR is to use many additional features, on a smaller top N, to tie-break documents up and down relative to this rough first pass. LTR is about fine-tuning, not about a complete reshuffling.
Another important lesson learned: many of our features will correlate. Check out this set of features
```
Or, in English, we have three features
We can see that a high post_title_bm25 likely corresponds to a high “post_title_match_any_terms”, etc. As one feature increases, the other likely will. The same would be true if we added phrase matching, or other features for the title. It might also be expected that terms in the title occur in the post body a fair amount, so these would be moderately correlated. Less correlated still, would be perhaps a match of a keyword on a subreddit name, which might be something of a strange, very specific term, like CatCelebrity.
If we loaded our query-document features for every query-document pair into a Pandas dataframe, Pandas provides a convenient function corr to show us how much each feature correlates with every-other feature, giving us a dataframe that looks like:
With a little more Python code, we can average this per row, to get a sense of the overall information gain - average correlation - per feature
Dumping a nice table, showing us which feature has the least to do with the other features:
I want features that BOTH add information (something we haven’t seen yet) AND can give us a positive improvement in our evaluation (NDCG, etc). If I do indeed see a model improvement, I can now tie it back to what features provide the most information to the model.
That's all for now but, with this in mind, and a robust set of features, we can move onto the next step: training a model!
Hello Reddit!
I’m happy to announce the fifteenth episode of the Building Reddit podcast. In this episode I spoke with Reddit’s Chief Information Security Officer, Flee. He joined the company in mid-2023 and shared some amazing insight into how he views Reddit, how he approached entering a new company in the C-Suite, and his 5 (or 6) favorite musical artists of all time.
This is a really fun episode, so I hope you enjoy it! Let us know in the comments.
You can listen on all major podcast platforms: Apple Podcasts, Spotify, Google Podcasts, and more!
Building Reddit Ep. 15: Taking Security into SPACE with Reddit’s CISO Flee
As Reddit has grown over the years, maintaining the security of the company and user’s data has become an increasingly difficult task. The teams that manage this responsibility are spread out across the company, and internal organization has also become much trickier.
Enter Reddit’s new Chief Information Security Officer, Flee. He started at Reddit earlier this year and has already made a significant impact on Reddit’s organization and culture. In this episode, Flee describes the formation of the SPACE organization, shares how he approached entering the company’s c-suite, and reminisces about some early inspirations for his career in tech. He also shares some of his favorite music, programming languages and comic books.
Check out all the open positions at Reddit on our careers site: https://www.redditinc.com/careers
By Doug Turnbull
Reddit’s search relevance team is working to bring machine learning to search. Aka Learning to Rank (LTR).
We’ll be sharing a series of blog articles on our journey. In this first article, we’ll get some background on how Reddit thinks about Learning to Rank, and the training data we use for the problem. In subsequent posts, we’ll discuss our model’s features, and then finally training and evaluating our model.
In normal ML, each prediction depends just on features of that item. Ranking - like search relevance ranking - however, is a bit of a different beast.
Ranking’s goal is to sort a list - each row of which has features - as close as possible to an ideal sort order.
We might have a set of features, corresponding to query-document pairs, like follows:
In this training data our label - called a “grade” in search - corresponds to how the query ought to be sorted (here in descending order). Given this training data, we want to create a ranking function that sorts based on the ideal order using the features
We notice, off the bat, that more term matches in post title and post body correspond to a higher grade, thus we would hope our scoring function would strongly weigh the title term matches:
S(num_title_term_matches, num_body_term_matches, query_length) =
100 * num_title_term_matches + …
There are several ways to learn a ranking function, but in this series, we’ll make pairwise predictions. If we subtract every relevant from irrelevant document, we notice a clear diff - the num_title_term_matches diff is almost always positive. A scoring function that predicts the grade-diff using the feature diffs turns out to be a decent scoring function.
But enough on that for now, more on this in future posts, when we discuss model training.
With that background out of the way, let’s discuss what Reddit’s team has been up to!
Reddit search operates at an extremely high scale.When we build search we consider scalability and performance. Our goal has been to start simple and build up. To prove out LTR, we chose to take the following path
In other words, given the extremely high scale, we focus on practicality, leveraging the data already in our Solr index, but not marrying ourselves too deeply to one way of performing inference.
With some background out of the way, how do we think about training data? And what painful lessons have we learned about our training data?
Like many search teams, we focus primarily on two sources:
Indeed a major question of these early LTR trials was how much we trust our training data sources? How much do they correspond to A/B tests?
Many teams struggle with successful Learning to Rank because of poor training data.
One reason, they often put the ML-modeling cart before the training data horse. Luckily, you can get value from an LTR effort before shipping a single model. Because the training data we show here can also be used to evaluate manual search relevance solutions.
So, as part of building LTR, our search relevance team developed robust offline evaluation methodologies. If improving our manual solutions offline on training data positively correlated with online, A/B metrics, on our conversion / success metrics, then we could trust that training data points in a good direction.
The image below became the team’s mantra early on (search bench is our offline evaluation tool).
To be clear, the 95% time spent at the bottom is indeed the hard work! Search labels come with problems. Human labels don’t have tremendous coverage (as we said 20 results per query). Humans labeling in a lab don’t mirror how human lizard brains work when nobody is looking. Engagement data comes with biases - people only click on what they see. Overcoming these biases, handling unlabeled data, dealing with low confidence data and sparsity, do indeed require tremendous focus.
But solving these problems pay off. They allow the team to ship better experiments, and eventually, train robust models. Hopefully, in the future, Large Language models might help overcome problems in offline evaluation.
Speaking of training data problems, one thing we learned: our training data almost uniformly has some kind of relationship to the query. Even the irrelevant results, in either human or engagement data, might mention the search terms somewhere.
For example, one of our hand labeled queries is Zoolander. (The files are IN the computer!!!)
Here’s two posts that mention zoolander, but represent a relevant / irrelevant result for the query
How do we feel about Zoolander 2?
We named this beautiful kitten Derek Zoolander
One, clearly, about the movie. Even in a movie subreddit. The other about a cat, in a cat subreddit, about a pretty kitty named Derek.
Think about how this might appear in our training data. Something like:
Missing from our training data are obvious cases, such as the following::
In short, if the model just has the first table, it can’t learn that term matches on a query matter. As all the examples have term matches, regardless of the relevance of the result.
We need more negative samples!
To solve this, we sampled other queries labeled results as negative (irrelevant/grade=0) results for this query. We’ll add random documents about butterflies to zoolander, call these irrelevant, and now have a row like the 0 title terms one above.
Of course, this comes with risk - we might, though with very low probability, accidentally give a negative label to a truly relevant result. But this is unlikely given that almost always, a random document plucked from the corpus will be irrelevant to this query.
This turned out to be significant in giving our initial model good training data that subsequently performed well.
With this foundation in place, we’re ready to gather features and train a model. That’ll be discussed in our next post.
Happy searching on Reddit. Look out for more great stuff from our team!
Welcome to 2024, everyone! Thanks for hanging out with us here in r/redditeng. We'll be back next Monday with our usual content. Happy New Year!
Written by Fateeha Amjad
Hey there, I’m Fateeha Amjad and I joined Reddit as the Program Manager in the Enterprise Applications & Engineering Team in September 2022.
Me and Zayn, my adopted Alpaca
Born to a family of medical professionals, I’ve always been the odd one out. From a young age, I was fascinated with Math and ended up majoring in Math and Computer Science in college. From the moment I graduated college to now, my entire career has been at pre-ipo startups, wearing multiple hats as each company has gone through hyper growth phases.
I come from a background in Teaching, IT Management, Product Engineering, System Design and Technical Program Management. Each of my roles shared common elements of managing a project/launch in some way or form. However, a common theme in all of my roles was the love of IT and ensuring that my fellow employees were set up for success.
I’ve gotten to experience multiple roles as a people manager and an IC, and each role has had a significant impact on where I am today. My time at Reddit though has by far been my favorite and I continue to look forward to my future here as a Snoo.
What is Program Management in Corp Tech?
As one of two Program Managers in the Enterprise Apps & Engineering team, our time is split across numerous cross functional programs, often 6-10 programs of various “t-shirt sizes” per quarter. Each program has different goals, business value, stakeholders, delivery dates, and level of effort. Keeping all the above variables in mind, I often use O’Brochta’s Law: “Project Management is about applying common sense with uncommon discipline” on a daily basis. TL;DR: How can I highlight a harmonious environment with different (Stakeholders) talents and resources which are often tied to a specific timeline?
Some programs are year long initiatives, like the launch of a new company wide expense tool; while others might only last a quarter, such as improving our org’s agile methodologies. A good measure of success is having the ability to align on the scope/goals/business value of the program in the very beginning, laying out the roles and responsibilities of all the stakeholders involved (ARCI table, as I like to call it instead of a RASCI table) and mastering the art of communication. Your stakeholders should trust you, be vulnerable to you, and be able to hear you as well, especially when risks are discovered.
The Morning
While most people start their day with a cup of coffee, I start mine with a giant jug of water to jumpstart my day.
Even at Universal, here's me with my ice cold water
Unlike the majority of my team, I’m based out of New York City. As such, my NYC mornings are very quiet and are generally my “focus time” until 11AM - 12PM since the majority of my team/stakeholders are based out of the West Coast region. In my early morning focus time, I attempt to clean my inbox, which is used mostly for external communication with vendors.
Once I feel like it's in a much more manageable state, I review my To-Do list of the day based on items that must be completed today and schedule in nice to complete items as stretch goals for the day. I plan out my daily To-Do list on Mondays based on my status update schedules, priorities, launch dates, and buffers for unplanned work to give myself enough bandwidth for the week. I quickly glance over my calendar to ensure all the meetings are in fact meetings and if anything can be substituted by a quick Slack conversation. For the rest of my meetings, I ensure there is an appropriate agenda and customized meeting notes attached to each invite, and update any open comments/tasks from previous conversations. Each stakeholder has a different style and preference of communication, some requiring more detailed updates than others. A large portion of my notes is ensuring that my stakeholders are able to find the right information in the right location at the right time, whether this is a Confluence Page, weekly Slack Update, Monthly Email update, or a bi-weekly steerco. This often leads to a lot of scheduled Slack pings to stakeholders following on their tasks and actions items.
Another area of focus during my early morning is partnering with my fellow NYC PM to work on PMO methodologies, best practices and templates for our stakeholders to reference. This is also a great time for us to review ideologies we have tested and have mini retros on how to improve items we introduced to our stakeholders. Since we are the first PM hires of our team, we have the opportunity to cultivate how Program Management is run.
Types of Meetings
On a typical day in the middle of a program, my meetings consist of Working Sessions on that particular program, where stakeholders are gathered together to design/build ideas/integrations. During internal status syncs, the team meets in order to discuss the status of a particular program, and goes over the status of each deliverable within the program, along with the agreed upon business value, project blockers, risks and mitigations, and timeline discussions. These meetings are often similar to Steerco Meetings which occur with the executive sponsors, higher management, and all stakeholders where we share high level details about a program status and any associated risks.
My favorite type of meetings are the 1:1s I have with my stakeholders. Based on the stakeholders role and relationship, the meeting cadence varies from twice a week to monthly. This is the time where I build personal connections with my stakeholders and understand their bandwidth and details on what I can take off their plate/workload and how we can collaborate more effectively to hit our targeted level of success or program closure. This is also the time where I ask for direct feedback on how I can improve, what they love/loath about the ongoing program and vice versa.
Using the feedback on how to improve, I have some 1:1s where I am being mentored and working on ways to upsell my skill sets. For example, a big goal of FY’23 was to improve my corporate writing skills and I have spent weekly learning sessions with a Staff Engineer & my Manager working on this and look, here I am now writing to you.
One of the programs that I recently launched was the transition to a new company wide Expense tool. This program touched almost every org in Reddit and required a lot of alignment, cross functional communication, and A LOT of flexibility. Oftentimes, I would refuse to move onward to a different phase of this program until it was clear that every stakeholder was aligned and aware of what decisions were made. Due to a lean team, I spent a lot of hands-on time in the weeds for this program. However, for my other programs I tend to understand the deep layers of the program but use that information to help build more accurate high level summaries, status updates, roadmaps, and timelines for stakeholders/leadership involved. In addition, the ability to understand what is happening in the weeds helps me have meaningful conversations with stakeholders around me and allows me to be more effective in my role.
Today I….
Today, a busy day in Q4. I spent my day in three different program working sessions, two program check-ins, and two 1:1s with my stakeholders. After all my calls are done, I revise the meeting notes for each meeting to ensure that I have highlighted everything discussed and next steps. Once my notes are satisfactory, I work on updating our internal documentation. This is where I update/create Jira tickets based on recent updates from my meetings today, update the Program Page with all the latest program updates, update timeline/trackers/roadmaps, and review risks. I discover a new risk, and use my technical background to create a mitigation plan. I then set up a plan to review the risks with the appropriate audience and decide to utilize an upcoming status sync later this week. Once all my information is up to date, I draft or publish status comms to my stakeholders based on the previously agreed forms of communications. Once everything is sent out, I make sure to send out reminders to stakeholders for any open items that haven’t been closed out.
At this time, it's nearly 6pm and I revisit my to-do list and cross off completed items. The satisfaction of cross-outs on a to-do list gives the biggest confidence boost I need to end my day on a good note.
As I turn off my work laptop, I look forward to the rest of my evening where I attempt to cook something healthy for dinner, go kickboxing & plan my next vacation. Until then …
(Friendsgiving ft Turkey made by us, ok fine, mostly my husband but I helped A LOT)
Hello Reddit!
I’m happy to announce the fourteenth episode of the Building Reddit podcast. In this episode I spoke with Reddit’s Director of the Technical Program Management Office, Rachel O’Brien.
As an engineer, I don’t get to see the inner workings of Reddit’s planning process. I’m usually only privy to the initiatives that my team is tasked with, so I was curious to understand how the projects that all the Reddit teams are working on get organized and stay visible to higher level management. In this interview, Rachel talks about how Reddit plans, how TPMs work with project teams to drive execution, and the tools they use to ensure visibility at the highest levels.
Hope you enjoy it! Let us know in the comments.
You can listen on all major podcast platforms: Apple Podcasts, Spotify, Google Podcasts, and more!
Building Reddit Ep. 14: Scaling Program Management @ Reddit with Rachel O’Brien
Reddit is composed of many teams all working on various projects: everything from the iOS app to advertising, to collectible avatars. Keeping these teams focused and aligned to the core Reddit mission is no easy task.
Meet Rachel O'Brien, the driving force behind Reddit's Technical Program Management Office. She spearheaded the establishment of a centralized TPM function within the company, a new strategic ops & localization team and mission control all to accelerate, scale, and empower teams to advance Reddit’s Mission.
In this enlightening interview, Rachel shares insights into Reddit's planning strategies, the collaborative role of TPMs in project execution, and the powerful tools employed to maintain high-level visibility of projects.
Check out all the open positions at Reddit on our careers site: https://www.redditinc.com/careers
By Laurie Darcey (Senior Engineering Manager) and Eric Kuck (Principal Engineer)
Hello again, u/engblogreader!
Thank you for redditing with us again this year. Get ready to look back at some of the ways Android and iOS development at Reddit has evolved and improved in the past year. We’ll cover architecture, developer experience, and app stability / performance improvements and how we achieved them.
Be forewarned. Like last year, there will be random but accurate stats. There will be graphs that go up, down, and some that do both. In December of 2023, we had 29,826 unit tests on Android. Did you need to know that? We don’t know, but we know you’ll ask us stuff like that in the comments and we are here for it. Hit us up with whatever questions you have about mobile development at Reddit for our engineers to answer as we share some of the progress and learnings in our continued quest to build our users the better mobile experiences they deserve.
This is the State of Mobile Platforms, 2023 Edition!
In our 2022 mobile platform year-in-review, we spoke about adopting a mobile-first posture, coping with hypergrowth in our mobile workforce, how we were introducing a modern tech stack, and how we dramatically improved app stability and performance base stats for both platforms. This year we looked to maintain those gains and shifted focus to fully adopting our new tech stack, validating those choices at scale, and taking full advantage of its benefits. On the developer experience side, we looked to improve the performance and stability of our end-to-end developer experience.
So let’s dig into how we’ve been doing!
Glad you asked, u/engblogreader! Indeed, we introduced an opinionated tech stack last year which we call our “Core Stack”.
Simply put: Our Mobile Core Stack is an opinionated but flexible set of technology choices representing our “golden path” for mobile development at Reddit.
It is a vision of a codebase that is well-modularized and built with modern frameworks, programming languages, and design patterns that we fully invest in to give feature teams the best opportunities to deliver user value effectively for the future.
To get specific about what that means for mobile at the time of this writing:
Alright. Let’s dig into each layer of this stack a bit and see how it’s been going.
Like many companies with established mobile apps, we started in Objective-C and Java. For years, our mobile engineers have had a policy of writing new work in the preferred Kotlin/Swift but not mandating the refactoring of legacy code. This allowed for natural adoption over time, but in the past couple of years, we hit plateaus. Developers who had to venture into legacy code felt increasingly gross (technical term) about it. We also found ourselves wading through critical path legacy code in incident situations more often.
Memes about Endless Migrations
In 2023, it became more strategic to work to build and execute a plan to finish these language migrations for a variety of reasons, such as:
As a result of this year’s purposeful efforts, Android completed their Kotlin migration and iOS made a substantial dent in the reduction in Objective-C code in the codebase as well.
You can only have so many migrations going at once, and it felt good to finish one of the longest ones we’ve had on mobile. The Android guild celebrated this achievement and we followed up the migration by ripping out KAPT across (almost) all feature modules and embracing KSP for build performance; we recommend the same approach to all our friends and loved ones.
You can read more about modern language adoption and its benefits to mobile apps like ours here: Kotlin Developer Stories | Migrate from KAPT to KSP
Now let’s talk about our network stack. Reddit is currently powered by a mix of r2 (our legacy REST service) and a more modern GraphQL infrastructure. This is reflected in our mobile codebases, with app features driven by a mixture of REST and GQL calls. This was not ideal from a testing or code-complexity perspective since we had to support multiple networking flows.
Much like with our language policies, our mobile clients have been GraphQL-first for a while now and migrations were slow without incentives. To scale, Reddit needed to lean in to supporting its modern infra and the mobile clients needed to decouple as downstream dependencies to help. In 2023, Reddit got serious about deliberately cutting mobile away from our legacy REST infrastructure and moving to a federated GraphQL model. As part of Core Stack, there were mandates for mobile feature teams to migrate to GQL within about a year and we are coming up on that deadline and now, at long last, the end of this migration is in sight.
Fully GraphQL Clients are so close!
This journey into GraphQL has not been without challenges for mobile. Like many companies with strong legacy REST experience, our initial GQL implementations were not particularly idiomatic and tended to use REST patterns on top of GQL. As a result, mobile developers struggled with many growing pains and anti-patterns like god fragments. Query bloat became real maintainability and performance problems. Coupled with the fact that our REST services could sometimes be faster, some of these moves ended up being a bit dicey from a performance perspective if you take in only the short term view.
Naturally, we wanted our GQL developer experience to be excellent for developers so they’d want to run towards it. On Android, we have been pretty happily using Apollo, but historically that lacked important features for iOS. It has since improved and this is a good example of where we’ve reassessed our options over time and come to the decision to give it a go on iOS as well. Over time, platform teams have invested in countless quality-of-life improvements for the GraphQL developer experience, breaking up GQL mini-monoliths for better build times, encouraging bespoke fragment usage and introducing other safeguards for GraphQL schema validation.
Having more homogeneous networking also means we have opportunities to improve our caching strategies and suddenly opportunities like network response caching and “offline-mode” type features become much more viable. We started introducing improvements like Apollo normalized caching to both mobile clients late this year. Our mobile engineers plan to share more about the progress of this work on this blog in 2024. Stay tuned!
You can read more RedditEng Blog Deep Dives about our GraphQL Infrastructure here:Migrating Android to GraphQL Federation | Migrating Traffic To New GraphQL Federated Subgraphs | Reddit Keynote at Apollo GraphQL Summit 2022
The end of the year 2023 will go down in the books as the year we finally managed to break up both the Android and iOS app monoliths and federate code ownership effectively across teams in a better modularized architecture. This was a dragon we’ve been trying to slay for years and yet continuously unlocks many benefits from build times to better code ownership, testability and even incident response. You are here for the numbers, we know! Let’s do this.
To give some scale here, mobile modularization efforts involved:
Everyone Successfully Modularized, Living Their Best Lives with Sample Apps
The iOS repo is now composed of 910 modules and developers take advantage of sample/playground apps to keep local developer build times down. Last year, iOS adopted Bazel and this choice continues to pay dividends. The iOS platform team has focused on leveraging more intelligent code organization to tackle build bottlenecks, reduce project boilerplate with conventions and improve caching for build performance gains.
Meanwhile, on Android, Gradle continues to work for our large monorepo with almost 700 modules. We’ve standardized our feature module structure and have dozens of sample apps used by teams for ~1 min. build times. We simplified our build files with our own Reddit Gradle Plugin (RGP) to help reinforce consistency between module types. Less logic in module-specific build files also means developers are less likely to unintentionally introduce issues with eager evaluation or configuration caching. Over time, we’ve added more features like affected module detection.
It’s challenging to quantify build time improvements on such long migrations, especially since we’ve added so many features as we’ve grown and introduced a full testing pyramid on both platforms at the same time. We’ve managed to maintain our gains from last year primarily through parallelization and sharding our tests, and by removing unnecessary work and only building what needs to be built. This is how our builds currently look for the mobile developers:
Build Times Within Reasonable Bounds
While we’ve still got lots of room for improvement on build performance, we’ve seen a lot of local productivity improvements from the following approaches:
One especially noteworthy win this past year was that both mobile platforms landed significant dependency injection improvements. Android completed the 2 year migration from a mixed set of legacy dependency injection solutions to 100% Anvil. Meanwhile, the iOS platform moved to a simpler and compile-time safe system, representing a great advancement in iOS developer experience, performance, and safety as well.
You can read more RedditEng Blog Deep Dives about our dependency injection and modularization efforts here:
Android Modularization | Refactoring Dependency Injection Using Anvil | Anvil Plug-in Talk
Composing Better Experiences: Adopting Modern UI Frameworks
Working our way up the tech stack, we’ve settled on flavors of MVVM for presentation logic and chosen modern, declarative, unidirectional, composable UI frameworks. For Android, the choice is Jetpack Compose which powers about 60% of our app screens these days and on iOS, we use an in-house solution called SliceKit while also continuing to evaluate the maturity of options like SwiftUI. Our design system also leverages these frameworks to best effect.
Investing in modern UI frameworks is paying off for many teams and they are building new features faster and with more concise and readable code. For example, the 2022 Android Recap feature took 44% less code to build with Compose than the 2021 version that used XML layouts. The reliability of directional data flows makes code much easier to maintain and test. For both platforms, entire classes of bugs no longer exist and our crash-free rates are also demonstrably better than they were before we started these efforts.
Some insights we’ve had around productivity with modern UI framework usage:
Mobile UI/UX Progress - Android Compose Adoption
You can read more RedditEng Blog Deep Dives about our UI frameworks here:Evolving Reddit’s Feed Architecture | Adopting Compose @ Reddit | Building Recap with Compose | Reactive UI State with Compose | Introducing SliceKit | Reddit Recap: Building iOS
Remember that guy on Reddit who was counting all the different spinner controls our clients used? Well, we are still big fans of his work but we made his job harder this year and we aren’t sorry.
The Reddit design system that sits atop our tech stack is growing quickly in adoption across the high-value experiences on Android, iOS, and web. By staffing a UI Platform team that can effectively partner with feature teams early, we’ve made a lot of headway in establishing a consistent design. Feature teams get value from having trusted UX components to build better experiences and engineers are now able to focus on delivering the best features instead of building more spinner controls. This approach has also led to better operational processes that have been leveraged to improve accessibility and internationalization support as well as rebranding efforts - investments that used to have much higher friction.
One Design System to Rule Them All
You can read more RedditEng Blog Deep Dives about our design system here:The Design System Story | Android Design System | iOS Design System
All Good, Very Nice, But Does Core Stack Scale?
Last year, we shared a Core Stack adoption timeline where we would rebuild some of our largest features in our modern patterns before we know for sure they’ll work for us. We started by building more modest new features to build confidence across the mobile engineering groups. We did this both by shipping those features to production stably and at higher velocity while also building confidence in the improved developer experience and measuring this sentiment also over time (more on that in a moment).
Here is that Core Stack timeline again. Yes, same one as last year.
This timeline held for 2023. This year we’ve built, rebuilt, and even sunsetted whole features written in the new stack. Adding, updating, and deleting features is easier than it used to be and we are more nimble now that we’ve modularized. Onboarding? Chat? Avatars? Search? Mod tools? Recap? Settings? You name it, it’s probably been rewritten in Core Stack or incoming.
But what about the big F, you ask? Yes, those are also rewritten in Core Stack. That’s right: we’ve finished rebuilding some of the most complex features we are likely to ever build with our Core Stack: the feed experiences. While these projects faced some unique challenges, the modern feed architecture is better modularized from a devx perspective and has shown promising results from a performance perspective with users. For example, the Home feed rewrites on both platforms have racked up double-digit startup performance improvements resulting in TTI improvements around the 400ms range which is most definitely human perceptible improvement and builds on the startup performance improvements of last year. Between feed improvements and other app performance investments like baseline profiles and startup optimizations, we saw further gains in app performance for both platforms.
Perf Improvements from Optimizations like Baseline Profiles and Feed Rewrites
Shipping new feed experiences this year was a major achievement across all engineering teams and it took a village. While there’s been a learning curve on these new technologies, they’ve resulted in higher developer satisfaction and productivity wins we hope to build upon - some of the newer feed projects have been a breeze to spin up. These massive projects put a nice bow on the Core Stack efforts that all mobile engineers have worked on in 2022 and 2023 and set us up for future growth. They also build confidence that we can tackle post detail page redesigns and bring along the full bleed video experience that are also in experimentation now.
But has all this foundational work resulted in a better, more performant and stable experience for our users? Well, let’s see!
We’re happy to say we’ve maintained our overall app stability and startup performance gains we shared last year and improved upon them meaningfully across the mobile apps. It hasn’t been easy to prevent setbacks while rebuilding core product surfaces, but we worked through those challenges together with better protections against stability and performance regressions. We continued to have modest gains across a number of top-level metrics that have floored our families and much wow’d our work besties. You know you’re making headway when your mobile teams start being able to occasionally talk about crash-free rates in “five nines” uptime lingo–kudos especially to iOS on this front.
iOS and Android App Stability and Performance Improvements (2023)
How did we do it? Well, we really invested in a full testing pyramid this past year for Android and iOS. Our Quality Engineering team has helped build out a robust suite of unit tests, e2e tests, integration tests, performance tests, stress tests, and substantially improved test coverage on both platforms. You name a type of test, we probably have it or are in the process of trying to introduce it. Or figure out how to deal with flakiness in the ones we have. You know, the usual growing pains. Our automation and test tooling gets better every year and so does our release confidence.
Last year, we relied on manual QA for most of our testing, which involved executing around 3,000 manual test cases per platform each week. This process was time-consuming and expensive, taking up to 5 days to complete per platform. Automating our regression testing resulted in moving from a 5 day manual test cycle to a 1 day manual cycle with an automated test suite that takes less than 3 hours to run. This transition not only sped up releases but also enhanced the overall quality and reliability of Reddit's platform.
Here is a pretty graph of basic test distribution on Android. We have enough confidence in our testing suite and automation now to reduce manual regression testing a ton.
Another area we made significant gains on the stability front was in how we approach our releases. We continue to release mobile client updates on a weekly cadence and have a weekly on-call retro across platform and release engineering teams to continue to build out operational excellence. We have more mature testing review, sign-off, and staged rollout procedures and have beefed up on-call programs across the company to support production issues more proactively. We also introduced an open beta program (join here!). We’ve seen some great results in stability from these improvements, but there’s still a lot of room for innovation and automation here - stay tuned for future blog posts in this area.
By the beginning of 2023, both platforms introduced some form of staged rollouts and release halt processes. Staged rollouts are implemented slightly differently on each platform, due to Apple and Google requirements, but the gist is that we release to a very small percentage of users and actively monitor the health of the deployment for specific health thresholds before gradually ramping the release to more users. Introducing staged rollouts had a profound impact on our app stability. These days we cancel or hotfix when we see issues impacting a tiny fraction of users rather than letting them affect large numbers of users before they are addressed like we did in the past.
Here’s a neat graph showing how these improvements helped stabilize the app stability metrics.
Mobile Staged Releases Improve App Stability
So, What Do Reddit Developers Think of These Changes?
Half the reason we share a lot of this information on our engineering blog is to give prospective mobile hires a sense of what kind of tech stack and development environment they’d be working with here at Reddit is like. We prefer the radical transparency approach, which we like to think you’ll find is a cultural norm here.
We’ve been measuring developer experience regularly for the mobile clients for more than two years now, and we see some positive trends across many of the areas we’ve invested in, from build times to a modern tech stack, from more reliable release processes to building a better culture of testing and quality.
https://i.redd.it/2kk0csdajb4c1.gif
Developer Survey Results We Got and Addressed with Core Stack/DevEx Efforts
Here’s an example of some key developer sentiment over time, with the Android client focus.
Developer Sentiment On Key DevEx Issues Over Time (Android)
What does this show? We look at this graph and see:
We can fix what we start to measure. Continuous investment in platform teams pays off in developer happiness. We have started to find the right staffing balance to move the needle.
Not only is developer sentiment steadily improving quarter over quarter, we also are serving twice as many developers on each platform as we were when we first started measuring - showing we can improve and scale at the same time. Finally, we are building trust with our developers by delivering consistently better developer experiences over time. Next goals? Aim to get those numbers closer to the 4-5 ranges, especially in build performance.
Our developer stakeholders hold us to a high bar and provide candid feedback about what they want us to focus more on, like build performance. We were pleasantly surprised to see measured developer sentiment around tech debt really start to change when we adopted our core tech stack across all features and sentiment around design change for the better with robust design system offerings, to give some concrete examples.
To wrap things up, here are five lessons we learned (sometimes the hard way) this year:
Some Mobile Platform Insights and Reflections (2023)
We are proud of how much we’ve accomplished this year on the mobile platform teams and are looking forward to what comes next for Mobile @ Reddit.
As always, keep an eye on the Reddit Careers page. We are always looking for great mobile talent to join our feature and platform teams and hopefully we’ve made the case today that while we are a work in progress, we mean business when it comes to next-leveling the mobile app platforms for future innovations and improvements.
Happy New Year!!