With the new year, and since it’s been almost two years since we kicked off the Community, I thought it’d be fun to look back on all of the changes and progress we’ve made as a tech team in that time. I’m following the coattails here of a really fantastic post on the current path and plan on the mobile stack, but want to cast a wider retrospective net (though definitely give that one a look first if you haven’t seen it).

So what’s changed? Let me start with one of my favorite major changes over the last few years that isn’t directly included in any of the posts, but is a consequence of all of the choices and improvements (and a lot more) those posts represent--our graph of availability:

​

Service availability, 2020-2022

To read this, above the “red=bad, green=good” message, we’re graphing our overall service availability for each day in the last three years. Availability can be tricky to measure when looking at a modern service-oriented architecture like Reddit’s stack, but for the sake of this graph, think of “available” as meaning “returned a sensible non-error response in a reasonable time.” On the hierarchy of needs, it’s the bottom of the user-experience pyramid.

With such a measure, we aim for “100% uptime”, but expect that things break, patches don’t always do what you expect, and though you might strive to make systems resilient to, sometimes PEBKAC, so there will be some downtime. The measurement for “some” is often expressed by a total percentage of time up, and in our case our goal is 99.95% availability on any given day. Important to note for this number:

• 0.05% downtime in a day is about 43 seconds, and just shy of 22 min/month
• We score partial credit here: if we have a 20% outage for 10% of our users for 10 minutes, we grade that as 10 min * 10% * 20% = 12 seconds of downtime.
  

Now to the color coding: dark green means “100% available”, our “goal” is at the interface green-to-yellow, and red is, as ever, increasingly bad. Minus one magical day in the wee days of 2020 when the decade was new and the world was optimistic (typical 2020…), we didn’t manage 100% availability until September 2021, and that’s now a common occurrence!

I realized while looking through our post history here that we have a serious lack of content about the deeper infrastructure initiatives that led to these radical improvements. So I hereby commit to more deeper infrastructure posts and hereby voluntell the team to write up more! So instead let me talk about some of the other parts of the stack that have affected this progress.

Still changing after all these years.

I’m particularly proud of these improvements as they have also not come at the expense of overall development velocity. Quite the contrary, this period has seen major overhauls and improvements in the tech stack! These changes represent some fairly massive shifts to the deeper innards of Reddit’s tech stack, and in that time we’ve even changed the internal transport protocol of our services, a rather drastic change moving from Thrift to gRPC (Part 1, 2, and 3), but with a big payoff:

gRPC arrived in 2016. gRPC, by itself, is a functional analog to Thrift and shares many of its design sensibilities. In a short number of years, gRPC has achieved significant inroads into the Cloud-native ecosystem -- at least, in part, due to gRPC natively using HTTP2 as a transport. There is native support for gRPC in a number of service mesh technologies, including Istio and Linkerd.

In fact, changing this protocol is one of the reasons we were able to so drastically improve our resiliency so quickly, taking advantage of a wider ecosystem of tools and a better ability to manage services, from more intelligently handling retries to better load shedding through better traffic inspection.

We’ve made extremely deep changes in the way we construct and serve up lists of things (kind of the core feature of reddit), undertaking several major search, relevance, and ML overhauls. In the last few years we’ve scaled up our content systems from the very humble beginnings of the venerable hot algorithm to being able to build 100 billion recommendations in a day, and then to go down the path of starting to finally build large language models (so hot right now) out of content using SnooBERT. And if all that wasn’t enough, we acquired three small ML startups (Spell, MeaningCloud and SpikeTrap), and then ended the year replacing and rewriting much of the stack in Go!

On the Search front, besides shifting search load to our much more scalable GraphQL implementation, we’ve spend the last few years making continue sustained improvements to both the infrastructure and the relevance of search: improving measurement and soliciting feedback, then using those to improve relevance, improve the user experience and design. With deeper foundational work and additional stack optimizations, we were even able to finally launch one of our most requested features: comment search! Why did this take so long? Well think about it: basically every post has at least one comment, and though text posts can be verbose, comments are almost guaranteed to be. Put simply, it’s more than a factor of 10x more content to index to get comment search working.

Users don’t care about your technology, except…

All of this new technology is well and good, and though I can’t in good conscience say “what’s the point?” (I mean after all this is the damned Technology Blog!), I can ask the nearby question: why this and why now? All of this work aims to provide faster, better results to try to let users dive into whatever they are interested in, or to find what they are looking for in search.

Technology innovation hasn’t stopped at the servers, though. We’ve been making similar strides at the API and in the clients. Laurie and Eric did a much better job at explaining the details in their post a few weeks ago, but I want to pop to the top one of the graphs deep in the post, which is like the client equivalent of the uptime graph:

​

\"Cold Start\" startup time for iOS and Android apps

Users don’t care about your technology choices, but they care about the outcomes of the technology choices.

This, like the availability metric, is all about setting basic expectations for user experience: how long does it take to launch Reddit and have it be responsive on your phone. But, in doing so we’re not just testing the quality of the build locally, we’re testing all of the pieces all the way down the stack to get a fresh session of Reddit going for a given user. To see this level of performance gains in that time, it’s required major overhauls at multiple layers:

• GQL Subgraphs. We mentioned above a shift of search to GraphQL. There have been ongoing broader deeper changes to the APIs our clients use to GraphQL, and we’ve started hitting scaling limits for monolithic use of GraphQL, hence the move here.
• Android Modularization, because speaking of monolithic behavior, even client libraries can naturally clump around ambiguously named modules like, say, “app”
• Slicekit on iOS showing that improved modularization obviously extends to clean standards in the UI.

These changes all share common goals: cleaner code, better organized, and easier to share and organize across a growing team. And, for the users, faster to boot!

Of course, it hasn’t been all rosy. With time, with more “green” our aim is to get ahead of problems, but sometimes you have to declare an emergency. These are easy to call in the middle of a drastic, acute (self-inflicted?) outage, but can be a lot harder for the low-level but sustained, annoying issues. One such set of emergency measures kicked in this year when we kicked off r/fixthevideoplayer and started on a sustained initiative to get the bug count on our web player down and usability up, much as we had on iOS in previous years! With lots of work last year behind our belt, it now remains a key focus to maintain the quality bar and continue to polish the experience.

Zoom Zoom Zoom

Of course, the ‘20s being what they’ve been, I’m especially proud of all of this progress during a time when we had another major change across the tech org: we moved from being a fairly centralized company to one that is pretty fully distributed. Remote work is the norm for Reddit engineering, and I can’t see changing that any time soon. This has required some amount of cultural change--better documentation and deliberately setting aside time to talk and be humans rather than just relying on proximity, as a start. We’ve tried to showcase in this community what this has meant for individuals across the tech org in our recurring Day in the Life series, for TPMs,

Experimentation, iOS and Ads Engineer, everyone’s favorite Anti-Evil Engineers, and some geographical color commentary in from software Engineers Dublin and NYC. As part of this, though, we’ve scaled drastically and had to think a lot about the way we work and even killed a Helpdesk while at it.

Pixel by Pixel

I opened by saying I wanted to do a retrospective of the last couple of years, and though I could figure out some hokey way to incorporate it into this post (“Speaking of fiddling with pixels..!”) let me end on a fun note: the work that went into r/place! Besides trying to one-up ourselves as compared to our original implementation five years ago, one drastic change this time around was that large swathes of the work this time were off the shelf!

I don’t mean to say that we just went and reused the 2017 version. Instead, chunks of that version became the seeds for foundational parts of our technology stack, like the incorporation of the RealTIme Service which superseded our earliest attempts with WebSockets, and drastic observability improvements to allow for load testing (this time) before shipping it to a couple of million pixel droppers…

Point is, it was a lot of fun to use, a lot of fun to build, we have an entire series of posts here about it you want more details! Even an intro and a conclusion if you can believe it.

Onward!

With works of text, “derivative” is often used as an insult, but for this one I’m glad to be able to summarize and represent the work that’s gone on the technology side over the last several years. Since locally it can be difficult to identify that progress is, in fact, being made, it was enjoyable to be able to reflect if only for the sake of this post on how far we’ve come. I look forward to another year of awesome progress that we will do our best to represent here.