What Every Developer Should Know About Site Reliability Engineering

Imagine this... You're a software engineer or ML engineer, toiling away at building the next big API. It's sleek, powerful, designed for speed and scale. But then, reality hits:

• Latency spikes, users rage, threatening your app's reputation.
• Unforeseen traffic surges bring your carefully crafted system to its knees.
• Debugging becomes a black hole, swallowing precious time and resources.
• Metrics scream, but deciphering the chaos behind them feels impossible.

Make your API better with SRE

Site Reliability Engineering (SRE) is an approach popularized by Google and is used to increase the reliability, reduce organizational silos, and meet production standards of services, including APIs. SREs work closely with both the developer and operations teams, aiming to minimize the impact of possible failures throughout the development cycle.

The practice of SRE continuously monitor APIs, respond to disasters and system misconfigurations, plan API capacity, implement updates gradually, and automate processes to increase reliability.

SRE helps in automating systems to operate and strike a balance between speed and reliability, ultimately improving the overall reliability of APIs.

I think developers out there all can agree that Google's SRE books are the best for those getting starting with SRE. Google has published several books on Site Reliability Engineering (SRE), including "Building Secure & Reliable Systems," "The Site Reliability Workbook," and "Site Reliability Engineering." These books cover various aspects of SRE, such as principles, practices, and real-world examples from Google's experiences and case studies from Google's Cloud. It was an eye opener for me when i first read them.

SRE vs DevOps

Site Reliability Engineering (SRE) and DevOps are closely related disciplines that focus on streamlining operations in IT. Both demand discussion, leadership endorsement, and buy-in from the development team for effective implementation.

While DevOps represents a broad set of principles for whole lifecycle collaboration between operations and product development, SRE is a specific job role with a set of practices.

Both SRE and DevOps advocate for blameless postmortems to address service disruptions and learn from them. SRE provides more specific recommendations for refining work practices earlier in the journey, though it requires specific adaptation. DevOps, being more broadly focused, can be harder to define and implement, but its wider scope often meets with less initial resistance.

The by-products of SRE

Getting started with SRE sounds daunting, a big part of it about defining SLIs (Service Level Indicators), SLOs (Service Level Objectives), and error budgets. Think of it like setting the rules for how often your digital world is allowed to hiccup before everyone starts giving you the side-eye. It's a balancing act between innovating like a mad scientist and keeping things smooth like a jazz concert.

After you have these done, you can work on setting up the monitoring and alerting to track them, using various tools like Prometheus, Grafana, Datadog, etc.

Service Level Objectives (SLO)

Service level objectives (SLOs) specify a target level (a percentage) for the reliability of your service. Key to making data-driven decisions about reliability. For example, if SLOs are missed for last month, that is a prompt for allocating more resources to prioritize relevant fixes, improvements, and bug fixes.

Above this threshold, almost all users should be happy with your service (assuming they are otherwise happy with the utility of the service). Below this threshold, users are likely to start complaining or to stop using the service. This threshold is usually <100%.

SLOs usually needs to be discussed and approved by all stakeholders, and there is a process for refining SLO.

Service Level Indicators (SLI)

SLIs are basically the number tagged to an SLO.

Examples

• Number of successful HTTP requests / total HTTP requests (success rate)
• Number of search results that used the entire corpus / total number of search results, including those that degraded gracefully
• Number of “stock check count” requests from product searches that used stock data fresher than 10 minutes / total number of stock check requests
• Number of “good user minutes” according to some extended list of criteria for that metric / total number of user minutes

Error budget policy

Once you have an SLO, you can use the SLO to derive an error budget. In order to use this error budget, you need a policy outlining what to do when your service runs out of budget.

When a service has consumed its entire error budget for a given period of time, and specify what actions to take.

Examples

• The development team gives top priority to bugs relating to reliability issues over the past four weeks.
• The development team focuses exclusively on reliability issues until the system is within SLO. This responsibility comes with high-level approval to push back on external feature requests and mandates.
• To reduce the risk of more outages, a production freeze halts certain changes to the system until there is sufficient error budget to resume changes.
• if a downtime is within the scale of error budget, SRE can approve it. Else, changes will be rolled back.

Every company does SRE differently

1. Netflix

   • Netflix uses an operations model called “you build it, you run it.”, aka service ownership model. The software engineers who write the services are also responsible for operating those services. This means they must be full-cycle developers, capable of doing both the development and operations work. Things still occasionally go sideways and incidents happen that impact the customer experience.
   • The CORE team, consisting of Site Reliability Engineers, Applied Resilience Engineers, and Performance Engineers, come in and configures, maintains, and responds to alerts that monitor high-level business KPIs (stream starts per second, for instance).

2. Google

   • SRE and product development are separate organizations. Each group has its own focus, priorities, and management, and does not have to do the bidding of the other.

3. Evernote

   • The organization previously operated under a siloed "ops/dev split" structure. The operations team guarded the production environment, while the development team built new features for customers. This often led to friction, with developers feeling restricted by operational needs and operations frustrated by code-induced issues in production.
   • Two models were considered to bridge this gap: "You wrote it, you run it" (where developers managed their code in production) and "You wrote it, we run it for you" (where operations took over post-development). Ultimately, the organization adopted an SLO-centric SRE approach.
   • This model appealed to them because it acknowledged the inherent differences between operations and development, while fostering collaboration towards a shared objective. It avoided forcing either side to become the other, fostering mutual respect and understanding.
   • Their SLO journey began by defining a clear goal: ensuring smooth user access and content syncing across clients. Their first SLO focused on uptime, aiming for 99.95% availability over a month.

4. HomeDepot

   • The Home Depot implemented a new reliability framework using the acronym VALET to:
     • Volume (traffic): Assess how much user traffic their systems could handle.
     • Availability: Ensure services were operational when needed by users.
     • Latency: Guarantee fast response times for user actions.
     • Errors: Minimize errors encountered by users during service interactions.
     • Tickets: Reduce the need for manual intervention in completing user requests.

   • 

An Analogy for SRE

Think of it like understanding your budget. Sure, you can spend freely without planning, but wouldn't it be smarter to track income and expenses? SRE works similarly. They define Service Level Objectives (SLOs) – like your spending limits – outlining acceptable performance levels for your code. By understanding these limits, you can write Service Level Indicators (SLIs) – your income trackers – to monitor key metrics and proactively identify potential issues. It's not about building an invincible code fortress, but a resilient one, with an error budget built-in for occasional hiccups.

SRE might seem like a foreign language at first, filled with metrics and monitoring tools. But here's the key: understanding their perspective creates a powerful team. You bridge the gap between coding in isolation and collaborating with the guardians of the digital world. You learn their SLIs and SLOs, anticipate their concerns, and work together to build systems that are reliable, efficient, and ultimately, exceptional.