After an STPA talk at SREcon I put the unsafe-control-action questions into the rules my AI coding agent reads (not provided, wrong context, too early or late or out of order, stopped too soon or applied too long). A first replay looked great, so I ran it properly.

Setup: three real past changes, replayed from the commit before each one. A controller taking over a value another controller wrote, automatic recovery for a worker that stops responding, and a dispatcher picking from a cache that can be stale. Three arms: no rule, a plain blast-radius checklist, the STPA rule. Two models, three runs each, 54 plans. Two blind judges graded each plan against the bugs the real review or production found, with method words stripped first. They agreed on about 90% of calls.

Results, real bugs found per plan out of 8: no rule 5.7, plain checklist 6.1, STPA 6.1.

STPA won clearly in one case, the stuck-worker recovery. For one model, only the STPA rule raised "is this timeout too long" and "stop waiting when it can't help". The checklist never did. In the stale-cache case STPA made the other model drop basics it found without any rule: visible check errors, a metric for the fix, a test that proves it.

And my first headline (one question going 1 of 6 to 5 of 6) didn't survive stricter blinding. The plain checklist reached the same score.

What I kept: the timing questions, next to a plain checklist. Three changes is not a study. Curious whether people who run STPA properly, with losses and hazards first, see the same thing.

Write-up with the charts and the exact rule: https://www.catchkill9.dev/posts/stpa-in-my-agent.html

When reviewing system design RFCs, static diagrams (draw.io, Excalidraw) treat a 1,200 RPS Redis cluster and a 200 RPS payment worker as identical boxes, hiding where fan-in queues actually saturate first.

Here is an open-source (MIT) browser-based queuing physics & chaos sandbox for testing these topologies interactively (no signup required):

Interactive Sandbox (Stripe Ledger Topology):

https://graph-flow-virid.vercel.app/?template=stripe-webhook-ledger

Live Netflix Chaos Edge Stream Topology:

https://graph-flow-virid.vercel.app/?template=netflix-chaos-streaming

GitHub Repository (MIT):

https://github.com/Hanubaki/GraphFlow

At 4.5x ingress load, queue utilization crosses 100% on the downstream synchronous database/worker hops first, causing p99 tail latency to spike non-linearly until upstream circuit breakers (OPEN -> HALF-OPEN -> CLOSED) isolate the fault. You can also paste your own docker-compose.yml, main.tf, or package.json to reverse-engineer your stack and export to Mermaid.js, Grafana k6 load scripts, Kubernetes HPA, or AWS Terraform.

Does this M/M/1 queuing degradation curve match how you model capacity planning in production, or do you rely purely on k6/Locust staging runs?

Our client is a leading technology company operating a large-scale digital ecosystem with a wide range of online services. The company develops and operates highly scalable technology platforms serving millions of users worldwide. With a strong focus on technology, innovation, and customer experience, the company continues to expand its digital services and invest in large-scale infrastructure and engineering capabilities.

About the Team
The Search Engineering team is responsible for developing and operating large-scale search, discovery, and navigation platforms.
The team builds highly scalable, reliable, and high-performance distributed systems that support multiple business domains and large volumes of traffic and data.
The Search Platform team is responsible for the reliability and operation of mission-critical search services. Engineers work closely with software developers, product managers, and QA engineers to continuously improve system performance, stability, and operational efficiency.

About the Position
We are looking for a highly motivated Senior Site Reliability Engineer to join the Search Engineering team.
In this role, you will be responsible for maintaining and improving the reliability of large-scale distributed systems. You will work across infrastructure, platform, and application layers, while contributing to automation, monitoring, performance optimization, and incident management.

This position is ideal for an experienced SRE or Infrastructure Engineer who enjoys solving complex technical problems and working with large-scale production environments.

Responsibilities

Your responsibilities will include, but are not limited to:

Participate in the design and deployment of production environments for new services, from infrastructure to service level

Monitor production systems and participate in on-call operations

Respond quickly to production incidents, including reporting, triage, troubleshooting, and resolution

Investigate system issues and identify root causes

Support and execute production releases, including occasional nighttime or out-of-hours operations

Improve system reliability, availability, scalability, and performance

Drive continuous improvement of operational tools and automation

Develop and maintain automation for repetitive operational tasks

Collaborate with software engineers, product managers, and QA engineers

Identify testing requirements and design appropriate testing strategies

Participate in performance testing and optimization

Support security testing and reliability exercises

Participate in disaster recovery and emergency response drills

Contribute to the improvement of engineering and operational processes

Mandatory Qualifications

- More than 8 years of work experience working in IT
- Excellent problem solving and troubleshooting skills
- Excellent teamwork and communication skills
- Strong Linux experience with understanding of system performance and reliability
- Experience analyzing and tuning performance of distributed systems
- Experience deploying and using configuration management tools (e.g., Chef, Ansible)
- Experience with observability tools (e.g., Prometheus, Grafana, Loki)
- Experience with container technologies and orchestration (Kubernetes)
- Experience automating tasks using shell scripts and/or Python
- Strong understanding of computer networking and common protocols
- Experience managing services built with Java

Desired Qualifications

- Experience automating processes for software testing and deployment (Jenkins)
- Experience working with Spark, Solr, Cassandra, or Kafka
- Experience with cloud storage solutions (Ceph, S3, MinIO)
- Experience working with a Git-based workflow
- Software development experience

Technical Environment

You may work with technologies including:

Operating Systems: Linux

Container / Orchestration: Docker, Kubernetes

Configuration Management: Chef, Ansible

Monitoring / Observability: Prometheus, Grafana, Loki

Programming / Scripting: Java, Python, Shell

CI/CD: Jenkins

Data / Distributed Systems: Spark, Solr, Cassandra, Kafka

Storage: Ceph, Amazon S3, MinIO

Version Control: Git

Infrastructure: Large-scale distributed and cloud-based environments

What We Are Looking For

We are looking for someone who:
Enjoys solving complex infrastructure and production issues
Has a strong sense of ownership for system reliability
Can investigate problems independently and identify root causes
Is comfortable working with large-scale distributed systems
Enjoys automation and continuous improvement
Can collaborate effectively with engineers and cross-functional teams
Is willing to participate in on-call and production support activities
Has a strong interest in improving system performance, stability, and scalability

Languages
English: Fluent
Japanese: Optional / a plus

Work Environment
Fast-paced, dynamic global environment with collaborative teams across multiple locations

Salary: ¥10M – ¥12M JPY per year (Senior level)
Location: Hybrid (4 days in the office, 1 day remote)
Office Location: Tokyo, Japan
Working Hours: Flexible schedule with core hours from 11:00 AM to 3:00 PM
Visa Sponsorship: Available
Language Requirement: English only

Apply now or contact us for further information:
[Aleksey.kim@tg-hr.com](mailto:Aleksey.kim@tg-hr.com)

We do regular pre-release load tests in staging, but every damn time we sign off on a release we're still hitting unexpected issues in production. It's become clear that our staging infrastructure is missing the chaos of realistic user behavior and browser-level interactions. I suspect the problem is due to lack of geographic distribution in our test traffic. It's giving a false sense of security, or it was until this happened for the fifth time in a row. Now I don't trust any of it. We know that mimicking real production-scale traffic is hard to do without identical hardware, especially trying to account for third-party dependencies. But I suspect there are obvious blind spots at play here. I'd appreciate some insight from people smarter than me outside my team.

If you've been able to get your staging load tests to correlate reliably with production, what are some things you learned along the way that you were initially failing to reproduce?

If you run services, you already know the answers to most agent reliability problems. An agent that calls tools is a distributed system with a very unpredictable client, and it fails in familiar ways: a call hangs, a retry fires twice, a run never ends, nobody can tell what happened afterwards.

There’s a workshop on Saturday, Oct 31, from 9:30 AM to 1:30 PM ET where Bruno Gonçalves builds an AI data analyst with those habits built in.

What you build:

• Timeouts, retries, re planning and execution budgets around every run
• Structured traces, so every run ends in a defined state
• Permissioned tool access, so risky calls are stopped before they reach the server
• A fault injected test suite you can adapt into a CI check

Everything runs on a laptop with a mock model, so no GPU or API key is needed.
|
Full details and the agenda are here.

Our cloud priority list had 340 criticals on it last month and every one traced back to the same base image, which i reported upward before i understood what i was looking at.

Two hundred accounts, roughly 340 instances built off the same AMI, so one mistake gets replicated at whatever scale your estate is at and it looks like a catastrophe instead of an afternoons work.

Sort by severity and you get the same row 340 times. I tried deduplicating on title which fell apart the moment two titles differed by a word, grouping by account which lost the real total, and suppressing duplicates which took some real ones with them.

Whats actually missing is a parent. The unit of work here is the image, not the instance, and no console i have seen wants to give me that.

we have all 200 accounts in axonius next to the scanner output, and it shows the same 340 rows until somebody tells it to group on the image, which is where im at now.

The scanner isnt wrong either, the estate really does have 340 instances with that misconfiguration, its the unit thats broken not the tool.

Fixing the image was one change. The 340 findings stayed, because the instances were already built and running by then.

How are you parenting cloud findings so one mistake reads as one thing.

Can someone please suggest a youtube channel to learn SRE, at a very high level.

need this book review and pdf if u guys ever find it pls i tried with all websites but no luck

Site Reliability Engineering, 2nd Edition

Disclosure: I work at Icinga this is meant to both be a discussion and half-an-add for a free webinar we are hosting. 
I asked ahead if that was okay, and the mods approved this post.

Tool discussions tend to focus on setup: time to the first useful dashboard, how painful the rollout is. The recurring cost gets less attention, even though it’s the part you keep paying.

This upkeep looks different depending on the approach. With classic checks it’s threshold drift, flapping services, check tuning, and plugins nobody remembers writing.
With metrics and traces it’s cardinality, sampling policy, retention vs. budget, and the collector pipeline turning into its own service you have to run.

I'm trying to understand the realities of others better, both for building better tools and also for educational content like the webinar we're hosting soon.

So my questions for the thread:

• Do you track this time at all, or is it invisible toil?
• Is it planned work with an owner, or whatever is left after incidents?
• If you’ve run both kinds of setups, which one cost more hours per month at your scale?

Rough numbers are welcome, even “about a day a week for one person.”

Speaking of - and this is the half-an-ad part- that recurring cost is a big part of a free webinar we’re hosting on Oct 27 at 3 PM CET (14:00 UTC / 10 AM EDT).
Kris Buytaert from Percona and Blerim Sheqa from Icinga look at what each approach takes to run, and how that changes between 50 and 5,000 hosts.
Have a look and register, if you want to: Running Monitoring and Observability at Scale | Webinar

I’ve been tasked with testing k8sGPT. Was wondering if anyone else has used it and what were your likes/dislikes about it. We’re mostly Azure, but have a small footprint with on prem Nutanix NKP. Sidebar - I’ve also been playing with Azure SRE Agent as well. So not sure how the two tools can/should play along with each other.

I am currently working as a Performance Engineer. Even though the work I do is closer to performance engineering, the role is still under the umbrella of QA.
A lot of the work I enjoy involves building infrastructure tooling, observability stacks, and benchmarking frameworks. However, I feel like my experience is still being grouped with QA/testing, which makes it difficult to communicate the kind of engineering work I actually do.
I’m curious about how people in SRE and Platform Engineering view performance engineering/testing backgrounds. How much overlap do you see between these fields, and how is performance engineering experience generally perceived when hiring for infrastructure-focused roles?
I’d like to hear from people who have worked across these areas or have been involved in hiring for such roles.

disclosure: i'm thinking about building something in this area. no link, no name, i just want to know what people do.

our handoff is basically a email/slack message. then the next person finds out at 3am what "nothing major" meant.

the alerts are all sitting in alertmanager/grafana anyway. but nobody wants to dig through a week of them on a monday morning, so it doesn't happen. and the rotation lives in someone's head. who's been on most, who covered the long weekend, who's owed a swap... nobody really knows.

grafana oncall oss got archived in march, so the free self-hosted option is kind of frozen now too.

for teams around 5-30 engineers: what are you actually using for schedules and paging? and do you do any real handoff, or is it vibes?

Hey all, I wanted to share this blog post I wrote about inspecting AWS Network Load Balancer and the quirk that comes up from them not hanging up TLS

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

• no AI generated comments: please write them on your own!
• no commercial advertisements (this includes commercial products with a FOSS version)
• these must be related to SRE or of heavy interest to SREs

We broke prod routing with a rushed terraform change, rds snapshots and s3 were perfect but cloud recovery of the known good configuration was guesswork since half the estate is drifted, would love any tips from folks who can reliably recover infra, not just data.

I have 3 years experience as a SRE + Observability engineer. Recently I left my job and started searching got 3 offers one Datadog admin - setting up observability kinda role with 50% hike, Incident handling + Reliability engineer with 100% hike, and TSE with 150% hike. I’m deciding to go with the TSE role because of the salary and benefits. Is this a shitty career move? People are telling me it should be the other way around, I kinda agree, don’t know what to do. I can’t be jobless and search any longer as well.

I know I can probably "just use AI". But I want to avoid not invented here syndrome and use established methods and tools where possible.

The first part I am looking for is a well established format for storing architectural information in the repo with the code.

Next would be any tools for helping to generate it.

Lastly, we have mermaid for the actual diagrams. Any good tools for generating the diagrams?

We have multiple services in multiple repos. We use mainly gitlab,

How do help devs be aware of team wide guidelines, and keep their AI in the know as well. Like I could write a doc on the way we name alerts or write runbooks. But it would be in one repo. I need to make it visible to them and their AI.

So far my best idea is a repo with the information, and then checking in an edit to the Claude.md in every repo telling it to go look in the info repo for the guideline info and let the user know what isn't lineing up. That seems like that idea is probably full of holes though. So what do the teams y'all work with do?

As an SRE, what reports or monthly summaries do you provide to your manager, Head of Department, or CTO?

I’m trying to establish a good monthly SRE reporting process and would like to understand what other teams report.

EDIT:

It’s a tech company with a some internal and external solutions

Anthropic released the "On-call" slack agent that should be an ultimate killer of all "Agentic SRE" family products.
Did anybody use it? How it compares to existing platforms?

hey yall,

im a lead SRE at a global fortune 500 that you have heard of haha - i'm a bit of a lurker here

but just wanted to talk about runbooks and documentation for SOPs.

my feeling is that confluence docs kinda suck and runbooks are kind of a mess, our operators are jumping between the docs and their shell, the docs are often times missing a bunch of detail, they are brittle, poorly maintained, half the time the details are hidden in tribal knowledge and when dealing with the pressure of an active inc we notice the pain even more. extracting critical details & commands out of an inc can also be painful and doesnt always translate to improved operational readiness next time.

i admit we're not very mature.. but was curious if its only me feeling like this?

what are you guys doing with runbooks to solve these issues?

When planning a cloud migration (like AWS to Azure), most discovery work starts with a service mapping matrix:

• SQS ➔ Azure Service Bus
• DynamoDB ➔ Cosmos DB
• S3 ➔ Azure Blob Storage
• IAM ➔ Entra ID

That mapping is straightforward. The expensive failures happen when the target platform fails to preserve a subtle behavioral contract the application code quietly came to rely on over years.

A concrete example: SQS Visibility Timeout

Consider a standard worker loop:

Python

message = sqs.receive_message(
    QueueUrl=queue_url,
    VisibilityTimeout=300
)

process(message)

sqs.delete_message(
    QueueUrl=queue_url,
    ReceiptHandle=message["ReceiptHandle"]
)

At first glance, this is standard: receive, process, delete.

But look at the failure-recovery path. If the worker crashes mid-processing before deletion, the code relies on the SQS visibility timeout expiring so another worker can automatically pick up the message. SQS isn't just acting as a transport queue here—it is an active component of the application's failure-recovery design.

When migrating this workload to Azure Service Bus, the question isn't whether Service Bus has queues (it obviously does). The real question is: Does the target setup preserve those exact assumptions around lock duration, settlement, retries, and dead-lettering under failure?

Infrastructure Dependencies vs. Semantic Dependencies

Traditional discovery tools pick up: payment-worker ➔ SQS

That’s an infrastructure dependency. But what the application code actually cares about is the semantic dependency:

Plaintext

receive message
      ↓
process message
      ↓
delete after successful processing
      ↓
if processing fails before deletion,
rely on provider to make message available again

Service inventory tools show you what cloud products are used, but they can't tell you what behavioral assumptions are baked into execution paths.

This pattern shows up everywhere:

• DynamoDB ➔ Cosmos DB: Codebases relying on conditional write semantics, transaction boundaries, or specific read-after-write consistency assumptions.
• S3 ➔ Blob Storage: Workflows built around multipart upload timing, pre-signed URLs, or object visibility state.
• IAM ➔ Entra ID: Hidden assumptions around temporary credential lifespans, workload identity propagation, and role assumption paths.

When these surface late during integration testing or cutover, it forces architectural redesigns, derails timelines, and drags senior engineers into reactive war rooms.

Discussion

I'm currently putting together a semantic-risk checklist for pre-migration planning (and exploring static analysis rules to scan codebases for these execution patterns before cutover).

For anyone who has managed a major cloud migration or replatforming: What was the one hidden application dependency or behavioral difference that broke in staging or production after your infrastructure was already provisioned?

I wrote up a deeper dive into this concept with example scanning output on my Substack if you want to read more:https://substack.com/@sachm24/p-218428641

Are there any slack servers/channels for SRE discussions?

I finally got to write this blog post, I've wanted to write it for a long time now. It contains the different types of sensitive data we've seen so far, where it's coming from, and why it matters. Hope it can be helpful to some of you!