Our canary deployments catch obvious breakage, but subtle behavior changes often only show up when real traffic hits edge cases that a small canary percentage rarely sees. Increasing the canary percentage might help catch more of the issues, but it also increases the blast radius if something goes wrong.

Our team have hud running on the canary pods, which helps us spot changes in function behavior before the overall error rate starts moving. The tricky part is knowing when there is enough traffic to trust the comparison and when a difference is worth rolling back over.

For teams running canaries in production, how do you handle this? Do you rely on traffic thresholds, behavioral comparisons, or a mix of both? Also curious what signals have led to unnecessary rollbacks and what subtle regressions were hardest to catch.

Skyportal's agent asks before it runs a command on your hosts, unless the command is on an allow list. It also has a built-in check that guesses whether a command changes data, and that guess beats the allow lists. This week it flagged a read-only command as a file overwrite because of a 2>&1;, so no allow rule could have approved it.

So we're removing the guess. A command will run without asking only if every part of it is on an allow list and nothing matches a deny rule. Everything else asks. A few things always ask, whatever the rules say: credential files, secrets, RBAC objects, kubectl exec, and deletes you can't undo. And if the rules file is broken, everything on that repo asks.

The cost is that a broad entry like sudo will mean exactly that, with nothing behind it to catch sudo rm. If you've put policy in front of automation on hosts, did you keep a "looks dangerous" check as a backstop, or trust the allow list and review it like code?

When you're reviewing a Terraform PR you didn't write, the plan output is only part of the work. You still need to understand how the affected resources fit into the rest of the infrastructure.

I built Rootform to make that easier.

It's a local CLI that reads Terraform and OpenTofu plan or state exports and produces an interactive architecture you can explore, compare and check against policies.

In one of the sample plans, an Event Grid subscription switches from one Service Bus queue to another. Rootform shows the old and new delivery relationships alongside the unchanged source topic. You can inspect the evidence behind those relationships instead of treating every Terraform dependency as an architectural connection.

The part I think is particularly useful for platform teams is the output: you can export the entire interactive review as a single HTML file. A teammate can open it in their browser without installing Rootform or running another service.

It also integrates with GitHub Actions for CI workflows, including architecture comparisons and policy checks. The analysis runs locally, without executing Terraform or accessing your cloud accounts.

Interactive Playground (no signup): https://rootform.dev/playground/

GitHub: https://github.com/rootform-dev/rootform

For those managing Terraform across multiple teams: how do you currently give reviewers the architectural context behind a PR? Do you rely on plan comments, documentation, diagrams, or something else?

#devops #terraform #platformengineer