Resilience Coffee/2026/09/18

Measurements of health of systems/reliability/etc. Extension of a Community of practice discussion "How do you know if you're actually getting better from these investments in reliability?" "Am I reliable because I implemented SLOs or am I implementing SLOs because I'm reliable?" Lots of ideas about prescriptively how should you measure reliability, e.g. DORA https://dora.dev/ Post from community: https://non-functional.net/posts/2026-08-21-niall-s-naive-list-of-unsolved-problems-in-software-reliability/ Sharp end/blunt end knowledge/communication gap Trends in number of incidents isn't quite right We know what we shouldn't do but it's not clear what we should do? Are we operating off of an existing definition of what "better" looks like? No, that's part of the problem Different measures: Is it empirical? Or is it just "are people giving us money?" Until you define what better is, can you even have this conversation? "The best answer I have heard of better is likely from Nancy Levenson's work" Counting good things, not bad things Make it a control theory framing instead, STAMP/STPA The numbers there aren't clear though either Connectiong to budgeting, e.g. if spending goes up it's "bad" or if revenue goes up, it's always "good" Small but more frequent incidents may be better than large infrequent unit economics, per business metrics incidents per features (or per deployment) not on its own but in combination with others Cautious about that approach - it may be helpful but not in a strong causation sense, may be in motivational sense What information do I have in the world to know that things are improving? Even if we have too small of a sample size. How do I measure a specific thing in place, causality is often hard to infer, DAG causality is tricky to figure out. Cost of determining causality is expensive, so how can we have some confidence with out it Different ideas of measurement from public policy theory: https://www.researchgate.net/figure/Characteristics-of-Performance-Measures-for-Different-Purposes-The-purpose-To-help_tbl2_227533290 A single number doesn't work because the space is multi-dimensional, connected orthogonal metrics in tension Connection to sense-making/naturalistic decision making/intuition: When cause and effect are close, you build that sense over time But a lot of the connections in software are long distances between cause in effects so intuition breaks down Maybe the sense-making is key: defining better socially is equivalent to defining the metrics, and they should adapt to the current context over time Should we have metrics as targets or other things? Are we trying to get confidence above a threshhold (e.g., get to a point where it makes sense to even gather the measurement)? Or are we trying to incrementally improve the metric, regardless of the resulting metric's utility? e.g. MTTR may be directionally correct some of the time But we don't know for most metrics Distinction between indicators/diagnostic/etc. metrics Sales techniques may be more important than the right numbers, but if we care about reliability and these things in the world, we should do our best to see if we can have feedback that this is more than just feels But it may all be feels anyway: we rationalize decisions after the fact, especially socially I'm positive towards using feelings! I just would like to increase my pool of resources to train my intuition. Because I don't know if these things are working. Rationalization can influence others: Feeling -> Rationalization -> Communication -> Observation -> Feeling Optic system and split brains https://www.youtube.com/watch?v=_TYuTid9a6k Discussion with leadership about counterfactuals https://en.wikipedia.org/wiki/The_Black_Swan%3A_The_Impact_of_the_Highly_Improbable Distinction between counterfactuals in metrics vs incident analysis Rationalization with chaos engineering proof about it Chaos engineering of hypothetical incidents when lessons aren't being absorbed from actual incidents can be tricky What decision would this data inform? There's a cost of collecting data Not just storage but also False leads/false positives "Sociology is more difficult than physics" Frequentist vs Bayesian statistics How do you ramp up to professional work after time away? "Pacing the frontier" https://darioamodei.com/post/we-must-pace-the-frontier What is the frontier? Is it the most capable models? Or the Pareto frontier of capability and a second dimension, e.g. size Seems like we're pivoting to a third dimension https://www.deboraspar.com/ruling-the-waves History of control theory: 1600s: Leibniz/Newton arguments about calc today: reinforcement learning https://www.incontrolpodcast.com/1632769/episodes/18684924-ep41-a-minimal-history-of-optimal-control