SRE Capacity Intelligence — Methodology Reference

Policy version: 1.0 Maintained by: SRE Source of truth: backend/app/recommendation.py

What the tool does

The Capacity Intelligence Dashboard scores every active FI stack (HQ, Ardent, Krayt, MemCache) against a set of signal thresholds derived from 30-day P95 telemetry. It produces one of four verdicts — Undersized, Overallocated, Right-sized, or Unknown — and generates a Jira ticket for any FI that crosses a threshold.

All recommendations are read-only. No automatic scaling is applied. SRE and the service owner must review and approve before any Nomad job spec change is made.

Data sources

Signal thresholds

These are the exact values coded in recommendation.py. All are applied to 30-day P95 measurements unless noted.

Memory is judged BASE-relative and TIERED (updated 2026-09-07). memory_p95_30d is measured against the base Nomad reservation, not the oversubscription ceiling (memory_max). The ceiling is burst room borrowed from co-located neighbors and is not guaranteed, so it does not soften the trigger — a stack whose p95 sits near its own reservation is tight regardless of ceiling headroom. But a single flat bar over-flagged: a >70 rule marked 464 stacks Undersized while *none* sat near their real cgroup ceiling — a non-credible flood mirroring the collapse it replaced. So memory now has two bands:

• p95 > 100% of base → Undersized. The stack is running past its own reservation, with no guaranteed floor left.
• p95 70–100% of base → Watch. Stays Right-sized, but risk is bumped to Medium and the memory_pressure flag + a reason are attached, so the pressure is visible without inflating the Undersized count. (This is the 3397-HQ shape at ~83% base p95 — its worst instance is only ~57% of its actual cgroup limit, so it is genuinely not Undersized, but it is not "all clear" either.) The memory_pressure flag mirrors the autoscaler_dependent pattern.

Latency signal — important caveat

Latency alone and error rate alone do not produce Undersized tickets.

Both latency and error rate are FI-level health signals aggregated across the application, upstream (core/vendor), and network layers. High latency or elevated error rates can result from upstream vendor timeouts, JVM GC pauses, adapter failures, or application-layer faults — none of which are resolved by adding Nomad instances or increasing CPU allocation.

Latency guard: The tool requires at least one resource signal (CPU, memory, growth, cache pressure, or PD incidents) to corroborate before issuing an Undersized verdict on latency. Latency-only cases fall through to Right-sized.

Error rate guard: Same logic. Error rate alone does not confirm a resource allocation problem. The tool requires at least one resource signal (CPU, memory, growth, cache, incidents) before entering Undersized on error rate alone. Error-only cases fall through to Right-sized; the signal remains visible in the evidence table.

Example: an FI with 9% error rate but CPU at 48%, memory at 74%, and no incidents will not receive an Undersized ticket — the errors are likely app-layer (Fiserv adapter, GC, health check misconfiguration) and must be investigated separately.

This means: if an FI shows 900ms latency but CPU is at 30%, memory at 40%, error rate at 0.1%, and zero incidents — the tool will not create an Undersized ticket.

Per-core CPU — the sizing signal (soft-cap corroboration guards removed 2026-09-07)

Per-core CPU p95 > 75% triggers Undersized on its own for any non-autoscaled (or pinned-near-max) HQ, Ardent, or Krayt stack. No corroborating signal is required.

nomad_client_allocs_cpu_total_percent (the figure the collector reports as cpu_p95) is PER-CORE saturation — 100% = one full core — not a percentage of the Nomad MHz reservation. Verified 2026-09-07 on 3397 HQ: the instance used ~113 MHz of a ~2820 MHz core = 4.0%, which is exactly the reported total_percent; dividing by the 800 MHz reservation instead would have read ~14%. So:

• cpu_p95 > 100 → a single instance sustains more than one whole core at its 30d p95 (while typically reserved a fraction of a core) — genuinely CPU-heavy and burst-dependent, independent of the reservation size.
• cpu_p95 > 75 → the trigger bar = > 0.75 core/instance.

1. total_percent is per-core, not %-of-reservation (above), so a high reading is real per-instance CPU load, independent of how large the MHz reservation is. 2. CFS throttle is ~0 fleet-wide — max 0.88% avg-30d, and even the worst stack's 14d 1-hour PEAK throttle is 2.6% (5138 HQ) / 2.25% (3667 Krayt); no stack anywhere near the 20% bar. Q2's soft caps + roomy hosts let jobs burst instead of throttle, so throttle can never corroborate a soft CPU verdict — the guards simply suppressed every soft-cap CPU verdict permanently.

Per-core CPU p95 now stands on its own. Throttle is retained as an informational / leading-indicator signal only (it still shows in the evidence table and would still promote via the pressure path if it ever exceeded 20%, which fires for 0 stacks today). The 3667 / 3402 Krayt stacks that the old guard "rescued" from a false Undersized are, on the corrected per-core reading, genuinely CPU-heavy (108% per-core = > 1 full core/instance) and legitimately Undersized — the earlier "false positive" framing was itself the artifact of the wrong %-of-reservation premise.

The 14d CPU peak does NOT trigger Undersized — it stays in the downsize safety guard only (a recent spike blocks a *downsize* but does not flag Undersized). This matches the tiered memory model (the 14d memory peak is likewise informational-only): folding the peak into the trigger reintroduced the ~38% peak-only day-to-day churn the volatility fix removed, and CPU is spikier than memory RSS, so if the memory peak doesn't trigger, neither should the CPU peak.

The autoscaler gate is untouched: cpu_triggered = cpu_p95 > 75 AND (not is_autoscaled OR pinned ≥ 90% of max). An autoscaled-not-pinned HQ/Ardent group meets p95 by scaling out, so it is diverted to Right-sized + autoscaler_dependent (below), not Undersized.

Backout: restore both guard blocks in recommendation.py from git history (the commit prior to 2026-09-07) — the if record.app == "Krayt" and cpu_triggered and not : pressure = False clause and the if record.app in ("HQ","Ardent") and not is_autoscaled and cpu_triggered and not : pressure = False clause — and reinstate the old != "Undersized" assertions. Tests pinning the current behavior: test_recommendation_percore_cpu.py (whole file), plus test_krayt_rawexec_cpu_only_triggers_undersized, test_non_autoscaled_hq_cpu_only_triggers_undersized, test_non_autoscaled_hq_high_cpu_is_undersized_not_flagged, and test_pinned_at_max_hq_cpu_still_undersized.

How CPU throttle % is measured (retained as an informational signal)

CPU throttle % is no longer a classification gate — the corroboration guards it fed were removed 2026-09-07 (above). It is retained as an informational / leading-indicator signal (cpu_throttle_pct_30d > 20 still promotes via the pressure path, but fires for 0 stacks today). It answers a question per-core p95 can't: *was the job actually held at a CFS ceiling?* The collection mechanics below are unchanged and were validated as part of item 2 (the stored value matches Thanos; the series is present for the big prod FIs; and the 30d-avg vs 1h-peak coarseness hides nothing — peak throttle is < 3% even for the worst stacks).

Throttling is a cgroup CFS effect: each enforcement period the scheduler grants the cgroup its CPU quota, and if the job wants more it is *throttled* until the next period. cAdvisor/Nomad expose nomad_client_allocs_cpu_throttled_periods (a counter of throttled periods) but not a total-periods counter. That total isn't in Thanos and can't be — but it doesn't need to be: the Linux default cpu.cfs_period_us is 100 ms, so there are exactly 10 enforcement periods per second per instance, fixed, regardless of core count (CFS periods are per-cgroup). So:

` cpu_throttle_pct = avg by (exported_job)( rate(nomad_client_allocs_cpu_throttled_periods[30d]) ) / 10 * 100 `

The avg-per-instance rate (not sum/increase) is robust to counter resets and alloc churn — a stack that scaled from 2→8 instances mid-window isn't diluted, and a short-lived alloc contributes its own rate.

The bug this replaced (silent since inception): the collector divided increase(throttled_periods) by increase(nomad_client_allocs_cpu_allocated). But cpu_allocated is a gauge (the MHz reservation), so its increase() over any window is ≈ 0 → total <= 0 → the loop continued and dropped every stack. The result: cpu_throttle_pct_30d was None for all 2081 records fleet-wide, so throttle_triggered never fired once — not for HQ, not for Ardent, not even for Krayt reading 300%+ of its soft reservation. The corroboration meant to validate P95 was structurally dead. (This is why the worked example on line 82 shows "throttle uncollected (None)".) Measured directly from Thanos after the fix, current fleet severity is mild — hot HQ login tiers sit ~3–4% (well under the 20 bar) — so no verdict flips today, but the leading indicator that would catch a stack tipping 4%→30% is now actually connected.

Backout: in backend/app/collectors/grafana_thanos_live.py, restore the two-query throttled_map/total_map form of _batch_query_cpu_throttle with the increase(nomad_client_allocs_cpu_allocated…) denominator and the if total <= 0: continue drop; remove _CFS_PERIODS_PER_SEC / _throttle_rate_to_pct; delete backend/tests/test_cpu_throttle_collector.py. Reverting re-blanks cpu_throttle_pct_30d fleet-wide (the corroboration goes dark again). No server restart needed to revert the code; the served values only change after the next throttle collector run.

The second bug (the merge silently dropped it): even after the query was fixed, cpu_throttle_pct_30d still read None fleet-wide because _merge_preserving_good_values in the same collector preserved cpu_p95 / memory_p95 / memory_worst_instance_pct from a transient-empty fetch but never copied the freshly-fetched throttle onto prev. sync_live_metrics seeds its existing map from the throttle-less JSON fallback file (imported/grafana_metrics.json has cpu_throttle_pct_30d: null), so for every already-existing (fi, stack, app) the just-queried throttle was thrown away and the null seed was written straight back to both the JSON and the SQLite grafana_metrics table. This is why a direct DB backfill of throttle survived only until the next 30-minute auto-refresh (_run_full_refresh → sync_live_metrics), which re-nulled it. The fix applies the same preserve-on-transient-empty rule to throttle: a fresh non-None value propagates to prev; a None (empty run) does not clobber a good prior value. Locked by test_merge_propagates_fresh_throttle_over_null_seed, test_merge_updates_throttle_over_prior_value, and test_merge_transient_empty_throttle_does_not_clobber_good_value. Backout: delete the new_throttle block in _merge_preserving_good_values and those three tests — throttle goes dark again on the next refresh.

Autoscaler-dependent (masked-CPU) flag

The autoscaler CPU gate (cpu_triggered = cpu_p95 > 75 AND (not is_autoscaled OR pinned≥90% of max)) correctly stops an autoscaled HQ/Ardent group with high per-instance CPU from being called Undersized — the group absorbs the load by scaling out. But silently returning that group as plain "Right-sized" hides a real distinction the SRE capacity practice cares about: it meets p95 only via burst + autoscaler, not via provisioned headroom.

This matters because on HQ/Ardent the Nomad MHz "reservation" is a soft cpu.shares weight, not a hard CFS quota — so per-instance CPU p95 routinely reads >100% of reservation while CFS throttle stays ~0. Measured fleet-wide 2026-09-07 (throttle backfilled onto 1035/2085 rows): 30 HQ/Ardent stacks at 75–178% of reservation, worst throttle 0.34%, zero above the 20% bar. That confirms the per-core reading is real (per-instance CPU load, throttle ~0), and the autoscaled group meets it by scaling out — the reason this case is Right-sized + flagged rather than Undersized. (This also explains the 2026-09-07 "≈180 → 4 Undersized" drop: the first fleet-wide live-Nomad backfill populated scaling_min/max, activating the autoscaler gate on stacks that were previously flagged on the soft-cap over-read alone.)

The flag: when a stack is Right-sized *solely* because the autoscaler gate cleared a genuine CPU breach — record.app in ("HQ","Ardent") AND cpu_p95 > 75 AND is_autoscaled AND not pinned_near_max AND not throttle_triggered — recommendation.py sets autoscaler_dependent = True, bumps risk Low→Medium, and rewrites the action/reason to name the dependency ("verify the autoscaler minimum holds burst headroom"). The status stays "Right-sized" (additive, non-breaking — no verdict flips), so the tool surfaces the autoscaler dependency instead of hiding it. Krayt (raw_exec) and MemCache (fixed, inherits HQ) are excluded — neither is a cpu.shares autoscaler-absorbed burst; a non-autoscaled Krayt/HQ/Ardent stack with high per-core CPU triggers Undersized directly (per-core CPU section) rather than being masked. A corroborating CFS throttle (>20%) still promotes the stack to a genuine Undersized verdict via the normal pressure path (fires for 0 stacks today).

Backout: in recommendation.py remove the _autoscaler_masked_cpu computation and the autoscaler_dependent dataclass field, and revert the Right-sized return to the single unconditional form (risk="Low", action="No change; review next cycle.", reason="Current allocation has acceptable utilization and safe headroom."); drop the "autoscaler_dependent" keys added to _rec_dict and the confirmed-inventory payload in main.py; delete backend/tests/test_recommendation_autoscaler_dependent.py. The field defaults False, so reverting is behavior-neutral for every existing consumer.

HQ/Ardent non-autoscaled CPU — guard removed (2026-09-07)

Earlier revisions carried a non-autoscaled HQ/Ardent soft-cap CPU guard that (like the Krayt one) required a corroborating throttle before per-core CPU could flag Undersized. It was removed 2026-09-07 together with the Krayt guard, for the reasons in the per-core CPU section above: total_percent is per-core saturation (not %-of-reservation), and CFS throttle is ~0 fleet-wide, so it could never corroborate. A non-autoscaled HQ/Ardent stack with cpu_p95 > 75 now triggers Undersized directly.

The autoscaler gate still diverts an *autoscaled-not-pinned* group to Right-sized + autoscaler_dependent; a pinned-at-max group (≥ 90% of scaling_max) still triggers Undersized (it can't scale out — test_pinned_at_max_hq_cpu_still_undersized). See the per-core CPU section for the full trigger rule and backout.

Region-aware metrics

Q2 runs an active/passive topology via Cloudflare (steering_policy: off). The standby region receives only ~1.5 req/s Cloudflare health probes, not real customer traffic.

The tool uses Nomad deployment data to identify each stack's primary region (USE1 or USW2) and prefers that region's metrics when both regions report data. If a stack's metrics appear to reflect only health-probe noise (CPU p95 < 2% AND memory p95 < 15%), the stack is flagged as Unknown / low signal and no sizing recommendation is produced.

MemCache CPU inheritance

MemCache stacks have no dedicated Thanos exporter. Their mchq / mcard Nomad task groups run under the HQ exported_job, so the collector never queries MemCache CPU directly — instead, during enrichment, each MemCache stack inherits CPU and memory p95 from its co-located HQ stack for the same (fi, stack) key (grafana_metrics_static.py, MemCache branch).

Because MemCache borrows HQ's numbers, it is exposed to any gap in HQ's live CPU collection. To protect against that, MemCache — like HQ, Ardent, and Krayt — falls back to the HQ history p95 (grafana_metrics_history, ≥30 snapshots) whenever the HQ live CPU query returns empty. This history fallback is re-applied *after* the live-metric copy, so a transient 1–2 day HQ Thanos gap no longer forces the MemCache stack to Unknown.

> History (2026-08-06): A field-ordering bug previously let the live-metric copy clobber the HQ history CPU back to None for MemCache only (the generic history re-apply was gated != "MemCache"). This silently forced 208 MemCache stacks (all USW2) to Unknown on a 2-day HQ collection gap, and the Unknown reason string mis-blamed *"cache-specific telemetry"* (hit ratio / evictions) — which are enhancing signals, never capacity blockers. Fixed: MemCache now gets the history re-apply, and the Unknown reason now names the real missing metric (CPU/memory p95). Regression tests: test_memcache_cpu_survives_hq_live_gap_via_history, test_memcache_missing_cpu_blocks_with_cpu_reason_not_cache_reason.

Traffic growth — near-zero-baseline suppression

Growth is (recent − baseline) / baseline. When the baseline RPS is near zero, that ratio explodes to the ±500% cap even for a perfectly normal ramp. The classic case is a pre-production go-live stack: the standby/pre-live period sees only ~1.5 RPS of Cloudflare health-probe traffic, so when real customers begin landing the "growth" reads as thousands of percent off a noise-floor denominator.

That is a denominator artifact, not a capacity-pressure signal — and left unchecked it drives a bogus *"traffic +500% → add N instances"* recommendation on a stack that has no real load yet. So when the baseline is below 2 RPS (the health-probe noise floor) and the raw increase exceeds the +500% cap, the value is suppressed to null rather than written as a confident number. The stack then scores on its real CPU/memory signals instead of a phantom traffic surge.

A large negative swing from a *real* baseline that goes dark is still kept — that is a genuine traffic drop worth surfacing. The suppression only applies to the near-zero-baseline blow-up. Suppressed stacks are reported in the sync result under suppressed_low_baseline / suppressed_stacks (no silent drops). Regression test: test_near_zero_baseline_ramp_is_suppressed_not_capped.

Traffic growth — projected-utilization gate

Growth is a leading signal: a stack trending up will eventually need capacity. But a high growth *rate* on a near-idle footprint is not a capacity problem — a +78% jump on 1.7% CPU is still ~3% CPU. Left ungated, the plain growth > 20% rule manufactured false Undersized / CRITICAL — "Fix Now" rows on fast-growing but near-idle stacks (the fleet's MemCache tier sits at CPU p95 median ~1.6%, memory ~12%, yet routinely shows large month-over-month growth off a low base).

So growth only triggers Undersized when carrying the current p95 forward by the growth rate would actually approach a ceiling:

` projected_cpu = cpu_p95_30d × (1 + growth/100) projected_mem = memory_p95_30d × (1 + growth/100) # base-relative, matching mem_triggered (>100) growth triggers Undersized ⇔ growth > 20% AND (projected_cpu > 75% OR projected_mem > 100%) AND (not autoscaled OR pinned ≥ 90% of scaling max) `

The memory threshold matches mem_triggered: 100% of the base reservation (base-relative, tiered — the Undersized bar; see the "Memory is judged BASE-relative and TIERED" note above). The autoscale clause is unchanged — an autoscaled group absorbs growth by scaling out, so growth only warrants a *manual* add-instances recommendation when the group is already pinned near its max.

Worked example (FI 4432 Morton Community Bank, MemCache): +78% growth, CPU p95 1.7%, memory 19%. Projected forward: CPU ~3%, memory ~34% — nowhere near a wall. Under the old rule this rolled the whole FI up to CRITICAL / Undersized in the Customer Priority Queue purely on the growth rate; the FI's genuine posture is driven by its Krayt stack (Overallocated → HIGH). The gate keeps a stack whose growth *does* project past a ceiling (e.g. memory p95 60% growing 30% → 78% > 70%) flagged. Regression tests: test_growth_on_idle_footprint_not_undersized, test_growth_projecting_into_memory_ceiling_triggers, test_autoscaled_growth_pinned_near_max_triggers, test_non_autoscaled_growth_still_triggers.

Backout: revert the _growth_would_breach gate in recommendation.py (growth_triggered = … and growth > 20 and (not is_autoscaled or _pinned_near_max), dropping the projected-utilization term) and revert this section; the four tests above pin the intended behavior.

Recent-peak downsize guard (14-day high-water mark)

The 14-day CPU/memory peak (cpu_p95_peak_14d / memory_p95_peak_14d) is a downsize-only guard. recommendation.py folds it into _cpu_downsize_guard = max(cpu_p95_30d, cpu_p95_peak_14d) (and the memory equivalent), and a stack is only marked Overallocated when *both* the sustained p95 and the recent peak are low (_cpu_downsize_guard < 35 AND _mem_downsize_guard < 55). This keeps us from shrinking a stack that spiked recently even though today's reading looks calm — conservative-on-downsize.

The peak does NOT drive the Undersized verdict. Undersized keys on the 30-day p95 (cpu_p95_30d > 75), memory, latency and errors. A stack whose CPU dropped well below its recent peak is simply *held out of "safe to downsize"* — it is not thereby Undersized. The Metrics Trend modal reflects exactly this: when today's CPU sits below the 14-day peak it shows a "⚠ holds off downsize" badge and explains the peak guards the downsize direction only. (Earlier copy wrongly framed the peak as a "used for sizing decision … stays Undersized until 14 clean days" hold, and fired it on peak-divergence alone — so healthy stacks like FI 3368 Krayt at 16% CPU read as "Why is this Undersized?". Corrected; regression test test_frontend_metrics_trend_explainer.py.)

Backout: restore the prior modal copy in frontend/index.html (showMetricsTrend, the peakDivergent badge + explainer) and revert this section; test_frontend_metrics_trend_explainer.py pins the corrected wording.

Inspecting the Undersized count (drill-down)

The Executive Summary Undersized FIs KPI card and the Stack Health Distribution Undersized tile are click-through: they open a drill-down (showUndersizedDrilldown in frontend/index.html) that lists every stack classified Undersized with the exact signal(s) currently above threshold, taken from recommendation.evidence (only entries whose result === "triggered"), plus each stack's recommended action. This makes the headline count verifiable at a glance — e.g. a growth-on-idle candidate shows up as a lone traffic +78% chip on a near-idle MemCache stack (which the growth-into-a-ceiling gate then suppresses), rather than hiding inside an opaque number. (A CPU p95 108% per-core chip on a Krayt stack is now a *genuine* Undersized — per-core saturation, not a soft-cap over-read; see the per-core CPU section.) Test: test_frontend_undersized_drilldown.py.

Backout: remove the onclick="showUndersizedDrilldown()" wiring on the two cards, the onclick parameter added to the kpi() helper, and the showUndersizedDrilldown function; revert this section.

> Worked example (FI 5115 / AFCU, go-live 2026-08-17): baseline 1.57 RPS (health probes) → recent 101.8 RPS (early ramp) = raw +6,364%, formerly capped and displayed as "+500% → add 12 instances". Now suppressed to null; the stack is sized on CPU/memory until real post-go-live traffic accumulates.

SLI risk driver — capacity is a driver, not just a symptom (2026-09-15)

The SLI drill-down (openSliDrilldown → /api/v1/sli/stacks) explains *why* a stack is At Risk or Burning in one Risk Driver label. That label used to consider only **restarts, error burn, latency, and incident count** — no capacity signal at all. The result was misleading on exactly the stacks that matter: FI-3296 sat at memory p95 120% and CPU p95 196% and reported its driver as Latency 4224ms, while its own Action column read *"Increase memory to 1.02 GB per instance (mem p95 120%)"*. The capacity data was on the same record the whole time (cpu_p95_30d / memory_p95_30d); the driver simply never looked at it, so the panel blamed the symptom (latency) and hid the cause (saturation).

Driver priority (_derive_risk_driver in app/sli.py), highest first:

1. Restarts N.NNx — Nomad alloc burning / at_risk (a crashloop outranks everything) 2. Error burn N.NNx — burn rate > 1.0x 3. Memory N% of reserv. — memory p95 ≥ 100% of the base reservation 4. CPU N.NN cores/inst — CPU p95 ≥ 100% per-core (over one full core per instance) 5. Latency Nms + errors — latency over target *and* error rate over half the budget 6. Error burn N.NNx — burn rate > 0.5x 7. Latency Nms — latency over the service's SLO target 8. Memory p95 N% — elevated, ≥ 80% of reservation (thin headroom, not yet over it) 9. CPU N.NN cores/inst — elevated, ≥ 75% per-core (over 0.75 core/instance) 10. N incidents/30d — more than 3 de-noised incidents

> Mind the denominators — they are different, and conflating them misreads the fleet. > cpu_p95_30d is per-core saturation (total_ticks / one-core-MHz * 100), *not* a > percentage of the MHz reservation: 100% = one full core sustained by a single instance. FI-3296 > HQ reads 195.9% = 1.96 cores/instance (5,682 MHz against a ~2,900 MHz core), while reserved > only 800 MHz — legitimate, because cpu.shares is a soft cap enforced only under host > contention. That is why the label is rendered in cores, and why cpu_throttle_pct_30d > (also now on the payload) is the corroborating signal: 0% throttle means the host is still > supplying the CPU. memory_p95_30d is percent of the base reservation; over 100% means the > reservation understates real usage (a sizing/scheduling-correctness problem, since Nomad > bin-packs on it) and is not by itself an imminent OOM — OOM proximity is the separate > worst-instance-vs-hard-cgroup-limit signal, and the hard limit is typically far larger than the > reservation (3296 HQ: 120% of a 0.68 GB reservation, but only 38% of its 1.95 GB limit).

Both capacity bars rank above latency because they are causal for it; memory outranks CPU because exceeding a memory reservation is harder to absorb than CPU, which cpu.shares only clamps under contention. The elevated-but-not-over labels rank *below* latency (thin headroom is not yet a breach) but above a bare incident count, which explains nothing on its own.

This changes labels only — not classification. _derive_status takes no capacity arguments and is untouched, so no stack is reclassified and the fleet's At Risk / Burning counts do not move. Pinned by test_status_classification_is_unchanged_by_capacity.

Drill-down columns. The table now shows sortable CPU/core and Mem p95 columns, amber past the elevated threshold and red at/over 100%, with a dash when the metric is genuinely absent so "no data" is distinguishable from "0%". The column tooltips spell out the two denominators above, so a 196% CPU reading isn't misread as "2x over its cap". Two capacity passthrough fields were also misnamed — the endpoint read current_cpu_cores / current_mem_mb, but a recommendation carries current_cpu / current_memory_gb, so both always serialised as null; fixed, and cpu_p95_30d / memory_p95_30d / cpu_throttle_pct_30d added to the payload.

Cold-cache latency. confirmed_inventory_recommendations() takes ~13s to score the 2,027 confirmed rows and used the default 300s cache TTL — so the entry expired every 5 minutes and roughly every other interaction paid the rebuild on the request path (measured 18s cold vs 0.4s warm). It now has an _CACHE_TTL_OVERRIDES entry of 1800s and is **pre-warmed at startup** in a background task. Correctness is unaffected: _bust_all_caches() already drops the key on every refresh/import that could change it, so the TTL only avoids needless recomputation (pinned by test_refresh_still_busts_the_recommendation_cache).

Tests: test_sli_capacity_risk_driver.py (driver priority, thresholds, backward-compatible optional args, no reclassification), test_sli_stacks_capacity_fields.py (payload fields, the misnamed-key regression, cache TTL + bust), test_frontend_sli_capacity_columns.py (the two sortable columns, threshold colouring, colspan, tooltips).

Backout: in app/sli.py remove the CPU_P95_ELEVATED_PCT / MEM_P95_ELEVATED_PCT / SATURATED_PCT constants, the cpu_p95 / mem_p95 parameters and their four branches in _derive_risk_driver, the two record.get(...) lookups feeding them in compute_stack_sli, and the cpu_p95_30d / memory_p95_30d keys from its return dict. In main.py revert the sli_stacks capacity keys (or simply drop them — no consumer requires them), remove the confirmed_inventory_recommendations TTL override, and delete the _startup_warm_recommendations task. In frontend/index.html remove the two s, the two th(...) headers, the two fmtPct(...) cells, the fmtPct helper and the four saturation tooltip branches, and restore colspan="6". Delete the three test files. All additive — the capacity args default to None, so every existing caller behaves exactly as before.

Posture stability

Why posture could fluctuate: A Thanos instant query with a [30d] lookback evaluated at a single point in time has a window edge — a CPU spike from exactly 30 days ago rolls out of the window at midnight, so a naive p95 could drop 80–100 points overnight and flip a stack Right-sized when it was Undersized the day before. That would create false confidence and erode trust in the tool.

How stability is achieved now (two mechanisms — neither folds the 14d peak into the Undersized trigger):

1. CPU/memory classification keys on the STABLE 30-day p95, not the 14-day peak. The 30d p95 is a sustained-load measure that barely moves day to day, so a single spike aging out of the window does not flip the verdict. (Folding the 14d peak into the *trigger* was tried and reverted 2026-09-07 — it reintroduced the ~38% peak-only churn the volatility fix removed; see the per-core CPU and recent-peak-downsize-guard sections. The peak is retained only in the downsize safety guard.) 2. The collector preserves last-known-good values. _merge_preserving_good_values keeps a prior good cpu_p95 / memory_p95 when a single fetch comes back transient-empty, so a dropped snapshot can't silently clear a posture between refreshes.

Downsize direction still uses the peak: a stack must have peak CPU < 35% and peak memory < 55% across the last 14 days before it qualifies for Overallocated — this prevents a false downsize ticket immediately after a seasonal spike rolls off. (Conservative-on-downsize; volatile-on-upsize was the problem the volatility fix cured.)

Display: Both values are shown in the Stack Details modal — "CPU p95 (30d)" is the sustained reading used for classification and sizing math; "CPU p95 Peak (14d)" is the high-water mark that guards the downsize direction. When the peak is materially higher than today, the Metrics Trend modal shows a "⚠ holds off downsize" badge explaining the peak guards the downsize direction only.

Verdict logic (decision tree)

` 1. Current CPU or memory = 0? → Unknown (fix allocation data) 2. metrics_source = "low_signal"? → Unknown (standby-region noise) 3. CPU p95 or memory p95 missing? → Unknown (insufficient telemetry) 4. Stack age < 28 days? → Observe (launch stabilization) 5. Any pressure signal triggered? → Undersized CPU classification uses the stable 30d p95 (>75% per-core); memory uses the base-relative 30d p95 (>100% = Undersized). Both 14d peaks are downsize-guard only. (but latency-only with no infra signal → falls through to step 6) 6. Peak CPU p95 < 35% AND peak mem < 55% AND no incidents AND age ≥ 56d AND growth ≤ 10%? → Overallocated 7. Otherwise → Right-sized (memory p95 70–100% of base → Right-sized + memory_pressure flag, risk = Medium) `

Scaling type selection

When a stack is Undersized, the tool selects the appropriate scaling lever:

CPU recommendation formula

This sets the magnitude of a CPU upsize only. It does not decide Undersized vs Right-sized — that is the per-core cpu_p95 > 75 trigger above. _cpu_recommendation is only called on an upsize path.

Preferred basis — measured MHz (ticks), 2026-09-07

nomad_client_allocs_cpu_total_ticks 30d p95 is the actual MHz a single (hottest, max by (exported_job)) instance consumes — captured into cpu_ticks_p95_mhz. When present it is the sizing basis:

` recommended = round((cpu_ticks_p95_mhz / 0.60) / 50) × 50 ← size to 60% target, nearest 50 MHz floor = current_cpu_mhz × 1.5 result = max(recommended, floor) `

Why this is the rigorous basis: the Nomad CPU reservation is a soft cap (a token for Krayt raw_exec — typically 100 MHz — and cpu.shares for HQ/Ardent — typically 800 MHz), often *far* below real consumption. The measured ticks p95 is what the instance actually burns, independent of the reservation size.

Example — 3667 Krayt: 100 MHz reservation, 30d p95 ticks ≈ 10,087 MHz → 10087 / 0.60 = 16,811 → 16,800 MHz. (The per-core fallback below gave 600 MHz — ~28× low — because it scaled the *100 MHz token* by the per-core %.) For HQ/Ardent (real 800 MHz cpu.shares) the two bases roughly agree; ticks mainly corrects the Krayt soft-cap under-read.

The recommendation reason cites the measured value (~N MHz measured p95) when ticks is available, so the number is traceable to real consumption.

Fallback basis — per-core heuristic (ticks unavailable)

When cpu_ticks_p95_mhz is absent (older evidence, or a high-cardinality query that returned nothing):

` actual_avg_mhz = current_cpu_mhz × (cpu_p95 / 100) target_mhz = actual_avg_mhz / 0.60 ← size to 60% utilisation target recommended = round(target_mhz / 50) × 50 ← round to nearest 50 MHz Nomad increment floor = current_cpu_mhz × 1.5 ← minimum: 1.5× current result = max(recommended, floor) `

cpu_p95 is per-core saturation (100% = one core), not %-of-reservation, so this only *biases* correctly (heavier per-core → larger reservation); for tiny soft-cap reservations it under-recommends — hence ticks is preferred. The current_cpu × 1.5 floor applies to both bases.

Example (fallback): HQ stack at 850 MHz, CPU p95 = 107% → actual = 850 × 1.07 = 909.5 → target = 909.5 / 0.60 = 1515.8 → 1500 MHz. Floor = 1275 MHz. Result: 1500 MHz.

The 60% target leaves 40% headroom above measured P95 — intentional, because P95 is not the absolute peak and traffic can spike.

How ticks is collected: _batch_query_cpu_ticks (collectors/grafana_thanos_live.py) runs a 30d raw-range p95 of nomad_client_allocs_cpu_total_ticks (same max by (exported_job), task_group + logging-sidecar exclusions, chunking, and primary-region dedup as the CPU% query), captured once per UTC day under the same gate as CPU%. It flows through grafana_metrics.cpu_ticks_p95_mhz → the _METRIC_FIELDS enrichment gate → FiStackRecord.cpu_ticks_p95_mhz.

Backout: in recommendation._cpu_recommendation, drop the ticks_p95_mhz parameter and its if ticks_p95_mhz … branch (keep only the per-core heuristic); remove the three call-site record.cpu_ticks_p95_mhz args and the _measured_note. Optionally leave the collector/DB plumbing in place (the field is harmless when unread) or revert the collector (_batch_query_cpu_ticks, _targeted_ticks_query, the ticks_map/rec-dict/merge/setdefault edits), db.py (schema + two ALTERs + snapshot/mapper keys), grafana_metrics_static._METRIC_FIELDS, and the main.py trend serializer key. Tests pinning intended behavior: test_recommendation_cpu_ticks_sizing.py, and the _LOGGING_SIDECAR_EXCL == 8 count + two ticks tests in test_cpu_sidecar_exclusion.py. The DB column is additive (nullable REAL) — no data migration to reverse.

Confidence levels

Undersized

Overallocated

Right-sized

Cost impact calculation

Cost impact is reported as a monthly delta (negative = savings, positive = investment). It is an estimate, not a CloudZero projection.

Horizontal scaling: impact = -(rec_instances / cur_instances - 1) × monthly_cost

Vertical scaling: Impact is derived using Nomad resource weights (CPU at $0.048/GHz, memory at $0.006/GB) as relative weights applied to the actual CloudZero or Nomad-estimated cost. Absolute dollar values are not taken from these weights — they serve only to proportion the cost delta.

CloudZero actuals are used for HQ stacks when available. Other stacks use Nomad allocation estimates.

Priority scoring

Tickets are prioritized using a composite score combining:

• FI tier (Enterprise > Key > Premier > Community)
• MRR (from FI Directory)
• Pain score (from CX Four Signals grid — Lou Senko's Health/Friction/Pain/Clean Case model)
• Number of Sev 1/2 incidents YTD
• Escalations YTD
• SLI burn rate

What the tool does not do

• It does not distinguish between internal HQ call latency and external core (vendor) latency. The latency signal is a single FI-level aggregate. If a ticket shows high latency with no CPU/memory signal, the issue is likely upstream, not Nomad allocation.
• It does not compare your thresholds against external benchmarks. The thresholds above were set by SRE based on Q2's observed fleet behavior and are not derived from vendor SLAs or industry standards.
• It does not apply changes. Every recommendation requires human review and approval before any Nomad job spec is modified.

Questions and working sessions

If a recommendation doesn't match what you see in Grafana or Nomad Operator, the most common causes are:

1. Region mismatch — the tool is querying the primary region; Grafana may be showing a different region or aggregate 2. Time window — the tool uses a 30-day P95; Grafana defaults vary 3. Allocation vs utilisation — Nomad Operator shows reserved MHz; Thanos shows % of that reservation consumed

To validate a specific FI together, contact Mark or schedule a working session via SERVSRE.

SRE Capacity Intelligence · capacity-analysis-methodology.md