By Sridhar Vanka · All writing
I compared a local 7B LLM with classical ML and specialized models on tables, time series, graphs, grids, and physical systems. I wanted to see how their errors, calibration, and runtime differed on my 24 GB M5 Pro Mac.
These are 16 illustrative local case studies across 15 experiment phases, not a broad model leaderboard. Evaluation sizes and model training differ by experiment. Confidence intervals appear only where the artifacts contain them; overlapping intervals do not establish a reliable ranking. Energy and most 100,000-item runtimes are estimates. Inspect the complete results.
A useful way to think about model design is that architecture should respect the structure and symmetries of the problem (a principle explored in depth in Bronstein, Bruna, Cohen, and Veličković's 2021 work on Geometric Deep Learning).
In practice, stepping outside LLMs means looking across different data structures (tables, time series, grids, graphs), physical domains (3D geometry and continuous fields), and problem settings (streaming and closed-loop control). These categories overlap in real systems: language is itself a sequence, a stream can carry numbers or images, and control loops can use any of them. I have organized this lab into seven areas and benchmarked each one locally:
Shuffling row order does not change the dataset. Built for structured records with heterogeneous columns like software defects, credit risk, churn, and hardware telemetry.
Numeric values ordered by time or sequence position, such as server CPU load, request latency, passenger demand, and financial ticks.
Nodes and edges where relationships carry the signal: scientific citations, payment fraud rings, supply chains, and microservice dependency graphs.
Non-stationary live event streams where data distributions shift abruptly and models must adapt on every event without batch retraining.
Settings where every control action changes the next state observation under strict sub-20ms real-time stability constraints.
Continuous physical fields governed by partial differential equations (fluid shock waves, heat diffusion, weather fields) that obey conservation laws.
Regular pixel and sensor grids where 2D sliding filters preserve spatial topology even when patterns shift across the frame.
Falsification-first experiments testing where pairing a local 7B LLM with PageRank, TabPFN-v2, CatBoost, and Constrained MPC beats—or loses to—either model alone during hyperscale incidents.
For the first experiment, I tested a pre-trained Tabular Foundation Model (TabPFN-v2, 29 MB Apache-2.0 weights) and a pairwise difference meta-estimator (pdll-RandomForest via PairwiseDifferenceClassifier, Belaid et al., arXiv:2406.20031) against classical gradient-boosted decision trees (XGBoost, CatBoost, LightGBM), a scaled linear baseline (LogisticRegression), and a local 7B language model (qwen2.5:7b via Ollama, 4.7 GB) on real software engineering telemetry.
The dataset is NASA JM1 (Deduplicated): 8,817 unique C/C++ modules after removing 2,061 exact duplicate rows (with majority-vote labels) from the raw 10,878-module NASA ground system telemetry, measured across 21 McCabe and Halstead code complexity metrics (`CYCLOMATIC_COMPLEXITY`, `BRANCH_COUNT`, `HALSTEAD_EFFORT`, `LOC_TOTAL`) with a 21.0% defect rate and median imputation fitted strictly on the training split.
Using a fixed stratified test set of 2,000 deduplicated modules, I scaled the historical training data from 250 rows to 5,000 rows to compare in-context tabular priors (TabPFN-v2) and pairwise difference learning (pdll-RandomForest) against standard single-row tree and linear models.
| N_train | Model | Architecture Family | ROC-AUC ↑ | PR-AUC ↑ | Brier ↓ | ECE ↓ | Fit (ms) | Batch 2k (ms) | 1-Row P50 (ms) |
|---|
In cold-start scenarios where a team only has 250 labeled incidents, how do pairwise difference learning (pdll-RandomForest), uncalibrated GBDT defaults, post-hoc Platt scaling, and a local 7B LLM compare? I evaluated eight configurations on a stratified 200-module held-out slice of deduplicated JM1 (with 95% bootstrap confidence intervals): default LogisticRegression, XGBoost, and CatBoost; pdll-RandomForest (PairwiseDifferenceClassifier wrapping RandomForestClassifier); XGBoost (Tuned + Platt Scaling) (shallow depth = 3, L2 = 10, 3-fold sigmoid calibration); TabPFN-v2; and qwen2.5:7b (both raw verbalized JSON probabilities and post-hoc Platt-calibrated probabilities fitted on a 50-row validation split).
| Model | Approach | Weights Size | ROC-AUC (95% CI) ↑ | PR-AUC ↑ | Brier ↓ | Calibration Error (ECE) ↓ | Latency / Row |
|---|
On the 200-module cold-start slice of deduplicated JM1, pdll-RandomForest achieved the highest ranking and best overall Brier score (0.7423 ROC-AUC, 0.1483 Brier, 5.28% ECE in 0.77 ms/row batched), ahead of default XGBoost (0.7121 ROC-AUC, 10.54% ECE) and TabPFN-v2 (0.6748 ROC-AUC, 5.82% ECE). Regularizing XGBoost (max_depth=3, reg_lambda=10) with 3-fold Platt scaling delivered the lowest calibration error (3.99% ECE, 0.7061 ROC-AUC). Meanwhile, fitting a 1D Platt calibrator on 50 validation rows cut qwen2.5:7b's verbalized calibration error by 5x (24.80% → 4.88% ECE, Brier 0.2521 → 0.1583 while preserving 0.6946 ROC-AUC), confirming that post-hoc calibration fixes verbalized probability miscalibration on small tables.
Raw NASA JM1 contains 2,061 exact duplicate rows that previously allowed high-capacity GBDTs to memorize leaked feature vectors across splits at N ≥ 2,500. Once deduplicated to 8,817 unique modules, TabPFN-v2 and pdll-RandomForest beat standard single-row GBDTs across every training size on the 2,000-module test set: TabPFN-v2 leads at N = 250 (0.6943 ROC-AUC / 3.43% ECE), N = 500 (0.7181), N = 1,000 (0.7107), and N = 5,000 (0.7237), while pdll-RandomForest finishes #2 at N = 250–1,000 (with the best overall calibration at N = 500, 2.57% ECE), edges into #1 at N = 2,500 (0.7098 ROC-AUC / 0.4158 PR-AUC), and serves single rows in ~26 ms vs. 6,014 ms for TabPFN-v2 at N = 5,000.
For the second experiment, I tested whether zero-shot Time-Series Foundation Models (Amazon Chronos-2 at 120M parameters and Chronos-Bolt-Small at 48M parameters) can forecast production traffic and server metrics without training, and whether their 10th-to-90th percentile (P10–P90) uncertainty intervals are reliable enough to drive infrastructure auto-scaling.
I evaluated five models across 15 rolling chronological backtest windows (5 windows per dataset, context length = 512 steps, forecasting 48 steps ahead) on three real series from the Numenta Anomaly Benchmark (NAB): AWS CloudWatch ELB Request Count (5-minute bursty load), AWS EC2 CPU Utilization (5-minute server telemetry with a step-change regime shift), and NYC Passenger Demand (30-minute diurnal cycle).
| Dataset | Model | MASE ↓ | WQL (Quantile Loss) ↓ | 80% Band Coverage (Target: 80%) | P90 Breach Rate (Target: 10%) | Train (ms) | Forecast 48 Steps (ms) |
|---|
Unlike TabPFN-v2 (which attends across every individual row token), Chronos-Bolt-Small groups the 512 historical steps into patches and decodes all 48 future steps and quantiles in a single forward pass. It generated a 48-step probabilistic forecast in 6.2 ms on the M5 Pro with 0 ms training time, running 800x faster than the local 7B LLM (4,932 to 7,239 ms).
On AWS ELB requests, supervised LightGBM-Quantile tied Chronos-2 on median point error (0.9012 vs. 0.9011 MASE). However, because recursive lag models feed their own smoothed predictions back into future steps, its P10–P90 interval only covered 54.2% of actual traffic (and 26.7% on NYC demand), causing real traffic spikes to breach its P90 auto-scaling ceiling 30.8% to 36.7% of the time.
On noisy mean-reverting server load (AWS ELB), qwen2.5:7b achieved a competitive point error (0.8828 MASE) at the cost of 4.9 seconds per forecast. On strongly cyclical multi-day series with large magnitudes (NYC Demand), the LLM's numeric generation drifted badly (2.8644 MASE vs. 0.4408 for Chronos-Bolt-Small, a 6.5x higher error, at 7.2 seconds per window).
The larger Chronos-2 (120M) won on noisy infrastructure telemetry (AWS ELB and EC2 CPU, hitting 80.8% interval coverage and 0.4883 MASE). On NYC Demand's sharp diurnal cycle, the smaller patch-based Chronos-Bolt-Small (48M) outperformed Chronos-2 (0.4408 vs. 0.9366 MASE and 78.3% vs. 50.4% coverage) while running 3.3x faster.
What happens when the signal lives in the connections between items rather than inside an isolated row? For the third experiment, I compared three approaches to relational data:
I evaluated all six models across three distinct network topologies: Cora Scientific Citation Network (2,708 machine learning papers, 5,278 citation links, 7 topics), CiteSeer Real-World Citation Network (3,327 papers, 4,552 citation links, 3,703 vocabulary features across 6 computer science fields), and a Software Dependency Cascade Graph (1,500 microservices and packages, 4,180 dependency links, where a healthy service can fail because an upstream library 1 to 2 hops away is unstable).
| Model | Relational Paradigm | Hops Used | Weights Size | Accuracy ↑ | Macro-F1 ↑ | ROC-AUC ↑ | Calibration (ECE) ↓ | Latency / Node |
|---|
On Cora, training XGBoost-Flat on node word features alone (ignoring citation links) reached only 57.1% accuracy. Manually engineering 1-hop neighbor averages for XGBoost-1Hop lifted accuracy to 64.3%. Propagating representations across 10 hops with APPNP reached 90.5% accuracy (0.984 ROC-AUC) in 0.003 ms per node.
Standard GCN averages a node's features together with its neighbors, which works well when connected nodes are identical (homophily). In a software dependency graph, a clean downstream service can depend on a broken upstream library. Because GraphSAGE concatenates a node's own telemetry separately from its neighbors' telemetry, it led on dependency cascades with 81.0% accuracy, 0.905 ROC-AUC, and 2.4% calibration error.
To show a local LLM (qwen2.5:7b) a node's 2-hop neighborhood, the graph around each node has to be serialized into a JSON text prompt. Even a compact 2-hop ego-graph required hundreds of tokens per node and 1,001 to 1,441 ms per prediction—over 300,000x slower than a GNN forward pass—while achieving only 45.2% to 47.6% accuracy and 28.5% to 35.2% calibration error.
GCN, GraphSAGE, and APPNP occupy just 0.38 to 0.72 MB in memory—over 6,500x smaller than the 4.7 GB local LLM—because message passing reuses a single compact weight matrix across every node and edge in the graph.
Most production ML models are trained once on a historical batch of data and deployed as static weights. What happens when the underlying world shifts abruptly—such as fraudsters rotating attack vectors or a microservice rollout changing baseline telemetry?
I simulated a 4,500-event live anomaly stream with two abrupt regime shifts (at event t = 1,500 and t = 3,000) and evaluated five models under strict prequential (test-then-train) evaluation: every incoming event is scored first, and online models update their weights on that single event in sub-millisecond time.
| Model | Streaming Paradigm | Footprint | Overall Acc ↑ | Pre-Drift Acc | Regime Shift 1 Acc ↑ | Regime Shift 2 Acc ↑ | ROC-AUC ↑ | Latency / Event |
|---|
Before the first regime shift, Static-XGBoost achieved 88.3% accuracy. The moment the relationship between features and anomalies flipped at t = 1,500, its accuracy plunged to 33.9% (worse than a coin flip) because static trees cannot update their split thresholds without a batch retraining pipeline.
Online-SGD + ADWIN and Adaptive-Hoeffding-Ensemble monitor their own rolling prediction error. When error spikes at t = 1,500 and t = 3,000, ADWIN triggers an immediate reset and adapts to the new regime within ~100 events, maintaining 92.7% and 90.3% overall accuracy at 0.21 to 1.28 ms per event.
In closed-loop control, the model does not predict a static label—its output is a continuous physical force that changes the next state of the system 20 milliseconds later (50 Hz loop). I tested five controllers on stabilizing a non-linear Inverted Pendulum / Cart-Pole hit by two external wind impulses at step 25 (+5.5 N) and step 70 (-6.0 N).
| Controller | Control Paradigm | Footprint | Steps Survived / 100 ↑ | Control Cost (J) ↓ | RMS Tilt Error ↓ | Latency / Step ↓ | Meets 50 Hz (<20ms)? |
|---|
A 50 Hz physical control loop requires an action every 20 ms. LQR (0.0007 ms), PID (0.0003 ms), PPO-Policy (0.0025 ms), and MPC (0.787 ms) easily run in real time. At 362.7 ms per step, qwen2.5:7b takes 18.1x the time budget for a 50 Hz control loop—by the time it finishes generating a JSON token, the pendulum has already fallen.
Even with the simulator waiting for each response, qwen2.5:7b survived only 28 of 100 steps (28% of this single trajectory, not an episode success probability), with quadratic control cost 2,053.76 (20.2x worse than LQR at 101.68 and PPO at 108.43).
Physical systems—fluid flow, shock waves, heat diffusion, weather fields—are governed by partial differential equations (PDEs) and strict conservation laws. I tested four models on solving the non-linear 1D Viscous Burgers' Fluid Shock-Wave Equation (∂tu + u ∂xu = ν ∂xxu), predicting the terminal steepened shock-wave velocity field u(x, T) across 64 spatial points from unseen initial wave profiles u(x, 0).
| Model | Operator / Physics Paradigm | Footprint | Relative L2 Error ↓ | Shock-Front MAE ↓ | Mass Conservation Error ↓ | Rollout Latency ↓ |
|---|
By parameterizing convolution weights directly in Fourier frequency space (torch.fft.rfft), FNO-1D (0.24 MB) achieved 0.92% relative L2 error and 0.0070 shock-front MAE in 0.053 ms—beating an unconstrained black-box MLP (8.36% error) by 9x and the local 7B LLM (56.24% error, 2,907 ms) by 61x.
Adding a mass-conservation residual penalty to a circular convolutional ResNet (PINN-ResNet, 0.18 MB) reduced relative L2 error to 2.56% and cut mass conservation violation in half compared with an unconstrained MLP.
What happens when you flatten a 2D image or spatial sensor grid into spreadsheet columns for XGBoost, or serialize it as an ASCII intensity matrix for a text LLM? I tested four models on 1,797 real optical 2D grids (UCI Optical Sensor Grids upsampled to 16 × 16, 10 classes) under two conditions: Centered Grids and 2-Pixel Shifted Grids (Δx = +2, Δy = +1 pixels).
| Model | Spatial Inductive Bias | Footprint | Evaluation N | Centered Acc (95% CI) ↑ | Shifted Acc (95% CI) ↑ | Translation Drop ↓ | Macro-F1 ↑ | Latency / Grid ↓ |
|---|
This separate Phase 10 artifact reports five configurations. The patch encoder script evaluates 24 grids (and shifts the original 8×8 grids by +1 pixel in both axes); native-model values are copied from Phase 8. The artifact does not include confidence intervals or per-model sample counts. Do not treat it as the same 100-grid comparison.
| Model | Encoding | Centered accuracy | Shifted accuracy | Latency / grid |
|---|
When patterns are perfectly centered, flattening a 16 × 16 grid into 256 tabular columns lets XGBoost reach 95.1% accuracy. But because tree splits hardcode exact column indices (like pixel_134 > 0.5), shifting the object by just 2 pixels caused XGBoost to crash to 27.3% accuracy. With sliding 2D kernels and spatial pooling, ConvNeXt-Micro (0.14 MB) held 98.2% centered and 96.0% shifted accuracy.
When a 2D sensor grid is serialized as rows of ASCII characters in a prompt, qwen2.5:7b scored just 4.2% accuracy at 307.5 ms per grid. Adding an explicit 2D Bounding-Box ROI Patch Encoder (Benchmark 09) reported 12.5% centered (and 8.3% shifted) at 503.3 ms per grid—still far behind a 0.14 MB ConvNeXt-Micro (98.2% centered / 96.0% shifted in 0.048 ms).
A skeptical reader might ask five fair questions about the seven benchmarks above: (1) What if a table mixes numbers and unstructured text? (2) What if the LLM writes Python code to call specialized models instead of guessing numbers in tokens? (3) Do the graph and streaming gaps hold on real-world non-stationary streams and larger graphs? (4) Does Chain-of-Thought or Self-Consistency fix LLM probability calibration? (5) What happens to time and energy when scoring 100,000 items in production?
Real enterprise tables often pair numeric telemetry with free-text notes. On a seeded synthetic PR-triage dataset (N_train = 300, N_eval = 100) where commit messages carry the dominant signal (0.9512 ROC-AUC text-only), I tested whether prompting qwen2.5:7b on raw text + numbers beats extracting 12 compact text-embedding coordinates and feeding the combined table into CatBoost or TabPFN-v2.
| Model / Pipeline | Architecture | Footprint | ROC-AUC ↑ | PR-AUC ↑ | Brier ↓ | Calibration (ECE) ↓ | Latency / Row ↓ |
|---|
I prompted qwen2.5:7b to generate Python code calling TabPFN-v2 or Chronos-Bolt-Small. The evaluation script measures code-generation time, then reuses archived model metrics; it does not execute and validate the generated code. This illustrates delegation, not an end-to-end orchestrator benchmark. These archived values come from an earlier run and differ from the refreshed tables above.
| Domain & Task | Role of Local 7B LLM (qwen2.5:7b) | Primary Quality Metric | Calibration / Coverage | One-Time Codegen | Per-Item Latency ↓ |
|---|
To test what fixes LLM verbalized overconfidence on numeric tables, I evaluated four prompting and calibration modes on 100 stratified held-out NASA JM1 modules (with 12 in-context examples): Few-Shot Direct JSON (T=0.0), Few-Shot Chain-of-Thought, 3-Sample Self-Consistency (T=0.6), and Post-Hoc Platt Scaling.
| Inference Mode | ROC-AUC (95% CI) ↑ | Brier Score ↓ | Calibration Error (ECE) ↓ | Latency / Row ↓ |
|---|
A 300 ms single-item latency feels instant in a chat window. In a batch job or microservice scoring 100,000 items, the gap between specialized models and sequential text prompting becomes a multi-hour wall-clock divide. Note: Energy figures (Wh) are engineering estimates calculated as runtime × assumed active power (8–32 W), not hardware power-meter measurements. Only LogisticRegression, CatBoost, XGBoost, and LightGBM are timed on 100,000 synthetic rows in the scale script. The other six runtimes are hard-coded scale estimates, not measured 100,000-item runs. Rounded zero Wh means below the stored precision, not zero energy.
| Model | Domain | Weights Footprint | Time for 100,000 Items ↓ | Throughput (Items / Sec) ↑ | Est. Energy (Wh, Runtime × Assumed Power) ↓ |
|---|
On the synthetic PR triage task where commit text holds the primary signal (0.9512 ROC-AUC text-only), numbers-only XGBoost is blind to the text (0.5352 ROC-AUC), while prompting qwen2.5:7b on raw text + numbers reaches 0.9004 ROC-AUC (with 20.85% ECE at 444.8 ms/row). Combining compact 12-d text embeddings with numeric features in Hybrid CatBoost wins on accuracy (0.9648 ROC-AUC, 0.8389 PR-AUC, 0.0731 Brier) while running ~1,000x faster than Hybrid TabPFN-v2 (0.042 ms vs. 43.4 ms/row).
In the archived comparison, code generation took 1.5 to 1.8 seconds. Reusing recorded metrics for TabPFN-v2 or Chronos-Bolt-Small gives a tabular calibration comparison from 20.98% down to 5.90% ECE (at 3.54 ms/row batched) and a NYC Demand forecasting comparison from 2.8644 MASE down to 0.4408 MASE (at 6.12 ms/window).
When evaluated on a 100-row stratified slice of deduplicated JM1, TabPFN-v2 achieves 0.7156 ROC-AUC, 4.83% ECE, and 0.1479 Brier in 15.5 ms/row. For the local 7B LLM, Chain-of-Thought hurt both discrimination and calibration (0.4627 ROC-AUC, 45.68% ECE at 542.7 ms/row), whereas fitting a simple 1D logistic Platt calibrator on 50 validation outputs reduced direct-JSON ECE from 23.91% down to 5.61% (Brier 0.2387 → 0.1699) at 247.7 ms/row.
On 3,000 real chronological telemetry windows (AWS ELB + EC2 + NYC), online Adaptive-Hoeffding-Ensemble beat frozen batch XGBoost by +14.8 points (74.6% vs. 59.8%), and on the 3,327-node CiteSeer graph, 2-layer GNNs beat flat XGBoost by +19.5 points (63.9% vs. 44.4%). At 100,000 rows, CatBoost finishes in 5 milliseconds, whereas sequential Ollama prompting takes an estimated 9.7 hours (~309 Wh at 32 W). A hypothetical 20x LLM batching speedup would still leave a large gap here, but vLLM batching was not measured and these models perform different tasks; this is not a controlled production benchmark.
Download complete case-study JSON (All 16 Case Studies · Phases 1–15) →
Benchmarks 01–13 showed why serializing tables, time series, graphs, or control loops into text tokens underperforms specialized non-LLM models. But real production incidents are rarely purely numerical or purely textual: symptoms arrive as high-frequency numbers and graph cascades, root causes hide in messy stack traces and commit diffs, and remediation must obey hard capacity math.
Rather than setting up strawman baselines where a hybrid pipeline wins by default, I pre-registered three hypotheses across the lifecycle of a hyperscale chaos incident (Detect → Triage → Remediate) and designed strong non-LLM and calibrated-LLM baselines explicitly to falsify each hypothesis.
Reads code/config diff when it ships.
Saves 3 simple yes/no risk flags to cache.
• Graph Finder traces call map to Top-3 nodes.
• Risk Scorer combines metrics + cached flags.
• Handles 95% of incidents with zero LLM calls.
Wakes up ONLY on unseen log lines from
the Top-3 suspect services (~1.5s).
Suggests which broken pool to drain.
Only component allowed to move traffic:
1. Capacity Rule: Never push any healthy pool past 96% capacity (0 secondary crashes).
2. Ramp Rule: Shift load gradually so receiving caches & connections warm up.
3. Sanity Rule: Cap unverified log drains so a bad log cannot black out the cluster.
| Chaos Stage & Scenario | Compound Architecture | Empirical Verdict | Key Falsification Finding |
|---|
When a deep storage or microservice node fails in a 40-service call DAG, retry amplification causes 15–20 upstream services to page red simultaneously with higher raw latency Z-scores than the origin. I tested whether Reverse-DAG Personalized PageRank (0.15 ms) filtering the 40 services down to the Top-3 topological candidates for qwen2.5:7b beats a full-alert LLM dump or PageRank + CatBoost (TF-IDF) across both seen log templates and zero-leakage unseen log phrasing with ruled-out subsystems.
| Model / Pipeline | Origin Node Acc ↑ | Unseen Fault Diagnosis ↑ | Unseen Joint Root-Cause Acc ↑ | Seen-Template Joint Acc | Prompt Words / Alert ↓ | Latency / Incident ↓ |
|---|
During a canary rollout, the system has 30 seconds of 8 golden telemetry signals plus a git commit summary to predict whether a latency dip will spiral into a full cascading outage. To test genuine semantic reasoning rather than keyword memorization, the held-out evaluation split includes unseen negation traps—safe unit-test or documentation commits that share exact high-risk keywords from training (e.g., "added unit test verifying distributed mutex lock is never bypassed").
| Model / Pipeline | Architecture Role | ROC-AUC (95% CI) ↑ | PR-AUC ↑ | Brier ↓ | Calibration (ECE) ↓ | Negation-Trap False Alarm ↓ | Latency / Alert ↓ |
|---|
In the flagship closed-loop experiment, a 4-pool cluster (primary-az1 cap 55%, secondary-az2 cap 45%, canary-v2 cap 25%, fallback-lite cap 40%) is hit by three back-to-back chaos regimes over 60 seconds:
Regime A (t=12–25s) CPU noisy-neighbor throttle + retry storm on primary-az1;
Regime B (t=26–42s) a "Poison-Pill" schema bug on canary-v2 that fast-fails 92% of requests with HTTP 500 in 2.0 ms (tricking latency-only load balancers into routing more traffic to the broken canary); and
Regime C (t=43–59s) a Silent Semantic Auth Corruption on primary-az1 that returns fast HTTP 200 OK in 19 ms with 0% numeric 5xx error while corrupting 85% of ledger sessions in the application logs.
| Controller Architecture | Overall Failed Req (%) ↓ | Regime A (CPU Storm) ↓ | Regime B (HTTP 500 Canary) ↓ | Regime C (HTTP 200 Silent Corrupt) ↓ | Avg P99 Latency ↓ | Secondary Outages ↓ | LLM Calls / 60s | Control Loop ↓ |
|---|
A seasoned production engineer reviewing Scenarios 1–3 would immediately ask three hard questions: (1) What happens when failures come from shared infrastructure (NAT gateways, DNS, hosts) or dead nodes that stop sending metrics? (2) What happens when an attacker injects fake "CRITICAL: drain pool" text into error logs via an HTTP header? (3) What happens when shifting traffic in 1 second crashes cold caches on backup pools? Rather than leaving those pushbacks unanswered, I simulated all three attacks and tested the hardened fixes:
| Production SRE Attack Vector | Unhardened Hybrid Result (What Broke) | Architectural Fix Added | Hardened Hybrid Result |
|---|
In Scenario 1, dumping 16–20 alerting services into qwen2.5:7b yielded just 37.5% origin localization (and 32.5% joint accuracy) at 3,453 ms/incident because the LLM got distracted by upstream gateways screaming with retry-amplified timeouts. A 3-iteration Reverse-DAG Personalized PageRank filter pinpointed the exact origin node at 100.0% accuracy in 1.78 ms. Furthermore, when log templates repeated from training history, PageRank + CatBoost (TF-IDF) hit 100.0% joint accuracy in 0.60 ms—falsifying the need for an LLM on routine, seen alerts. The PageRank → LLM hybrid only won when logs contained unseen phrasing and negated clauses (72.5% vs. 15.0% for CatBoost and 32.5% for Pure LLM).
In Scenario 2, having qwen2.5:7b extract a 3-bit semantic risk rubric for TabPFN-v2 or CatBoost achieved 0.9822 / 0.9805 ROC-AUC, 2.41% ECE, and 0.0% false alarms on negation traps (and once cached per commit, Hybrid CatBoost scores telemetry windows in 0.007 ms). By contrast, our Confidence-Gated Cascade hypothesis failed (0.7232 ROC-AUC, 65.2% trap false alarms): when a safe unit-test commit contained risky words like "mutex lock", the fast TF-IDF CatBoost gatekeeper was not uncertain—it was confidently wrong (>80% probability) and never woke up the LLM.
In Scenario 3, adding a numeric HTTP 5xx cost penalty to a zero-LLM Constrained MPC controller completely solved Regime A (0.3% failed) and Regime B's 2ms Poison-Pill Canary (1.1% failed in 2.5 ms/tick)—beating the Hybrid in Regime B (9.7% failed). Why did the Hybrid drop 9.7% in Regime B? Because qwen2.5:7b misread "primary-az1 recovered" in the Regime B alert and hallucinated a quarantine on both canary-v2 and primary-az1! Yet the MPC Safety Shield contained that hallucination with 0 secondary outages by capping traffic at 96% of the remaining pools and gracefully shedding ~18% batch load.
In Regime C—where primary-az1 returned fast HTTP 200 OK responses with silent KMS auth corruption—pure numeric Error-Penalized MPC was completely blind (34.7% corrupted requests). Meanwhile, the Pure Local-LLM Agent read the log and evacuated primary-az1, but naively commanded a [0.0, 0.50, 0.50, 0.0] split that overloaded both secondary-az2 (45% cap) and canary-v2 (24% cap), triggering 30 ticks of secondary cascading outages (56.3% failed, 189.7 ms P99). By letting the LLM set only the quarantine constraint (bound_primary = 0) and letting the MPC solver balance the math across all three healthy pools, the Hybrid cut Regime C failures to 0.8% (3.10% overall, 0 secondary outages, 3 LLM calls).
Download complete case-study JSON (All 16 Case Studies · Phases 1–15) →