34 Iterations of a Bounded AIOps Improvement Loop

Classification: not recursive self-improvement

ANXEngine does not autonomously rewrite its own architecture or train a foundation model without bounds. The process studied here is a bounded, agent-assisted MLOps loop: a hypothesis is registered in advance, implemented as the smallest testable change, evaluated against a control, and then kept, reverted, or paused.

The distinction matters. Recursive self-improvement usually implies that a system changes its own improvement process. ANXEngine automates parts of hypothesis testing, backfill, training, and evaluation. Metrics, budgets, and promotion gates are human-defined. A later audit nevertheless found that the weekly cron did not reliably enforce the intended approval boundary and promoted models automatically.

System context and objective

The production path under study scores the daily incident risk of roughly 170 network devices. Operational tickets provide delayed, incomplete labels. The data is heavily imbalanced: real incidents are rare, ticket creation does not always match technical onset, and a useful alert has to arrive before the ticket.

The primary metric is improvement in precision-recall area under the curve over static rules at a one-day prediction horizon. The count and median lead time of true early warnings are co-primary outcomes. Recall@20 and incident-day performance are guardrails. A change is a candidate only when all required conditions hold together.

• Primary: delta PR-AUC over static rules at a one-day horizon.
• Co-primary: count of true early warnings and median lead time.
• Guardrails: Recall@20 and no regression on the incident day.
• Control: champion and challenger are remeasured on the same current data.

Experimental protocol

Each iteration starts with a preregistered hypothesis. The loop cannot switch to a more convenient idea mid-run. Data preparation, training, selection, and reporting are separated. A challenger is evaluated next to a control retrain with unchanged features so that a result is not credited to the new change when it merely came from retraining.

• Register the hypothesis and expected metric before the run.
• Implement the smallest testable change.
• Separate training, gate-evaluation, and report windows in time.
• Measure the challenger, same-data control, and current champion together.
• Return KEEP only when co-primary metrics and guardrails all pass.
• Record KEEP, REVERT, or PAUSE separately from the actual deployment action.

Results from the experiment ledger

The 34 iterations were not 34 model improvements. Most of the value came from falsified hypotheses and gaps found in the evaluation design. The table summarizes the most important decisions; it is deliberately not a leaderboard.

Failure mode 1: training data looked like evidence

The first protocol trained on rows that partly overlapped the later fitness window. The model reached PR-AUC values between 0.72 and 0.76 there. With clean temporal separation, approximately 0.51 remained. This was not a small optimism error; it was a different experiment. Memorization had been reported as predictive performance.

The corrected protocol uses separate training, gate, and report windows. It also runs a control retrain with the old features. Only the difference between challenger and control can be attributed to the hypothesis.

Failure mode 2: a holdout is a consumable resource

Even an untouched holdout loses independence when its results repeatedly influence the next hypothesis. Every inspection leaks information about the window into the development decision. Thirteen tests on the same day revealed not only candidate differences, but three weaknesses in the test harness itself.

The loop therefore received a budget of eight selection queries per window. After that, the window can be used only as gate history. Promotion additionally requires a never-queried fresh slice or two independently positive windows. This is the adaptive-data-analysis problem described by Dwork and colleagues in the reusable holdout work.

Failure mode 3: one metric can win while the system gets worse

A probe/ramp feature family visibly improved ranking: one chronically difficult device moved from rank 37 to rank 7, and Precision@10 rose from 0.50 to 0.70. At the same time, the number of true early warnings fell from four to two. That would have been an operational regression.

ANXEngine therefore uses a conjunctive gate. A higher score cannot compensate for a loss in lead time or recall. The rule prevents the loop from optimizing the easiest metric while damaging the operating objective.

A feature win is not automatically a model promotion

Trajectory and SNMP-health features produced genuine early signals on the enriched development subset: four early warnings at a median 5.5 days versus none for the starting model. The model trained there was still not deployed.

Across the full fleet, the advantage was smaller, the number of emitted flags fell sharply, and Precision@10 became worse. The features were allowed to accumulate more history while the model remained unchanged. Separating feature acceptance from model promotion prevents a local win from being mistaken for a global rollout decision.

Why PAUSE is a normal system state

An autonomous loop without a no-op state treats every run as a request for change. That creates activity, not evidence. The ANXEngine state machine therefore includes GATE_CLEARED, CYCLE_EXHAUSTED, DATA_STALE, BLOCKED, and PAUSED_FRESH_DATA.

When data is not fresh, optimization stops. The next run waits for new labels or a named unblocking event. The intended architecture required human approval for promotions, but the ledger audit found automatic cron promotions. That is an open governance gap: a documented approval rule is not the same as a technically enforced control boundary.

Limitations of this research note

This is not a randomized public benchmark. Results come from one operational system with ticket-based proxy labels. The development subset deliberately contains difficult and data-rich devices, so it is not representative of the full fleet. Positive events are sparse, and ticket timestamps only approximate technical incident onset.

Raw data and the internal ledger cannot be published because they contain customer and operational data. This note documents the protocol, aggregate results, and decision criteria. It does not show that the same features or thresholds will work on another fleet.

• One system and one fleet, with no external replication.
• Delayed and incomplete ticket labels.
• Enriched development subset with selection risk.
• Small positive-event count per window.
• No public dataset and no independent audit.

Reproducible decision framework

The specific data remains internal, but the decision framework transfers. A similar loop should document not only its best model score, but every query, every negative result, and every reason a promotion did not happen.

• Preregister the hypothesis, primary metrics, and guardrails.
• Use temporally disjoint training, gate, and report windows.
• Run a same-data control retrain next to the challenger.
• Budget and record selection queries per window.
• Evaluate local feature wins separately from model promotion.
• Record KEEP, REVERT, PAUSE, and BLOCKED as equal outcomes.
• Technically enforce human approval for promotion and external communication in the execution path.