# EXPERIMENT_001

## Baseline
- Current normalization path used `int(bool(raw_pass_value))` for each benchmark entry.
- This treated any non-empty string (e.g., `"0"`, `"false"`, `"failed"`) as truthy and overcounted pass rate.
- Baseline reliability risk: pass/fail metrics can be inflated by summary schema drift from other tools or logs.

## Intervention
- Targeted deterministic reliability fix applied to active benchmark post-processor:
  - File changed: `ai_research/benchmarks/normalize_benchmark_summary.py`
  - Added `_coerce_pass(...)` helper with explicit handling for bool/int/float/string inputs.
  - `"passed"`, `"1"`, `"true"`, `"ok"`, `"success"`, `on`, `yes` map to `1`.
  - `"0"`, `"false"`, `"failed"`, `"error"`, `"timed_out"`, `"invalid"` map to `0`.
  - Missing pass value now falls back to `status`: only `status == "passed"` maps to `1`.
- Command path: this module is invoked by `ai_research/run_benchmarks.sh` after run completion.

## After
- Expected effect on pass-rate stability:
  - Remove false-positive pass counts from non-empty string/legacy values.
  - Pass ratio becomes schema-agnostic and deterministic across mixed producers.
- Expected effect on runtime stability:
  - No benchmark execution/runtime change (post-processor only); summary normalization remains O(num_benchmarks) and should be stable.
- Smoke evidence run:
  - Input with mixed values (`"1"`, `"0"`, `"false"`, `True`, `0`, missing, `""`) normalized to `[1, 0, 0, 1, 0, 1, 0, 0]`.

## Diff
- `_normalize_benchmark_entry` now computes `pass` via `_coerce_pass(...)`.
- New `_coerce_pass` helper added with deterministic normalization semantics.
- Existing behavior for `name`, `status`, `runtime_sec`, and `metric` preserved.

## Decision
- Keep if downstream consumers depend on mixed-type `pass` fields from benchmark producers; they now receive stable 0/1.
- Rollback plan: restore previous behavior in `ai_research/benchmarks/normalize_benchmark_summary.py` to
  `normalized["pass"] = int(bool(raw.get("pass", 0)))` and remove `_coerce_pass`.