Skip to content

ADR-0012 — Scoring methodology: neutral-prior baseline

Status: Superseded
Date: 2026-05-21
Supersedes: ADR-0001 (coverage-weighted scoring)
Superseded by: ADR-0026 (baseline anchor → 100, 2026-05-24)
Related: ADR-0004 (RAG thresholds), ADR-0010 (universal rule layer), docs/design/coverage-diagnostics.md

Context

The product is a single-purpose ESG screening tool producing go/no-go investment screening decisions. It was re-scoped to this purpose at the cycle 7.5 strategic shift (2026-05-21), which also de-scoped the v0.5 two-stage financials triangulation design.

Through v1.5, the scoring engine used a coverage-as-confidence approach: every institution starts at 0/100; each rule with live evidence contributes points; rules without live signals contribute zero. Coverage % is computed separately and rendered alongside the score as a confidence indicator.

Run #19 (cycle 7 close, 2026-05-21) showed zero institutions above 0.5 confidence. Coverage on some institutions was as low as 6%. Under coverage-as-confidence, an institution with 6% coverage and two negative findings produces a score around 16/100 — visually identical to a well-covered institution with confirmed poor ESG performance.

The conflation is honest in research terms ("we don't know much, what little we know is bad") but broken in screening terms: the user cannot distinguish "this institution is a poor ESG performer" from "we don't have enough data on this institution." At the coverage levels actually observed in the pilot, the current approach produces scores that are systematically uninterpretable for their intended purpose.

Decision

Adopt a neutral-prior scoring methodology. Every institution starts at 50/100 (neutral). Rules with live evidence move the score up (positive signals) or down (deduction signals) from neutral. Rules without live signals do not affect the score.

Coverage is preserved as a separate, first-class measure rendered alongside the ESG score — renamed "Research depth" per cycle 14 surface decisions. Coverage becomes a hygiene measure ("is the system doing enough work to produce a meaningful read?"), not an input to the score itself.

Rationale

Three reasons:

(1) The current methodology breaks at the coverage levels actually observed. Run #19 demonstrated that at coverage as low as 6% on some institutions, coverage-as-confidence produces scores that cannot distinguish "poor ESG performer" from "limited data." For a screening tool whose output drives go/no-go decisions, this is not a fixable calibration issue — it is a category error in what the score represents.

(2) Neutral-prior maps cleanly to the screening use case. Go/no-go screening asks "what does the available evidence say about this institution, weighted from a neutral starting point?" — which is exactly what neutral-prior measures. Coverage-as-confidence asks "how much have we gathered, and what was its sign?" — a different question, useful for research but not for screening.

(3) The research-quality dimension is preserved, not lost. By rendering coverage (now "Research depth") as a first-class sibling measure rather than folding it into the score, the user can read both dimensions independently: what the screen thinks, and how much it had to work with. This is more honest than the current approach, which collapses both signals into one number where neither is fully legible.

Consequences

Positive

  • Scores become directly interpretable as go/no-go evidence-weighted reads from a known neutral baseline.
  • Limited-data institutions are visibly limited-data (low Research depth) rather than mislabelled as poor performers.
  • The two dimensions stay independent; the user can weight them as they see fit in any given screening decision.

Things to watch

  • Held institution scores will change at cutover. The four currently held institutions (Amazon, HSBC Holdings, JPMorgan Chase, Deutsche Bank) will show materially different scores post-shift. This is communicated silently — the portal has zero users beyond the operator (cycle 14 surface decisions).
  • RAG band thresholds need recalibration. The existing ≥75 green / ≥50 amber bands were calibrated against the coverage-as-confidence distribution. Under neutral-prior the score distribution will shift; the bands need to be reset against post-implementation data. Captured as cycle 16 (PRB seed completeness + RAG green band calibration).
  • Cycle 13 role shifts from decision to verification. Cycle 13 runs the final coverage-as-confidence scoring as a baseline and assesses whether the post-build coverage data supports the rationale above. If it does, cycle 14 implements as planned. If coverage surprises (e.g. cycle 11–12 scraper builds lift median coverage above 70%, undermining reason 1), the decision is revisited before cycle 14. This is a contingency, not the default path.
  • Standing test for absence/failure of any signal source (D2 consequence): The neutral-prior contract is: no live signal → no effect; record as coverage, not as a score input. This applies to both rule types and both absence cases — failed extractions (extraction_ok=0) and deduction rules that ran clean with no findings. Either is recorded as a coverage gap (research depth measure); neither moves the score. This is now the standing test for any future "how do we score the absence/failure of X" question. Implementation: raw_score=2.5 omit-sentinel in score_sub_criterion; pillar.js excludes sentinel rows from the weighted mean denominator (cycle 14, D2).

Implementation

  • Cycle 14: scoring engine change from coverage-as-confidence to neutral-prior; "Coverage" → "Research depth" rename on user-facing surfaces; pink-box explainer rewrite to describe the neutral-prior model. Full implementation scope is on /backlog/ under cycle 14 surface decisions.
  • Cycle 16: recalibrate RAG band thresholds against the post-implementation score distribution.

Out of scope

This ADR records the methodology choice. It does not specify the exact scoring formula under neutral-prior — that is cycle 14 implementation detail.

This ADR does not commit to specific RAG band thresholds under neutral-prior — that is cycle 16.

References

  • Cycle 7.5 strategic shift handoff (Slack #esg-screening, 2026-05-21 08:41 BST, TS 1779349271.949059)
  • Cycle 14 surface decisions resolution (2026-05-21)
  • ADR-0001 (coverage-weighted scoring) — superseded by this ADR
  • ADR-0004 (RAG thresholds) — recalibration captured as cycle 16
  • docs/design/coverage-diagnostics.md — coverage diagnostics surface built cycle 4
  • Run #19 results (cycle 7 close, 2026-05-21)