Skip to content

ADR-0001: Coverage-Weighted Scoring

Status: Superseded by ADR-0012 (2026-05-21) Supersedes: — Superseded by: ADR-0012 (scoring methodology: neutral-prior baseline)

Context

The v0.4 framework methodology assigns a base value to rules without live signals: 50 for direct rules (neutral assumption), 100 for deduction rules (no-deduction-by-default). Pillar and composite scores are specified as weighted averages over all applicable rules, base-valued or otherwise. The implicit assumption is that base values represent a small uncovered tail of a mostly-covered system.

At May 2026 the system has 3 covered rules out of ~27 applicable for the financial-institution pilot — coverage of ~11%. The headline composite is therefore approximately 90% base-value noise and 10% real signal. The displayed number is mathematically valid per v0.4 but semantically meaningless: it conflates "what the system found" with "what the system didn't look at." A user cannot tell whether a 19.2 composite means "this institution is genuinely poor on ESG" or "this institution mostly hasn't been measured yet."

This was first flagged in cycle C (12 May 2026) as a low-coverage-induces-score-inflation problem. It is the score-display equivalent of reporting an average over a sample that's 90% missing data.

Decision

The composite and pillar scores as displayed are computed only over rules with live signals. Base-valued rules are excluded from the headline averages.

displayed_pillar_score    = weighted_avg(covered_rules_in_pillar)
displayed_composite_score = weighted_avg(displayed_pillar_scores, by_pillar_weight)
displayed_coverage_pct    = count(covered_rules) / count(applicable_rules)

Where covered_rules are rules with a non-no_scraper signal and confidence > 0, and applicable_rules are rules where applies_to(institution) is true per peer-group, GICS, and section applicability.

When a pillar has zero covered rules, its displayed score is null and the UI shows "no covered rules" rather than a number. The composite is computed across whichever pillars have at least one covered rule, with pillar weights re-normalised to the covered set.

The 50/100 base values are retained in the data model (score_sub_criterion.base_value) for forward compatibility, per-rule display in the rules table, and methodology audit trails. They are simply no longer included in the headline averages.

Consequences

Positive. - The score becomes interpretable: it answers "of what we know, where does this institution sit." Coverage answers "how much we know." - Coverage becomes a first-class metric in the UI (see ADR-0002), structurally visible rather than a footnote. - Pilot institutions stop scoring identically due to base-value dominance. As scrapers land (see ADR-0003), real differentiation emerges. - Forward-compatible: as coverage rises, the displayed score and the v0.4 raw composite converge. Above ~80% coverage they are within rounding.

Negative. - Two numbers in the headline (score + coverage) is more cognitive load than one. Mitigated by giving them equal visual weight and RAG colour-coding (ADR-0004). - Diverges from the literal v0.4 workbook spec. Acceptable: workbook remains the framework reference, ADR records the implementation choice. - Peer comparison is imperfect when peers have different coverage levels (Barclays at 67/11% vs HSBC at 70/35% — which is "better"?). Peer table shows both score and coverage; a single comparability index is deferred. - At zero coverage the score is null, not zero. UI must handle this rendering case.

Implementation. - Schema migration 011_coverage_weighted_scoring.sql: - score_composite: add composite_raw_v04, composite_coverage_weighted, coverage_pct_overall. Existing composite column becomes alias of composite_coverage_weighted. - score_pillar: add pillar_raw_v04, pillar_coverage_weighted, coverage_pct_pillar, covered_rule_count, applicable_rule_count. - Backfill composite_raw_v04 from existing composite value for historical runs. Recompute composite_coverage_weighted for all existing runs. - Both raw and coverage-weighted scores are stored. UI displays coverage-weighted; JSON export includes both for debugging and audit.

Convergence thresholds (tuning, not load-bearing).

Coverage UI behaviour
0% No score; UI shows "no covered rules"
1–49% Score + coverage given equal visual weight
50–79% Score primary, coverage secondary
≥80% Single composite; raw and coverage-weighted equivalent

Re-evaluate once 3–5 institutions cross each band.

Alternatives considered

  • Drop normalisation until coverage ≥ threshold — too binary; rewards no incremental coverage gain.
  • Tune base values to peer median — self-referential at low coverage (peer median is the base value when most rules are uncovered). Masks differentiation.
  • Show two scores (raw + coverage-weighted) — close to chosen approach but exposes the raw v0.4 number on the page, where it would anchor users to a misleading figure. The raw is kept in the DB and JSON export instead.
  • Keep v0.4 methodology as-is, add caveats in copy — tried at cycle C, doesn't solve the framing problem because the misleading number is still the headline.

References

  • v0.4 workbook, Scoring Methodology sheet
  • Cycle C handoff (12 May 2026)
  • ADR-0002 (UI architecture) — coverage displayed as co-equal hero metric
  • ADR-0004 (RAG thresholds) — colour-coding for both score and coverage
  • Migration 011_coverage_weighted_scoring.sql

ADR-0001 amendment — deduction-rule coverage semantics

Date: 2026-05-19 Status: Accepted (amends ADR-0001, supersedes the ~66.7 verification target)

Context

Migration 011 pre-implementation dry-run (2026-05-19) surfaced that ADR-0001's covered = confidence > 0 semantics were not pinned precisely enough for deduction rules with no scraper run.

CC's worked example for run 3, UK banks:

  • Only three rules feed CW composite under confidence > 0: E1 (NZBA/SBTi membership), E6 (carbon disclosures), G3 (framework tiers)
  • E pillar CW = 13.3, G pillar CW = 60.0, S pillar = null (no covered rules)
  • Renormalised composite = 33.3, not the ~66.7 figure in the mock and original ADR

The 66.7 figure was a pre-coverage design-time estimate that quietly assumed deduction rules at base value (E7/G5/G7/S6, score=100, awaiting BHRRC) would contribute to the CW composite. They don't under confidence > 0 — and they shouldn't.

Decision

Deduction rules with no scraper run remain uncovered until their scraper lands.

Specifically:

  • covered = confidence > 0 stays. No filter change.
  • A deduction rule with score=100 and no signal row is not covered. Its confidence is 0, and that is the correct value.
  • Coverage rises as scrapers run, not as base values are interpreted as evidence.
  • This applies symmetrically to all rule types — there is no special case for deduction rules.

Rationale

Coverage-weighting was introduced (per the 2026-05-13 cycle C handoff) to stop scores being inflated by rules the system hasn't actually checked. Treating a deduction rule with no scraper as "covered at base value" would recreate that inflation problem in a different form: it would count a base value as evidence.

"No controversy found yet" is a real assertion only when a scraper has actually looked. Pre-scraper, score=100 is a default, not a finding.

Consequence

  • UK banks pilot lands at CW composite ~33.3 at 11% coverage, not ~66.7
  • Composite will climb toward 66.7 (and beyond, in either direction) as cycles 4–7 ship BHRRC, UK MSA, UK GPG, BankTrack and PAX scrapers
  • The mock's 66.7 figure is retired
  • Migration 011 verification target updated: UK banks should land at 33.3 ± 1 on score_coverage_weighted post-backfill

Out of scope for this amendment

  • Per-rule coverage policy. A future ADR could allow individual deduction rules to declare base-value-counts-as-covered (e.g. rules keyed to a public register where absence is meaningful). Not pursued for v1 — the simple uniform rule is sufficient and avoids a new axis of configuration this early.
  • Client-facing presentation. v1 is internal. Any v1.5 client-facing summary will need to think through how "33.3 at 11% coverage" reads to an external audience. Out of scope here.

Cross-references

  • ADR-0001 (this ADR, original) — coverage-weighted scoring
  • Migration 011 — coverage-weighted scoring columns (pending application)
  • 2026-05-13 cycle C handoff — coverage_pct introduction and the score-inflation caveat
  • 2026-05-19 migration 011 pre-implementation report (CC, Slack) — worked example