Scraper Register
The system's data sources, their state, and what rules they feed.
This page is the standing reference of intent and design. The
live source-status surface is the /methodology page in the
product, which renders the same register from the DB joined to recent
scrape_run data.
For the runbook on adding a new scraper, see Add a scraper.
For the decision background, see ADR-0003.
Status legend
| Status | Meaning |
|---|---|
| live | Scraper exists, runs successfully on the weekly cron, produces signals |
| pending | Scheduled for an upcoming cycle, prioritised on the backlog |
| planned | On the roadmap but not yet scheduled |
| deprecated | Was live or planned, no longer used |
Live scrapers carry an effective sub-status from recent run history:
| Sub-status | Definition |
|---|---|
| live · healthy | Last 3 runs successful |
| live · degraded | 1–2 failures in last 5 runs |
| live · broken | 3+ consecutive failures |
The product page computes effective status from scrape_run outcomes
nightly. This page records the canonical intent.
Register
As of obs-1 (2026-05-24). The live source-status surface is /methodology in the product.
| Code | Name | Type | Access | Feeds rules | Status |
|---|---|---|---|---|---|
NZBA-MEMBERS |
UNEP FI Net-Zero Banking Alliance | membership list | seed-backed | E1.1 | live · healthy |
SBTI-VALIDATED |
Science Based Targets initiative (financials) | membership list | free Excel download | E1.5, E6.4 | live · healthy |
SBTI-VALIDATED-CORP |
Science Based Targets initiative (all sectors) | membership list | free Excel download | UN-E1.3, UN-E1.4 | live · healthy |
PRB-SIGNATORIES |
UN Principles for Responsible Banking | membership list | seed-backed | G3-PRB.1 | live · healthy |
BHRRC |
Business & Human Rights Resource Centre | NGO database | free HTML + RSS | watchlist | live · healthy |
UK-MSA-REGISTRY |
UK Modern Slavery Act Registry | regulatory register | free HTML | S1.3, UN-S1.3 | live · healthy |
UK-GPG-REGISTRY |
UK Gender Pay Gap Service | regulatory register | free HTML | S4.1 | live · healthy |
CBI-DB |
Climate Bonds Initiative certified bonds DB | NGO database | free HTML | E3.4 | live · healthy |
PAX-WEAPONS |
PAX "Don't Bank on the Bomb" | NGO database | seed-backed (annual PDF) | flag_weapons + watchlist | live · healthy |
BANKTRACK-PROFILES |
BankTrack institution profiles | NGO database | free HTML RSS | E2.4, E7.1, S5.4, S6.3 + watchlist | live · failed |
BOCC-REPORT |
BankTrack "Banking on Climate Chaos" | NGO database | free PDF + data | E2.3 | live · pending |
GOOGLE-NEWS-RSS |
Google News RSS | news feed | RSS | G7.1–G7.3, NF-G4.1–NF-G4.3 | live · healthy |
NGO-TARGETS |
FoE / RAN / ShareAction campaign targets | NGO database | seed-backed (quarterly) | E7.3, S6.2 | live · healthy |
CHRB |
WBA Corporate Human Rights Benchmark | NGO database | seed-backed (biennial XLSX) | NF-S3.3, NF-S4.3 | live · pending (empty seed) |
FOREST500 |
Global Canopy — Forest 500 | NGO database | seed-backed (annual) | NF-E6-STAP.2, E7.4 | live · pending |
TNFD-ADOPTERS |
TNFD adopter list | membership list | seed-backed | E5.1 | live · pending-first-run |
IFRS-S2-MANDATE |
IFRS S2 jurisdiction mandate proxy | regulatory mandate | jurisdiction lookup | E4.4, UN-E1.5 | live · pending-first-run |
TPI-SECTOR |
Transition Pathway Initiative sector reports | NGO database | seed-backed (MQ only) | E1.6–E1.10, UN-E1.6–UN-E1.10 | live · pending-first-run |
ETHICAL-CONSUMER |
Ethical Consumer ratings | NGO database | mixed (paid for detail) | watchlist only | planned |
PROFUNDO-REPORTS |
Profundo published research | commercial reports | free PDF | E5, E6, S5 | planned |
CORP-WEBSITE |
Corporate website scraping | corporate disclosure | corporate HTML/PDF | (multiple) | dropped (ADR-0027: external-only screen) |
IFRS-S2-CORP |
Corporate sustainability disclosure (IFRS S2) | corporate disclosure | corporate PDF | E4.1 | dropped (ADR-0027: external-only screen) |
TCFD-SUPPORTERS |
TCFD supporters list | membership list | — | E4.1 (historical) | deprecated |
Plus 8 candidate sources (CA100PLUS, EP-MEMBERS, EP100, FAIRR, INFLUENCEMAP, RSPO-MEMBERS, WALK-FREE, and others) and 7 deferred sources (CDP-CORPORATE, CDP-PUBLIC, ECB-EU-REGS, FCA-NOTICES, PCAF-SIGNATORIES, SEC-EDGAR, TCFD-SUPPORTERS) classified in cycle 1.6. CORP-WEBSITE and IFRS-S2-CORP are permanently dropped (ADR-0027, 2026-05-24).
Per-source notes
Live sources
NZBA-MEMBERS
UNEP FI Net-Zero Banking Alliance. Seed-backed — the upstream page
at unepfi.org/banking/net-zero-banking is AJAX-loaded and direct HTTP
fetch returns no useful HTML. Members list is maintained in
data/seed/nzba-members.json, updated manually ~quarterly.
is_seed_backed = 1.
Pilot: all 4 UK banks are founding members (April 2021). Doesn't differentiate them.
SBTI-VALIDATED and SBTI-VALIDATED-CORP
Science Based Targets initiative. Live Excel download from
sciencebasedtargets.org. Parsed via xlsx npm package. Columns:
company_name, near_term_status, long_term_status, net_zero_status,
lei, sector. Only "Targets set" treated as validated.
Two source IDs: SBTI-VALIDATED (all-sector corporate target lookup) and
SBTI-VALIDATED-CORP (financial-sector corporate). Both route through
the same sbti.js scraper and share one Excel download cache per run.
Known signals: NatWest has "Commitment removed" status. Unilever SBTi withdrawal also detected as of run 19.
Confidence: 0.5 for companies where no match is found (not confirmed absent), 1.0 for found or confirmed withdrawn.
PRB-SIGNATORIES
UN Principles for Responsible Banking. Seed-backed — upstream page
is server-rendered HTML but the full signatory count is dynamic. Seed
file at data/seed/prb-signatories.json covers ~356 signatories as of
2026-05-20. Refresh script: scripts/refresh-prb-seed.js (~quarterly).
HSBC Holdings plc confirmed absent (founding Sep 2019 — likely quiet
administrative exit). DBS Bank Ltd found as founding member via
operating_arm_lei inheritance. PRB LEI gaps: 203 of 356 entries have
lei=null and fall back to match_name.
BHRRC
Business & Human Rights Resource Centre. Live scraper — per-institution
allegation list from the BHRRC Explore API. Feeds watchlist entries only
(no scored rules directly — watchlist entries feed deduction rules E7, S6,
G5, G7 indirectly via the scoring engine). BHRRC company profiles are
linked via institution.bhrrc_pk.
FETCH_LIMIT=500: several large institutions (Amazon, Unilever, VW) have
500 allegations. Oldest items may be missed. Pagination is a parked backlog item.
UK-MSA-REGISTRY
UK Modern Slavery Act Statement Registry. Government register of modern-slavery statements filed by UK companies. Feeds S1.3 (financials) and UN-S1.3 (all sectors). Non-GB institutions emit a no-data signal (confidence=0.0) per the UN-S1 fallback chain (ADR-0010 amendment 1).
UK-GPG-REGISTRY
UK Gender Pay Gap Service. Government register of gender pay gap reports. Feeds S4.1. UK-only — non-GB institutions are skipped (scrape_skip table, migration 016).
PAX-WEAPONS
PAX "Don't Bank on the Bomb". Seed-backed — dontbankonthebomb.com
renders a 6.6KB navigation shell with no institution names in HTML.
Seed file data/seed/pax-hall-of-shame.json compiled from the 2025
"At Great Cost" report (Tables 5+6). Annual refresh when the new
report publishes (typically Feb–Mar).
As of 2025 report: 49 institutions listed, including all 4 UK pilot banks
and most major international banks. AIB Group and Bank of Ireland are
not listed. Feeds institution.flag_weapons (tripped/clear) and
watchlist entries for each tripped financial. Financials-only.
BANKTRACK-PROFILES
BankTrack institution profile pages and RSS feeds. Live scraper —
intermittently failing (source_status=failed) due to server-side
outages on banktrack.org. Feeds E2.4 (project finance E&S), E7.1
(deduction: project finance controversies), S5.4 (conflict finance),
S6.3 (deduction: social controversies). Financials-only.
Institutions without a BankTrack profile (banktrack_slug=null —
currently UniCredit and Bank of Ireland) are skipped; this reduces
their coverage_pct vs institutions where BankTrack attempts but fails.
CBI-DB
Climate Bonds Initiative certified bonds database. Feeds E3.4 (green
bonds certified by CBI). Live scraper. Returns not_found for most
institutions — CBI-certified bonds are a strict subset of all green
bonds; result may be correct pending manual verification.
BOCC-REPORT
BankTrack "Banking on Climate Chaos" annual report. Stub scraper live
(source_status=pending — live scraper, no 2026 report yet published).
Expected March–April 2026. Will auto-populate E2.3 on next run after
publication. Financials-only.
CHRB
World Benchmarking Alliance — Corporate Human Rights Benchmark. Seed-backed.
- Cadence: biennial; 5th iteration is 2022 + 2023 combined (different sectors each year). 6th iteration expected 2026.
- Coverage: ~110–130 companies per cycle across 5 sectors (apparel, extractives, food & agri, ICT manufacturing, automotive).
- Scoring: 0–100 native scale. Poor-performance threshold:
chrb_score_rounded < 40. - Feeds: NF-S3.3 (positive: CHRB coverage), NF-S4.3 (deduction: poor performance).
- Seed:
data/seed/chrb-companies.json. Refresh script:scripts/refresh-chrb-seed.js.is_seed_backed = 1. Non-financials only. - Decision note: Social Benchmark substitution rejected — see ADR-0014. CHRB remains distinct from WBA's Social Benchmark and Gender Benchmark instruments.
FOREST500
Global Canopy — Forest 500. Seed-backed. Both sides compiled C11.5b (2026-05-22).
- Cadence: annual. Company and finance rankings are published on different cycles
(company ~April, finance ~August of following year). Dual vintage is intentional and
recorded per-entry via
assessment_yearin the seed. See ADR-0015. - Coverage: Top 350 companies + top 150 financial institutions, selected by greatest
exposure to forest-risk commodities (beef, cocoa, coffee, leather, palm oil, pulp &
paper, rubber, soy, timber). This is a commodity-exposure-weighted universe — most of
our financial/tech/industrial universe is correctly absent (not a coverage gap). Absence
list recorded in
data/seed/forest500-companies.json_absent_from_forest500. - Current seed: 5 company entries (2025 assessment) + 24 finance entries (2024 assessment, republished 2025-08-14). Includes in-universe institutions + near-term expansion targets (cycles 15a–15d per ADR-0013).
- Feeds: NF-E6-STAP.2 (positive: company appearance, non-financials; scoring engine
applies rule only for sector-30 institutions until
applicable_sectorsis broadened), E7.4 (deduction: financial institution appearance in top-150 financiers list). - Scoring thresholds: E7.4 fires on presence only — being in the top-150 is the adverse finding regardless of F500 score. NF-E6-STAP.2 fires on presence (coverage signal). Score-based thresholds deliberately deferred to post-cycle-15 calibration (ADR-0015 §b).
- FI category weighting: Implementation dominates FI scoring (>56% of max); FI and company scores are not on a common scale and must not be compared across sides.
- Instrument distinction: Forest 500 ≠ Floresta 250 (Cattle) — a separate Global Canopy instrument for Brazilian cattle supply chains. Different universe; different scope. Do not conflate (same pattern as CHRB vs WBA Social Benchmark — ADR-0014).
- Seed:
data/seed/forest500-companies.json(replacedforest500-entities.jsonC11.5b). Refresh script:scripts/refresh-forest500-seed.js.is_seed_backed = 1. - LEI note: FI-side LEIs are operator-supplied candidates; verify against GLEIF on
ingest. Company side and DBS/Invesco/TIAA entries are name-fallback (
lei=null).
TNFD-ADOPTERS
Taskforce on Nature-related Financial Disclosures adopter list. Seed-backed — the upstream
page at tnfd.global/engage/adopters/ is dynamic. Seed file:
data/seed/tnfd-adopters.json. Feeds E5.1 (biodiversity & nature risk disclosure-presence).
Financials only. is_seed_backed = 1.
Status: live · pending-first-run. Will transition to healthy after the first Sunday scheduled
run completes. Until then, source_status reflects the first-run-pending state — not a defect.
IFRS-S2-MANDATE
IFRS S2 jurisdiction mandate proxy. Feeds E4.4 (climate disclosure mandate, financials) and
UN-E1.5 (TCFD-aligned disclosure, all sectors). Rather than scraping corporate PDFs, this
source uses a jurisdiction-level lookup: if the institution's domicile is in a jurisdiction with
a live IFRS S2 mandate (currently GB, AU, NZ, BR, SG), the signal is boolean=1, confidence=1.0.
This is a jurisdiction proxy, not a direct corporate disclosure scraper — hence "mandate proxy" in the name. CORP-WEBSITE (individual PDF scraping) was evaluated and permanently dropped for this signal: see ADR-0027.
Status: live · pending-first-run. Source_status was briefly live in migration 054 (a mistake),
corrected to pending in migration 056. Trigger guard added (migration 056) to enforce vocabulary.
TPI-SECTOR
Transition Pathway Initiative — sector-level Management Quality (MQ) scores. Seed-backed —
TPI publishes company-level assessments in Excel format (not the CSV originally anticipated).
Seed file: data/seed/tpi-sector-mq.json. Feeds E1.6–E1.10 (financials: MQ levels 1–5) and
UN-E1.6–UN-E1.10 (all sectors: same staircase). MQ-only — Carbon Performance scores are not
used.
Identity map: staircase sub-rules (≥L1 through ≥L5), each binary. Raw_score = MQ level, so a company at MQ Level 3 fires E1.6, E1.7, E1.8 (but not E1.9, E1.10). Score ceiling: SBTi target (E1.5) + TPI Level 3 = 3+2 = 5 pts → ceiling (100/100). See ADR-0025.
47 of 79 institutions matched in the seed as of 2026-05-24. is_seed_backed = 1.
Status: live · pending-first-run. Transition to healthy after first Sunday cron run.
Dropped sources (permanent)
CORP-WEBSITE
Corporate website scraping (HTML/PDF parsing of institutional sustainability reports). Permanently dropped (ADR-0027, 2026-05-24). Rationale: the system is designed as an external-only screen — all signal sources should be public third-party registries, NGO databases, or regulatory registers, not scrapers that crawl corporate-controlled content. Several sub-rules that listed CORP-WEBSITE as their source were de-scoped; remaining CORP-WEBSITE-sourced sub-rules either have third-party equivalents or are deferred.
IFRS-S2-CORP
Corporate PDF scraping against the IFRS S2 standard. Permanently dropped (ADR-0027, 2026-05-24).
Replaced by the jurisdiction mandate proxy (IFRS-S2-MANDATE) for the mandate-confirmation use
case. Deep PDF parsing of corporate sustainability reports may be reconsidered in a future phase
but is out of scope for Phase 1.
Planned / deferred sources
ETHICAL-CONSUMER
Ethical Consumer's published ratings.
Used as a watchlist source only, not a score input. Their published ratings are valuable as third-party context but their methodology is opinionated (blanket negative treatment of fossil and tax-haven exposure). Folding their judgements into our composite would make our screen a thin veneer over theirs.
The Methodology page in the product displays Ethical Consumer ratings alongside our own in a "Third-party views" panel — never blended into the composite.
PROFUNDO-REPORTS
Profundo published research.
Lower priority because each report is a one-off PDF parse, and underlying data often overlaps with BankTrack. Useful for deep enrichment on specific issues, not for headline coverage growth.
GOOGLE-NEWS-RSS
Feeds G7.1–G7.3 (financials) and NF-G4.1–NF-G4.3 (non-financials) — governance controversy deduction rules. Built in cycle 9b; live · healthy.
Per-institution query strategy: one Google News RSS query per institution using the legal name (quoted) plus a nine-term governance-controversy keyword set. Results filtered to 24-month window.
Relevance check: institution name must appear in the cleaned article title (publication suffix stripped). Strict option chosen to minimise false positives on first run.
Sub-rule classification (compound-phrase matching on cleaned title + stripped description): - .1 exec-departure words AND regulatory-context words (AND condition) - .2 shareholder revolt compound phrases only (avoids generic "vote against") - .3 audit/accounting irregularity compound phrases
Watchlist entries: items passing title-presence check plus a specific controversy keyword (investigation, fraud, scandal, misconduct, lawsuit, fine, penalty) OR a sub-rule phrase match. RSS response cached per institution; watchlist pass reads from cache — no double-fetch.
Rate limit: 1500 ms inter-institution. Finding ID: sha256 of instId + cleaned title + pubDate, first 24 hex chars (stable across runs).
Other rules referencing GOOGLE-NEWS-RSS in the rule table (NF-E4.1, NF-E4.3, NF-S4.2) are deferred to a later cycle; feeds_rules reflects only what the scraper currently writes (G7, NF-G4).
Deprecated sources
TCFD-SUPPORTERS
TCFD task force was disbanded October 2023. Supporters list explicitly
removed from fsb-tcfd.org November 2023. Monitoring transferred to
IFRS Foundation / ISSB under the IFRS S2 standard.
The framework's successor (IFRS S2) is a different shape of signal:
disclosure quality, not framework membership. See IFRS-S2-CORP
(priority 8) for the replacement.
This source remains in the register as deprecated for historical
context. The G3 evaluator was previously reading TCFD membership as
its third framework input — replaced by PRB during the 13 May cycle.
Notes on source-type taxonomy
The source_type field uses these values:
| Value | Examples |
|---|---|
membership_list |
NZBA, SBTi, PRB |
regulatory_register |
UK MSA, UK GPG |
ngo_database |
BHRRC, BankTrack, PAX, Ethical Consumer |
news |
Google News RSS |
corporate_disclosure |
IFRS-S2-CORP (corporate PDFs) |
commercial_reports |
Profundo |
The distinction matters because different source types have different reliability profiles and cadences. Membership lists are high-confidence boolean; NGO databases are noisier and need allegation-parsing; news feeds are high-noise and require LLM-grade filtering (deferred).
Access-mechanism taxonomy
| Value | Examples |
|---|---|
free_html |
Most regulatory registers, NZBA (if scrapeable), PRB (if scrapeable) |
free_api |
(none current) |
free_pdf |
BankTrack, PAX, IFRS-S2-CORP |
free_excel |
SBTi |
rss |
Google News RSS, BHRRC RSS feed |
seed_backed |
NZBA (AJAX-loaded upstream), PRB (AJAX-loaded upstream) |
paid_subscription |
Ethical Consumer (partial) — paid for detail |
mixed |
Ethical Consumer (some free, some paid) |