Data Sources

KindTech wraps two UK public data APIs that aren't prominently documented for developers. This page documents the APIs, their quirks, and how kindtech's ingestion scripts work.

ONS Open Geography Portal (ArcGIS)

The Open Geography Portal serves UK boundary data as an Esri ArcGIS Hub deployment under the ONS organisation ID ESMARspQHYMw9BZ9. It was launched in July 2020, though the ONS Geography team has used ArcGIS Online since June 2015.

Endpoints

Endpoint	Description
`https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services?f=json`	Service catalog (JSON)
`.../{service_name}/FeatureServer/0/query?where=1%3D1&outFields=*&f=geojson`	Query features as GeoJSON
`https://www.arcgis.com/sharing/rest/portals/ESMARspQHYMw9BZ9?f=json`	Organisation metadata
`https://www.arcgis.com/sharing/rest/community/users/ONSGeography_data?f=json`	User profile

Standard ArcGIS REST API. No authentication needed. Licensed under OGL v3.0. ONS does not publish their own API documentation — they rely on Esri's standard reference.

Catalog discovery

The portal's services listing page renders with client-side JavaScript — HTML scraping returns an empty <ul>. Appending ?f=json returns a machine-readable JSON catalog of ~3,700 services (~1,200 FeatureServers plus MapServers, lookup tables, and other types).

Rate limiting

The JSON endpoint is intermittently rate-limited. Some requests return an empty services array. If ingestion returns 0 services, wait and retry.

Service naming conventions

The catalog uses two naming conventions, both following the GSS Coding and Naming Policy (implemented 1 January 2011):

Short form (post-2021):

LAD_DEC_2024_UK_BGC
│   │   │    │  └── Resolution: Generalised Clipped Boundaries
│   │   │    └───── Coverage: United Kingdom
│   │   └────────── Year: 2024
│   └────────────── Month: December
└────────────────── Geography: Local Authority Districts

Long form (pre-2021):

Local_Authority_Districts_December_2024_Boundaries_UK_BGC

The transition happened around 2021 (see Boundary Dataset Guidance: 2021 Onwards).

Resolution codes

The ONS digital boundaries page defines five standard resolutions:

Code	Full name	Detail
BFE	Full resolution, Extent of the Realm	Mean Low Water mark
BFC	Full resolution, Clipped to coastline	Mean High Water mark
BGC	Generalised Clipped	20m tolerance
BSC	Super Generalised Clipped	200m tolerance
BUC	Ultra Generalised Clipped	500m tolerance

Older ArcGIS service names use rearranged abbreviations:

Old suffix	Current suffix
FCB	BFC
FEB	BFE
GCB	BGC
SGCB	BSC
UGCB	BUC

These old codes don't appear in official ONS documentation — they are an ArcGIS-internal convention. The ingestion normalises all of them to the current standard.

Catalog quirks

Historical datasets (1921-present):

Service	Year	Geography
`CTRY_DEC_1921_GB_BGC`	1921	Countries
`CTY_DEC_1921_EW_BGC`	1921	Counties
`ED_1971_EW_BGC`	1971	Enumeration Districts
`ED_1981_EW_BGC`	1981	Enumeration Districts
`WD_1998_UK_BGC`	1998	Wards

The 1921 boundaries were digitised around the 2021 Census centenary. The 1971/1981 entries are Enumeration Districts (EDs) — operational units used for censuses 1961-1991, each representing one enumerator's workload. EDs were superseded by Output Areas for the 2001 Census.

May vs December releases:

Administrative boundaries change during the year. May boundaries reflect changes taking effect on local election day (first Thursday in May). December boundaries provide a year-end snapshot aligned with the 1 December reference date used for electoral statistics since 2001.

Composite-key duplicates (9 cases):

Short-form + long-form names for the same dataset — harmless duplicates
NHSER variants — "NHS England Regions" (nhser18cd) vs "NHS England Region Local Offices" (nhsrlo18cd), different administrative levels. Local Offices were created in 2015 (from 27 Area Teams of 2013) and abolished by April 2020
OA RUC variants — Output Areas with and without the Rural-Urban Classification (12 vs 17 columns). The RUC, first introduced in 2004, classifies OAs as urban or rural
Trailing underscore typos — stripped during ingestion

All entries are kept in the catalog. No skip lists, no manual curation.

Geo ingestion

Parses ~3,700 ArcGIS services down to 615 boundary datasets across 24 geography types, spanning 1921-2025. Stdlib-only (csv, re, requests).

uv run python -m kindtech.geo._ingestion

NOMIS API

NOMIS is a web-based database of labour market and population statistics, operated by the University of Durham on behalf of ONS since 1981. Licensed under the Open Government Licence.

Endpoints

Endpoint	Description
`dataset/def.sdmx.json`	Bulk listing of all datasets (SDMX JSON)
`dataset/{id}.data.csv?{params}`	Data download as CSV
`contenttype/sources.json`	Source groupings
`dataset/{id}.overview.json`	Dataset overview
`dataset/{id}/{concept}/def.sdmx.xml`	Dimension metadata (SDMX-ML)

Base URL: https://www.nomisweb.co.uk/api/v01/

Full API documentation

SDMX format

NOMIS uses SDMX 2.0 structures (KeyFamilies, Dimensions, Codelists). The .sdmx.json format is a NOMIS-specific convenience — not the standard SDMX-JSON from SDMX 2.1 (ISO 17369). The standard format is XML (.sdmx.xml).

Rate limits and authentication

Access level	Limit
Guest (unauthenticated)	25,000 cells per request
Authenticated (with UID)	No cell limit
KML / RSS formats	1,000 cells

Authentication is free — create an account on nomisweb.co.uk and get a UID from "my account" > "web services".

Cell limit

The limit is 25,000 cells (not rows). If your result has exactly 25,000 rows, it's likely truncated. Pass a NOMIS UID to retrieve the full table.

Source annotations in the bulk listing

The bulk listing (dataset/def.sdmx.json) includes source annotations embedded in each dataset's annotations array:

{
  "id": "NM_1_1",
  "name": {"value": "Jobseeker's Allowance with rates and proportions"},
  "annotations": {
    "annotation": [
      {"annotationtitle": "contenttype/sources", "annotationtext": "jsa"},
      {"annotationtitle": "LastUpdated", "annotationtext": "2025-01-14"}
    ]
  }
}

1,572 out of 1,615 datasets have the contenttype/sources annotation in the bulk listing. The remaining 43 genuinely lack source information.

Catalog quirks

31 datasets from the 1961 Census (NM_1230-NM_1257) — digitised around January 2021, covering England and Wales at district, county, parish, ward, and enumeration district levels
1981/1991 Census data — small area and workplace statistics
1968 SIC VAT registrations — industrial classification data

NOMIS is a long-term statistical archive, not just current data.

NOMIS ingestion

Makes one HTTP request to dataset/def.sdmx.json, extracts IDs, names, and source annotations. Completes in ~6 seconds for 1,615 datasets. Stdlib-only (csv, requests).

uv run python -m kindtech.ons._ingestion

Community packages

No official client libraries from NOMIS, ONS, or Durham University. All wrappers are community-built:

Package	Language	Notes
nomisr	R	Evan Odell, rOpenSci. JOSS (2018). Removed from CRAN July 2025
UKCensusAPI	Python	virgesmith
nomisweb	Python	Tony Hirst
Consensus	Python	Ilkka-LBL

postcodes.io

postcodes.io is a free, open-source REST API that resolves UK postcodes to administrative and statistical geographies. KindTech's postcodes module wraps it so address data joins to the same geography_code used by the geo and ONS modules.

Endpoints

Base URL: https://api.postcodes.io

Endpoint	Description
`POST /postcodes`	Bulk lookup, up to 100 postcodes per request
`GET /postcodes/{postcode}`	Single postcode lookup
`GET /postcodes?lon={lon}&lat={lat}`	Reverse geocode — nearest postcodes to a point
`GET /outcodes/{outcode}`	Outcode info: spanned Local Authorities + centroid

No authentication is required. KindTech batches bulk lookups at 100 per request and preserves input order.

Returned geography codes

A postcode lookup returns a codes object with standard ONS GSS codes. KindTech surfaces these aligned to its geography types:

KindTech level	postcodes.io `codes` key	Example
LSOA	`lsoa21`	`E01034394`
MSOA	`msoa21`	`E02007008`
OA	`oa21`	`E00182613`
LAD	`admin_district`	`E09000023`
WD (ward)	`admin_ward`	`E05013727`
ICB	`icb`	`E54000030`
TTWA	`ttwa`	`E30000234`

2021 Census geographies (lsoa21, msoa21, oa21) are preferred so codes match Census 2021 statistics and current boundary data.

Licensing

postcodes.io code is MIT licensed; the underlying data is Open Government Licence v3.0, containing OS data © Crown copyright and database right, Royal Mail data © Royal Mail copyright and database right, and National Statistics data © Crown copyright and database right. Maintained by Ideal Postcodes.

UK deprivation indices (IMD)

KindTech's imd module serves each nation's official deprivation index for single-nation work, and the mySociety composite for comparing across nations. The official per-nation indices are the primary path; the composite exists only because there is no official UK-wide index.

Official national indices

A single-nation call fetches that nation's official index directly from its government source (not the composite):

`load_imd(nation=...)`	Index	Geography	Source	Format
`"England"` (default `year=2025`)	IoD 2025 (30 Oct 2025)	LSOA 2021	gov.uk File 7	CSV
`"England", year=2019`	IoD 2019	LSOA 2011	gov.uk File 7	CSV
`"Wales"`	WIMD 2019	LSOA 2011	gov.wales	ODS
`"Scotland"`	SIMD 2020v2	Data Zone 2011	gov.scot	XLSX
`"Northern Ireland"`	NIMDM 2017	SOA 2001	Open Data NI	CSV

England's File 7 carries all seven domains as score + rank + decile plus a population denominator. Wales/Scotland/NI publish ranks (1 = most deprived), so KindTech surfaces the overall imd_rank, a within-nation imd_decile derived from it, and a <domain>_rank per domain — with nation-specific domain sets (Wales adds community safety + physical environment, Scotland adds crime + a population denominator, NI adds living environment + crime & disorder). All sources are fetched live and cached for the session; the spreadsheet sources (Wales ODS, Scotland XLSX) are read with python-calamine.

Pending: load_imd(nation="Wales", year=2025) (WIMD 2025, 27 Nov 2025) is not wired up — StatsWales has no stable machine-readable download endpoint yet. Scotland and NI have no 2025 release.

Licensing: Open Government Licence v3.0 — England IoD (MHCLG), WIMD (Welsh Government), SIMD (Scottish Government), NIMDM (NISRA / Open Data NI).

Composite UK IMD

All single-nation rankings are within-nation — an English "decile 1" and a Scottish "decile 1" are not the same thing, because each official index ranks areas only within its own nation, on different geographies and methodologies:

Nation	Index	Publisher	Geography
England	IMD 2019	MHCLG	LSOA (2011)
Wales	WIMD 2019	Welsh Government	LSOA (2011)
Scotland	SIMD 2020	Scottish Government	Data Zone (2011)
Northern Ireland	NIMDM 2017	NISRA	Super Output Area

To compare across nations you need one shared ranking, and there is no official UK-wide index. So for nation="UK" KindTech uses the composite UK Index of Multiple Deprivation — a research dataset by mySociety — which re-ranks every area (42,619 in total) onto one scale using the income and employment domains shared by all four indices.

Item	Value
File	`data/uk_index/UK_IMD_E.csv` (England-anchored)
Host	`raw.githubusercontent.com/mysociety/composite_uk_imd`
Rows	42,619 (England 32,844 · Scotland 6,976 · Wales 1,909 · NI 890)
Key column	`lsoa` — LSOA/Data Zone/SOA code depending on nation
UK ranking	`UK_IMD_E_rank`, `UK_IMD_E_pop_decile`, `UK_IMD_E_pop_quintile`

Fetched live and cached in-process for the session (a static research release); column names are mapped to KindTech's conventions (lsoa → geography_code).

Licensing: the composite is Creative Commons Attribution 4.0 — attribute mySociety when redistributing. The underlying national indices are Open Government Licence v3.0; the composite code is MIT licensed.

Dataset Aliases

Instead of remembering NOMIS dataset IDs like NM_2002_1, you can use friendly aliases:

from kindtech import load_ons

# These are equivalent
df = load_ons("population", geography_type="LAD", time="latest")
df = load_ons("NM_2002_1", geography_type="LAD", time="latest")

Raw NM_* IDs are passed through unchanged — aliases are just a convenience.

Available aliases

Alias	NOMIS ID	Description
`population`	NM_2002_1	Mid-year population estimates
`population_by_age`	NM_2002_1	Same as `population`
`population_by_age_band`	NM_31_1	Population by broad age band
`population_lsoa`	NM_2014_1	Population estimates (LSOA level)
`jsa`	NM_1_1	Jobseeker's Allowance
`claimant_count`	NM_162_1	Claimant count
`annual_population_survey`	NM_17_1	Annual Population Survey
`earnings`	NM_30_1	Annual Survey of Hours and Earnings
`jobs_density`	NM_57_1	Jobs density
`vacancies`	NM_19_1	Vacancy survey
`vat_registrations`	NM_29_1	VAT registrations/deregistrations
`census_2021`	NM_2021_1	Census 2021
`census_2011`	NM_144_1	Census 2011
`census_2001`	NM_58_1	Census 2001

To list aliases programmatically:

from kindtech import list_dataset_aliases

for a in list_dataset_aliases():
    print(f"{a['alias']:30s} → {a['dataset_id']}")

Geography Crosswalk

The geo module uses ONS geography type codes (LAD, LSOA, MSOA, etc.) while NOMIS uses internal TYPE codes (TYPE424, TYPE151, etc.). KindTech maps between them so you don't have to.

Using `geography_type` in `load_ons()`

Instead of looking up NOMIS TYPE codes, pass the same geography type you use with load_geodata():

from kindtech import load_geodata, load_ons

# Same geography concept, different APIs
boundaries = load_geodata(geography_type="LAD", year="2024")
statistics = load_ons("NM_1_1", geography_type="LAD", time="latest")

The raw geography="TYPE424" parameter still works if you need it.

TYPE code mapping

NOMIS assigns different TYPE codes to the same geography at different time points. KindTech resolves these automatically based on the time parameter:

Geography	Year range	NOMIS TYPE
LAD	2023+	TYPE424
LAD	2021-2022	TYPE431
LAD	2019-2020	TYPE434
LAD	2015-2018	TYPE446
LAD	pre-2015	TYPE464
CTYUA	2023+	TYPE423
CTYUA	2021-2022	TYPE431
CTYUA	2015-2020	TYPE446
CTYUA	pre-2015	TYPE463
LSOA	2021+	TYPE151
LSOA	pre-2021	TYPE304
MSOA	2021+	TYPE152
MSOA	pre-2021	TYPE305
RGN	all	TYPE480
CTRY	all	TYPE499
WD	2025+	TYPE182
CAUTH	2025+	TYPE442
TTWA	2011+	TYPE447
TTWA	pre-2011	TYPE444
ITL	2025+	TYPE419
ITL	2021-2024	TYPE421

Source: NOMIS geography dimension metadata. To list all mappings programmatically:

from kindtech import list_geography_mappings

for m in list_geography_mappings():
    print(m)

Join keys

Both APIs return standard ONS geography codes (e.g. E06000001):

ArcGIS returns fields like LAD24CD (code) and LAD24NM (name), where the field name includes a 2-digit year suffix
NOMIS returns GEOGRAPHY_CODE and GEOGRAPHY_NAME columns

KindTech normalises both sources so they share a common geography_code column:

load_ons(normalize=True) (the default) lowercases NOMIS columns, producing geography_code and geography_name
geodata_to_properties() maps year-stamped ArcGIS fields (e.g. LAD24CD) to geography_code and geography_name

from kindtech import load_geodata, load_ons, geodata_to_properties
import pandas as pd

geojson = load_geodata(geography_type="LAD")
geo_df = pd.DataFrame(geodata_to_properties(geojson, "LAD", 2024))
ons_df = load_ons("population", geography_type="LAD", time="latest")
merged = geo_df.merge(ons_df, on="geography_code")

Under the hood, geo_code_field() and geo_name_field() derive the ArcGIS field names:

from kindtech._mapping import geo_code_field, geo_name_field

geo_code_field("LAD", 2024)  # "LAD24CD"
geo_name_field("LAD", 2024)  # "LAD24NM"

Architecture

KindTech ingests from two completely separate APIs into two independent CSV catalogs. They share no data or endpoints — one provides geographic boundaries (maps), the other provides statistical tables (numbers).

┌─────────────────────────────────────────────────────────────────────┐
│                        TWO SEPARATE DATA SOURCES                    │
├──────────────────────────────┬──────────────────────────────────────┤
│                              │                                      │
│  ONS Open Geography Portal   │  NOMIS                               │
│  (ArcGIS FeatureServer)      │  (Durham University for ONS)         │
│  geoportal.statistics.gov.uk │  nomisweb.co.uk                      │
│                              │                                      │
│  What: UK boundary maps      │  What: UK statistics                  │
│  Format: GeoJSON polygons    │  Format: CSV tabular data             │
│                              │                                      │
├──────────────────────────────┼──────────────────────────────────────┤
│                              │                                      │
│  INGESTION (dev-time)        │  INGESTION (dev-time)                 │
│                              │                                      │
│  Source URL:                 │  Source URL:                           │
│  services1.arcgis.com/       │  nomisweb.co.uk/api/v01/              │
│    ESMARspQHYMw9BZ9/         │    dataset/def.sdmx.json              │
│    arcgis/rest/services      │                                      │
│    ?f=json                   │  1 HTTP request → parse JSON          │
│                              │  → extract id, name, source           │
│  1 HTTP request → parse JSON │    from annotations                   │
│  → regex-parse ~3,700        │                                      │
│    service names             │  Output:                              │
│  → normalise resolution      │  ons/data/nomis_tables.csv            │
│    codes                     │  (1,615 datasets)                     │
│                              │                                      │
│  Output:                     │  uv run python -m                     │
│  geo/data/arcgis_services.csv│    kindtech.ons._ingestion            │
│  (615 boundary datasets)     │                                      │
│                              │                                      │
│  uv run python -m            │                                      │
│    kindtech.geo._ingestion   │                                      │
│                              │                                      │
├──────────────────────────────┼──────────────────────────────────────┤
│                              │                                      │
│  RUNTIME (user-facing)       │  RUNTIME (user-facing)                │
│                              │                                      │
│  load_geodata("LAD")         │  load_ons("population",              │
│    → look up CSV catalog     │           geography_type="LAD")       │
│    → query ArcGIS            │    → resolve alias → NM_2002_1        │
│      FeatureServer           │    → resolve geography → TYPE424      │
│    → return GeoJSON dict     │    → query NOMIS .data.csv            │
│                              │    → return DataFrame                 │
│                              │      (pandas or polars)               │
│                              │                                      │
└──────────────────────────────┴──────────────────────────────────────┘

Both ingestion scripts are stdlib-only (csv, re, requests). The resulting CSV catalogs ship with the package. Users never run ingestion — they call load_geodata() or load_ons() and get data back as GeoJSON or a DataFrame.

To refresh the catalogs when ONS publishes updates:

# Geo boundaries — from ArcGIS
uv run python -m kindtech.geo._ingestion

# Statistics — from NOMIS
uv run python -m kindtech.ons._ingestion

Then commit the updated CSV files.

References

Data Sources

ONS Open Geography Portal (ArcGIS)

Endpoints

Catalog discovery

Service naming conventions

Resolution codes

Catalog quirks

Geo ingestion

NOMIS API

Endpoints

SDMX format

Rate limits and authentication

Source annotations in the bulk listing

Catalog quirks

NOMIS ingestion

Community packages

postcodes.io

Endpoints

Returned geography codes

Licensing

UK deprivation indices (IMD)

Official national indices

Composite UK IMD

Dataset Aliases

Available aliases

Geography Crosswalk

Using geography_type in load_ons()

TYPE code mapping

Join keys

Architecture

References

ONS Open Geography Portal

NOMIS

Standards

Using `geography_type` in `load_ons()`