Onboarding a new country¶
A country is rows in the database, not code. Onboarding one is a single operator command on the deployment's side, one confirmation, and then the worker host does the rest: seed the regions, harvest the country, rebuild its tiles, repair the border data of its onboarded neighbours, check elevation, and go live. Nothing in this repository changes, nothing is committed and nothing is deployed.
Two data sources do two different jobs: the Overture Maps divisions theme (ODbL) draws the
region boundaries, and Geofabrik's OpenStreetMap extracts fill them with points
(Harvesting).
The command¶
The deployment's runner (a make target and a systemd template unit in the operator's own infrastructure repository, not described further here) strings together the commands this repository provides:
| # | Step | Command | Container |
|---|---|---|---|
| 1 | Plan, print it, ask for one confirmation | python -m onboarding.plan DK |
pipeline |
| 2 | Seed regions, derive, re-stamp | php bin/console app:country:apply DK |
worker |
| 3 | Harvest, repair neighbours, rebuild tiles | python -m onboarding.repair DK |
pipeline |
| 4 | Elevation check | php bin/console app:country:check-elevation DK |
worker |
| 5 | Go live | php bin/console app:country:mark-live DK |
worker |
The planner only writes the plan and prints it; the confirmation belongs to the runner. Steps 2-5 run unattended, one after the other; the first failure stops the rest and alerts like any other scheduled job. Every step checks the status it needs and is safe to rerun, so a failed run is fixed by starting it again; a failed elevation check is the exception (see Elevation check below). Staging always goes first; production then receives staging's exact rows (see Promoting to production).
A country moves through three states, stored in country.status:
- planned: a plan exists (
country_plan_region); nothing a rider sees has changed. - seeded: its
regionrows exist, so the harvest may run and its extracts count as onboarded. - live: every step passed; only now does the country show on the map.
The configuration lives in four tables: country (name, status, timezones, labels), country_extract
(its Geofabrik extracts, one row per country for an extract two countries share), country_plan_region (the plan) and region (the seeded regions, with
their labels). No YAML or source file lists countries; make country-lists-check fails when one
creeps back in.
What the planner decides¶
The planner never asks. It applies fixed rules, prints what it decided, and stops with a reason when a rule cannot decide. A seeded or live country is refused: its slugs are permanent identity. Every rule has an override flag for the case where it cannot decide.
- Level. It probes Overture's
regionandcountysubtypes and picks the first whose median subdivision is between 1,000 and 60,000 km². A country under 5,000 km² becomes one region at level 2, the Luxembourg rule. Otherwise it stops and prints the probe table. Override:--level county. - Slugs. From the native name: generic words such as Region or Provincia dropped,
transliterated (
Sjællandbecomessjaelland), lower case, dashes. A slug already used by another country gets-<cc>(limburg-nl). The country outline is the exception: it takes the English name (denmark, likebelgium). Override:--slug DK-84=hovedstaden, where the key is the ISO code, or the native name for subdivisions without one. An override cannot take a slug another region already holds. - Labels. Overture's name in English, French, Dutch, German and Spanish; where Overture has none,
the native name stands in and the table marks it with
*. The country phrase ("All Denmark") uses fixed templates per language. Fix one label later withapp:country:label(below). - Extract. The Geofabrik extract whose ISO country list is exactly that country, the shallowest
one first. None, or two at the same depth, stops the plan. Override:
--extract europe/denmark. An extract another country already holds is shared when Geofabrik lists the new country in it:europe/ireland-and-northern-irelandlists Ireland and also carries Northern Ireland, so Ireland shares it with Great Britain, and the plan prints(shared with GB). An extract that does not list the country stops the plan, so a mistyped--extractnever takes another country's extract.--sharedsays the extract carries the country though the index does not list it: San Marino and Vatican City lie insideeurope/italy, whose entry lists Italy only, so they are planned with--extract europe/italy --shared. - Neighbours. Seeded or live countries whose regions lie within this distance of the new country, just over the 0.1026° buffer Geofabrik cuts its extracts with:
--json prints the plan as JSON instead of the table. On the dev stack, make country-plan c=DK
runs the planner against your local database (it needs the network for Overture and Geofabrik); pass
flags with args="--level county". It writes the plan and nothing else.
On the dev stack: one command¶
make country-onboard runs all five steps for every country code it is given:
It plans each country, prints the plans, asks one confirmation (yes=1 skips it), applies each,
repairs them all in one pass, then checks elevation and marks each live. release=2026-10-21.0
passes an Overture release to the planner, and args="..." passes more planner flags to every
country it plans. A seeded country skips plan and apply and a live one is
left alone, so a run that stopped (a held lock, a failed elevation check) is finished by running the
same command again. tools/onboard-countries.sh does the work.
What each step does¶
Apply turns the plan into region rows in one transaction, with the same upsert, provenance
check and tessellation check the catalog import uses (RegionUpserter), then recomputes adjacency
and outlines, and re-derives membership, rider base areas and route surfaces (RegionDerivations)
for the rows within 0.11° of the new country and its neighbours only, writing a row only when its
value really changes. It re-stamps rider data for the same countries, and fills the country's
timezones (the map's home-country hint for visitors who are not signed in) from PHP's zone table
when it has none yet. It refuses a plan whose slug already belongs to another country. It waits at
most 10 seconds for a lock another writer holds (a harvest, a catalog import), then stops without
writing anything and says so; start it again once that writer is done. The full catalog import
still re-derives every row.
Re-stamp (app:regions:restamp --countries=DK,DE, also on its own; --dry-run counts without
writing, --recount refreshes the coverage counts afterwards) gives a submission the region and
country it now lies in when it had none or belonged to a listed country, and gives an item the
country of the region it lies in. Smallest containing region wins, the same rule SpatialResolver
applies when a rider submits.
Repair harvests the new country's extracts, re-harvests every neighbour's extracts with full
membership (their border rows must be re-owned now that new regions sit next to them), rebuilds the
point tiles, then the route and surface tiles of the new country and its neighbours. Several
countries onboarded together repair in one call, python -m onboarding.repair AD MC: France borders
both and is re-harvested once. Two countries of one call that border each other take the full
membership pass too. Each is one
call of the documented coverage.run command. When the nightly harvest holds the coverage, routes or
surface lock, repair exits with code 2 and says to start it again later. An extract the new country
shares with a neighbour is harvested once, in the neighbours' step: one harvest gives each point to
the country of its nearest region, so Ireland's points and Northern Ireland's come from the same run.
Elevation check samples 25 points inside the country and asks the elevation instance
ElevationEndpoints picks for each. It fails when more than 80 % of the answers are null or exactly
0, the two silent ways Valhalla says "no tiles here". An instance that does not answer at all (a
timeout, a refused connection, an HTTP error) is a different failure: the check stops at the first
one with elevation instance unreachable: <endpoint>, so fix or start that instance rather than
installing tiles. Missing tiles are fixed by a DEM install on the routing host, which
Building elevation tiles owns: install, restart that continent's instance and
check that the instance is listed in ELEVATION_URLS. Then finish by hand instead of starting the
whole run again, which would repeat the harvest:
Mark live records that every step passed, and only on a seeded country.
A country is on the map once it is live
The region registry behind the scope selector and the timezone hint serve a country's regions
only once its status is live, so a seeded country whose harvest or elevation check has not
passed yet is not shown. Moderation and the harvest already see it from seeded on. Regions of a
country that has no country row at all are always shown.
Promoting to production¶
Production never queries Overture. app:country:export DK writes the live staging country (its
country row, extracts and regions, geometry as GeoJSON) to stdout, and app:country:import-plan
reads it on production (from stdin, or --file) into a plan. The same apply, repair, check and live
sequence then runs there, so production gets staging's exact slugs, labels and borders. The bundle
names the other countries that share each extract; the import shares an extract with exactly those
and refuses one that another country holds there.
Adding regions to a live country¶
The United States and Canada are onboarded in part: a few states and provinces, plus the country
outline. More regions join a live country through the same five steps, with --add on the planner:
python -m onboarding.plan US --add US-AZ --add US-OR
php bin/console app:country:apply US
python -m onboarding.repair US
php bin/console app:country:check-elevation US
php bin/console app:country:mark-live US
- Plan. Each
--addnames one ISO 3166-2 code. The new regions take the country's own level, and the extract whose Geofabrikiso3166-2is exactly that code (north-america/us/arizonaforUS-AZ);--extract US-AZ=<slug>overrides one. A code the country already has stops the plan. The plan writescountry_plan_regionandcountry_plan_extractonly: the country stays live and its extracts stay as they were, so a nightly harvest never reads the new extract before its regions exist. - Apply writes the new rows with
region.pendingset, moves the planned extracts tocountry_extract, and re-derives and re-stamps only around the new regions. It refuses a plan row whose slug is a region already on the map. - Repair re-harvests every extract of the country with full membership, not just the new ones: the rows California's extract holds along the Arizona border belong to Arizona now.
- Check elevation samples the pending regions only. The DEM presets
USARIZONAandUSOREGONintools/elevation/fetch-glo30.shcover the two states. - Mark live clears
pendingand the plan rows. The country keeps itslive_at.
Until mark-live the region registry leaves a pending region out, as it leaves out a country that
is not live. app:country:export refuses a country with pending regions; once they are live, the
bundle carries them, and app:country:import-plan on a database where the country is already live
plans exactly the regions it lacks there, with their new extracts.
Telling the riders who asked¶
Once the country is live on production, its block on /admin/country-requests shows a button that
tells the riders who asked for it. Each rider gets one inbox message, which also emails them in
their own language, and is never told twice. Only requests made before the country went live count:
a later request asks for more inside a country that is already on the map. The same moment, the
country appears under "Next release" on "What's new", until the next release names it.
The regions' Wikipedia text and population¶
The steps above leave a new country's region pages without the two things fetched at build time:
the Wikipedia text (region.context) and the population with the year it was counted
(region.facts_harvest). Both are files a person reviews and commits, never fetched by the site
itself, so they follow the country by hand, once its regions exist:
-
Build both files from a database that holds the new regions; the slugs and ISO codes come from its
regiontable. A run without--isoasks Wikidata again for every region, so older counts are brought up to date in the same pass:python3 tools/wikimedia/region_context.py --dsn postgresql://cc:cc@localhost:5433/cyclingcommons python3 tools/wikimedia/region_population.py --dsn postgresql://cc:cc@localhost:5433/cyclingcommonsA run with
--iso DK-84 --iso DK-85asks only for those regions and adds them to the committed file. -
Review the changes in
tools/wikimedia/out/region-context.jsonandtools/wikimedia/out/region-population.json, then commit them. The end of a run names every region Wikidata had nothing for; that region stays without the figure. -
With the release deployed, on staging and then on production, copy both files into the app container and import them:
Each import writes its own column only. A text or figure a person sets on the region editor stays, and the page shows it over the fetched one. Nothing refreshes the counts on a schedule: a full run of step 1 before a release keeps them current.
Fixing a label¶
Region and country labels are rows (region.labels, country.labels), so the in-site translation
tool does not reach them. Translations it had approved before the labels moved were folded into these
rows by a migration. Edit a label with:
php bin/console app:country:label hovedstaden nl 'Hoofdstedelijke Regio'
php bin/console app:country:label DK fr 'Danemark (tout le pays)'
A display label falls back to English, then to the region's name, so a missing one never shows a raw key.
Deploy order¶
A release that adds onboarding tables ships web migrations first, then the new pipeline image: the planner and the coverage commands read the tables the migrations create.
What stays manual¶
- Curators. A new country's regions start without curators;
/join/<cc>switches to the curator form once regions exist. Until somebody is scoped, the country is live on the map and invisible in moderation. - The regions' Wikipedia text and population. Build-time files, see The regions' Wikipedia text and population above.
- A new UI language. Onboarding Denmark does not add Danish.
- Official data providers. Separate, per-licence work.
- Telling riders about added regions. The button on
/admin/country-requestsanswers requests made before the country went live; a request for Arizona inside the live United States is not told.
Try it¶
Hands-on: read a country's state, then its rows
php bin/console app:country:status prints one line per country: status, when it was planned,
seeded and made live, its region count and its extracts. Then look at the rows themselves:
SELECT c.code, c.status, e.slug AS extract,
(SELECT count(*) FROM region r WHERE r.country_code = c.code) AS regions
FROM country c JOIN country_extract e ON e.country_code = c.code
WHERE c.code = 'NL';
Exactly one region of the country must be level 2: the country outline every scoped query falls
back to. make -s coverage-regions lists every onboarded extract, straight from these tables.
Where to go deeper¶
pipeline/divisions/README.md: Overture provenance, the operational-versus-infrastructure rule for level-2 rows, and the notes of every country onboarded so far.- Harvesting: what the repair step's harvest does.