Downloadable OSM data with attributes in a language other than English, not with a web service, but with a platform dedicated to sharing geographic data and metadata, where you can understand, view, query and filter before downloading: that’s what this ETL (Extract, Transform, Load) approach in the Libres Géographes Spatial Data Infrastructure makes possible. In this post, I’ll go back over the background and history of this personal project carried out in my spare time, before explaining the technical approach implemented and, of course, how to access this data.
The context: moving away from “English fits for all”
While English dominates the OSM ecosystem and remains the project’s reference language, several initiatives allow non-English speakers to participate in and benefit from the project: a multilingual forum, translation of the wiki and certain self-learning platforms, translated user interfaces for applications and editors, including OSM tag presets.
But regardless of the technology or service used, the raw OSM data, once downloaded, remains exclusively in English, and any searching or filtering of OSM data in GIS software can only be done in that language
Having trained extensively in the use of OSM data in geomatics (particularly QGIS) since 2011, I was quickly confronted with the difficulties many non-English speakers have in exploiting the attributes of OSM data. These difficulties were even more frustrating in the case of developing countries, as this was often the first detailed data available for their territory. One barrier was removed, but another followed.
This limitation therefore tends to slow down the learning process for non-English speaking OSM contributors, but more importantly, it remains an obstacle to adoption by audiences outside the OSM community: for example, public services accustomed to creating/distributing/using data in the official language, or one of the official languages, of their country.
In fact, from the beginning of my involvement with OSM, I have always had the idea of making it possible to use OSM data without necessarily having to use English.
Background: a service that could have been here in 2013
Despite the enthusiasm generated by the community response to the Haiti earthquake in January 2010, it remained difficult to promote OSM to humanitarian geomaticians without a service that allowed them to retrieve OSM data in the most common GIS formats without having to deploy a local PostgreSQL database. In short, these geomaticians wanted shapefiles.
As part of the USAID-funded HOT STM020 project in Saint-Marc, Haiti, designed and implemented with Nicolas in the spring of 2012, we were eager to include in the budget a line for a “data download point” that would become HOT Exports (then with a final “s”), developed by GeoFabrik. The following year, the CAP103 project, this time in northern and northeastern Haiti, was an opportunity to fund the addition of tag transformation and translation capabilities to HOT Exports. One of the few surviving visualizations of this v1 testifies to this (below, in Expert Functions):
In fact, it would have been possible to offer a translation of OSM’s English tags into another language as early as 2013. But no one took it up immediately. For my part, I started working on it in the spring of 2015 during a volunteer stay in Dakar and made a first list of OSM tags translated into French and implemented in HOT Exports. Some geomatics students in Senegal were able to benefit from this during sessions dedicated to the use of OSM data in QGIS. But in mid-2015, HOT Export v2 was released, completely rewritten, and all tag transformation and translation capabilities were removed. Again, no choice, English or nothing.
I took over the project in my spare time from the 2020 confinement, this time with the aim of using as a platform IFL, the Spatial Data Infrastructure (IDS) Francophone Libre of the Libres Géographes (based on the geOrchestra project), which was already providing custom layers for some of the association’s projects. I’ve also completed the translation, which now covers 1400 tags linked to the most common OSM keys identified by taginfo, and using translations of OSM wiki Map features or JOSM presets.
The first provisional version, covering only a few West African countries, was presented at GeoCom 2023 (the annual geOrchestra conference), SotM France 2024 and global SotM 2024 in Nairobi. Optimized in response to audience feedback at these conferences, the service now covers all 28 French-speaking countries in the South (those listed here in Wikipédia + Mauritius, Lebanon and Haiti), with continuous updates via minute replications. Why not all the French-speaking countries? Because data for metropolitan France would take up too much space on the IFL.
The Technical Approach: Detailed Planet and Thematic OSM Layers, in English or French
Typically, data is hosted in pg databases (PostgreSQL/PostGIS) fed by imposm, which was considered the most efficient osm.pbf data transformation tool in 2020. This is probably no longer the case, as its development and maintenance have dried up, while osm2pgsql has been relaunched, largely thanks to funding from OSMF. At SotM 2024 in Nairobi, Jochen Topf explained to me the advantages of osm2pgsql over imposm, and maybe I’ll make the switch someday, but for now imposm covers the need, and the imposm run update feature comes in very handy. The OSM data is taken from extraction points and minute replications put online by the association OpenStreetMap France, whose manager has kindly agreed to add a few missing countries.
As for the layers, there are two ready-made datasets for each country concerned, one in the original English, the other translated on the fly into French using a reference translation table published here on GitLab. Comments, suggestions and additions are welcome.
Each dataset contains both ready-made thematic layers and planet layers (i.e. a point layer that groups all point objects in OSM, and two others dedicated to linear and polygonal objects, respectively), allowing you to create your own thematic layers. In fact, I’ve always found it a bit unfortunate that existing services offer only thematic layers by default (download from Geofabrik or Bbbike) or only planet layers (HOT Export). The themes available are: administrative boundaries, places, transportation, obstacles, buildings, land cover, hydrography, and points of interest (POI).
Some themes include several layers if they are represented by different types of geometry (points, lines or polygons). POIs are available not only in their original form (points or polygons), but also in a synthesis layer that includes point objects and the centers of polygonal objects.
In addition, each OSM layer in the IFL can be used to retrieve all tags: the most commonly used keys in each theme each have a dedicated field in the attribute table, but there is also a field in jsonb format that contains all the original tags for each object. Furthermore, if none of these layers contains OSM metadata, an added_at field provides the date when each object was integrated into the IFL pg database. This makes it possible to identify the most recently edited objects in OSM after the extraction date used to create the database.
These layers are therefore not accessible via a web service, but via an IDS. Why an IDS and not a specific web site? Both for practical reasons, but also to prove that an IDS does this very well. On the practical side, the IFL already exists, so there’s no need to develop an ad hoc web site. On the proof of concept side, an open source IDS like geOrchestra has all the functionality provided by OGC standards and open source software building blocks like PostgreSQL/PostGIS, GeoServer, GeoNetwork and MapStore to provide data processing, a searchable map interface, metadata and download tools. In the end, the data is ready to download without the need to create custom download points as with HOT Export, which requires server-side storage that is difficult to estimate.
Accessing the data: via metadata sheets or country-specific mapping applications
There are two ways to access OSM data on French-speaking countries in the South from the IFL: via metadata sheets or map applications.
Metadata includes :
- Metadata sheets dedicated to the presentation of each theme available;
- Metadata sheets dedicated to each French-speaking country covered, with an integrated download link via drop-down menus and a link to a cartographic application dedicated to each country;
- A parent metadata sheet describing the ETL in detail and linking the metadata sheets.
Map applications for viewing, querying, filtering and downloading are dedicated to each country. Thanks to MapStore’s functionality, it is possible to download a theme for an entire country, but also to filter on attributes, on a hand-drawn area, or on the area of another layer, such as administrative boundaries. It is therefore perfectly possible to retrieve, from the POI layer, the hospitals, clinics and pharmacies of a region or urban district, as shown below in the Yaoundé Urban Community:
Online or downloadable videos show these different access points:
So much for this long presentation. In conclusion, I’d like to say that at a time when announcements of mega datasets that mix multiple sources or come from AI, this ETL shows that there are still ways to do smaller, but still fine, meaningful things with OpenStreetMap that are adapted to local contexts.
