Project CODA: Reduce osm2pgsql Resource Usage

The amount of data in OSM is climbing continually, and therefore the memory and disk requirements of osm2pgsql have risen as well. In this project we want to reduce the memory and disk usage of osm2pgsql by implementing more efficient storage formats, specifically for “intermediate” data used while processing, often called the “middle”. We are tentatively calling this CODA, the Compact OpenStreetMap Data Archive.

This will not only help with resource consumption on the community-run OSM servers, but also enable wider use of OSM data, even on planet-scale, in low-resource environments available to small NGOs or to students.

Project Plan

The project starts in April/May 2026 and runs for about a year. We’ll first look at the requirements in detail, then research possible solutions using existing libraries for embedded databases as well as “from scratch” approaches. Based on the results of that research we’ll decide on an implementation. We’ll verify the design with a prototype and then integrate everything into osm2pgsql.

Progress

Here we’ll document the progress of the project.

Planning Phase

• The detailed requirements.
• Characteristics of OSM Data that can help with the implementation.
• The baseline gives us something to compare to when evaluating alternatives.
• As part of this project we looked at some other databases storing OSM data. The Imposm3 program is especially interesting.
• Some notes about DuckDB.
• The way we store node locations is really important. Not only are there a lot of them, we basically needs them for everything, so lookup has to be fast. There is another option: We could store the node locations on the ways!
• Storing tags in a sensible way is important. Tags are strings that can contain anything and they need a lot of space. But some tags are used very often (for instance 560 million ways, almost half of all ways, have a tag building=yes) so there must be better ways of storing them. We’ll also look at tag combinations.
• The implementation plan shows the next steps.

Funding

This project is funded through the NGI0 Commons Fund, a fund established by NLnet with financial support from the European Commission’s Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101135429. Additional funding is made available by the Swiss State Secretariat for Education, Research and Innovation (SERI).