News

The amount of data in OSM is climbing continually, and therefore the memory and disk requirements of osm2pgsql have risen as well. To address this issue we have applied for and won a grant from the NGI0 Commons Fund. In this project we want to reduce the memory and disk usage of osm2pgsql by implementing more efficient storage formats, specifically for “intermediate” data used while processing, often called the “middle”. We are tentatively calling this CODA, the Compact OpenStreetMap Data Archive. This will not only help with resource consumption on the community-run OSM servers, but also enable wider use of OSM data, even on planet-scale, in low-resource environments available to small NGOs or to students.

Nearly six years ago we introduced the “flex” output in osm2pgsql version 1.3.0 and then improved it over the years adding more and more functionality. About a month ago OpenStreetMap-Carto, the iconic OSM map style switched to using the “flex” output in version 6.0.0. And now the OSM Foundation has finished their gradual rollout of the new version. So the most important user of osm2pgsql doesn’t need the old “pgsql” output any more.

Styles (configurations) for osm2pgsql are written in the Lua programming language. And they are becoming more complex. Users have been asking for a long time for some way of testing those styles to help with writing and debugging the Lua code. We now have a solution for you.

We are excited to publish version 2.2.0 of osm2pgsql. This release adds two powerful building blocks that can be used to implement all sorts of new features. The new “Locators” and the “deleted callbacks” features seem simple by themselves, but they can be used as basis for many features that have been requested for years.

Over the last few weeks we have updated and overhauled the “Contributing” section on the osm2pgsql website. We want to make it easier for users and contributors to understand what our plans are for osm2pgsql, what are the open issues and what you can do to help.

New code will probably have some errors, the compiler finds some of them and refuses to produce the final program. So far so good. But code also contains lots of issues that are not outright errors, they just look “suspicious” to the compiler in one way or another. Some of them turn out to be actual errors, but sometimes the code is totally okay in the specific context. The compiler can’t tell for sure, so it emits a warning and keeps going. Different compilers and different versions of those compilers print different warnings.

This is a bug fix release which contains the following:

Today we released version 2.1.0, a bugfix and maintenance release. The main visible change is that osm2pgsql now works better together with connection poolers and similar database middleware: prepared statements now use protocol level prepare instead of PREPARE SQL commands and there is a new option to explicitly create a primary key on flex tables.

We are happy to announce that osm2pgsql maintainer Sarah has been selected as one of the fellows for the one-year pilot of the Sovereign Tech Fellowship programme of the Sovereign Tech Agency. The programme recognises and financially supports the significant amount of time that goes into the daily maintenance of an open-source project. Read more about the fellowship in Sarah’s post over at nominatim.org.

This release fixes some small issues with 2.0.0:

A few days ago Crunchy Data published a blog post Loading the World! OpenStreetMap Import In Under 4 Hours showing how improvements in PostgreSQL, osm2pgsql, and the hardware in recent years have improved the performance of loading the OSM planet into a database significantly leading to an import time for the planet of less than 4 hours.

The day has finally arrived where we are releasing version 2.0.0. of osm2pgsql, five years after the release of 1.0.0.

Looks like Artificial Intelligence (AI) is going to solve all the worlds problems. And we here at osm2pgsql headquarters certainly don’t want to be left out. Also all the research funding these days goes towards AI and we certainly don’t wan to be left out of that.

We are proud to announce version 1.11.0 of osm2pgsql.

Paul Norman and Jochen Topf gave a talk “Modern osm2pgsql tricks” at the recent State of the Map Europe (SOTM-EU) conference in Antwerp. It gives an overview about what modern osm2pgsql is all about.

This is a relatively small but still important release.

In 2021 we, the osm2pgsql developers and users, had a first virtual meetup. And now we want to do it again. We want to talk a bit about new features and chat about where we might want to go. But mostly this is a chance for osm2pgsql users to get questions answered and their feedback heard.

Today we switched all osm2pgsql repositories from the “openstreetmap” organization on Github to a new Github organization called osm2pgsql-dev. We want to make it is easier for our users to find all the pieces in our growing osm2pgsql ecosystem. There is…

Writing a config file for osm2pgsql from scratch is not easy. And it shouldn’t be something that everybody has to do. Most maps want to do some things different than other maps, but most things are probably the same as on many other maps. Themepark is a framework for creating osm2pgsql configuration by reusing existing building blocks. You can mix existing configurations with your special layers. And it all works together.

This release fixes a bug introduced in 1.9.0 with two-stage processing that will lead to crashes. If you are using any 1.9.x version, please upgrade to 1.9.2.

This release fixes some small issues with 1.9.0:

Development of an Open Source project is often driven by the current need for some features. But we also have frequent discussions about the long-term development of the project. Where do we need or want to go? How do all the different features fit together to make a coherent whole? What feature can we work on now that makes sense on its own but can also later be a part of something bigger?

We have just released version 1.9.0 of osm2pgsql. It brings a lot of new features which will open up osm2pgsql for many more use cases.

In osm2pgsql there is that code that stores the raw data it gets from the input files and makes it available to other parts of osm2pgsql for further processing. This code is called, in osm2pgsql-speak, the “middle”. Over the last months we worked on some huge improvements to that code which not only makes it much faster but also adds some functionality and will allow even more processing options down the line. You can read about all the details of this project in this blog post by Jochen Topf.

This release contains some fixes and minor changes.

We are happy to announce the new release 1.8.0 of osm2pgsql.

Osm2pgsql recently became a lot faster… And we didn’t have to do anything for it!

The new release 1.7.2 has some small changes only:

When starting out with osm2pgsql there is often the question: How large is the resulting database going to be and how long will the import take?

The new release 1.7.1 fixes a few small bugs in osm2pgsql and closes some gaps in the geometry processing code released in 1.7.0. It also contains some security-related fixes as a result of the security audit.

Ryan Lambert just announced the release of his book Mastering PostGIS and OpenStreetMap. It explains how to bring OpenStreetMap data into a PostgreSQL/PostGIS database (using osm2pgsql of course), cleaning it up in the process, and teaches you the ins and outs of data processing with PostGIS. Use the code “RPL-FRIENDS” for a 10% discount.

Our sponsor Thunderforest published a small interview with osm2pgsql developer Jochen Topf.

In the last couple of months the source code of osm2pgsql has gone through a security audit. This was part of the NGI Zero prject for Nominatim which uses osm2pgsql for its data import. The audit was done by the friendly people from RadicallyOpenSecurity (ROS).

OpenStreetMap has quite a lot of very detailed geodata. That’s great for generating very detailed maps. But it is sometimes not so easy to create overview maps in smaller map scales and for smaller zoom levels. We can’t see the forest for all the trees.

Any software needs a lot of testing and osm2pgsql is no different. For development we have automated unit, integration and regression tests, but they only run on tiny amounts of data. To figure out where performance bottlenecks are, how different configurations of osm2pgsql or the PostgreSQL database affect performance, or wether a planet import still works after some major changes, we need something else.

We are happy to announce the new release 1.7.0 with some great changes.

This is a bugfix release with only minor changes.

This is a bugfix release with only minor changes.

The talks by Jochen Topf on osm2pgsql from FOSSGIS 2021 (in German) and SotM 2021 (in English) are now available online.

This is a bugfix release. It contains some important bug fixes, so everybody is encouraged to update.

This release brings quite a lot of improvements. The most important one: The flex output which we introduced in version 1.3.0 almost a year ago as a new experimental feature is now ready for prime time: We are removing the “experimental” label. There are some small changes you might have to make to your flex configurations, see the Upgrading chapter of the manual for details. We have also introduced a new cluster table option in the flex config file which allows you to disable clustering of the table data by geometry.

I’ll be giving two talks about osm2pgsql soon. Both will be online. The first is at the German language FOSSGIS conference which takes place from June 6th to June 9th. The second is a the English language State of The Map from July 9th to 11th. Both talks will focus on new developments in osm2pgsql. You’ll need to register with these events to see the talks, but they will be available for free later.

This is a minor bugfix release that fixes the following issues:

Last week we had our first virtual osm2pgsql community meetup. There was quite a bit more interest than we had anticipated. 16 developers and users talked about plans and ideas and many questions could be answered.

This is a minor release with some bug fixes and internal cleanups and changes, mostly in the “middle” code and geometry processing.

Osm2pgsql development has picked up steam in the last months. Many new features have been implemented and we have many more ideas. And there are plenty of issues to work on. But to decide where we are going next and what to prioritize, we want to hear more from the osm2pgsql users.

The project has a new website (osm2pgsql.org) now with extensive documentation and examples, and with sections on support, contributing, news. etc. Most of the documentation from the repository and the OSM wiki was moved there. We still have a man page, it is now maintained in markdown format. All the documentation, man page, help texts etc. have been cleaned up, made more consistent and brought up to date.

Since version 11 PostgreSQL can use Just-in-Time Compilation (JIT) to speed up processing. But unfortunately using JIT can make osm2pgsql slower.

Osm2pgsql has been an important part of the OpenStreetMap software stack for a long time. But its documentation is scattered between the OSM wiki, the Github repository and other places here or there. Now, thanks to some money the OpenStreetMap Foundation is putting towards osm2pgsql development, we can unvail a new website for the project allowing us to bring together all the information about the project.

This release introduces the new “flex” output. It allows a more flexible definition of output tables and columns. It also adds a second stage of processing which makes it possible to get information from relations to their members, allowing, for instance, to render tags from bicycle route relations on their member ways. The “flex” output is configured through Lua scripts.