Release 1.9.0

We have just released version 1.9.0 of osm2pgsql. It brings a lot of new features which will open up osm2pgsql for many more use cases.

Schema Handling

First, a possibly breaking change for some users in the way we handle database schemas. Historically osm2pgsql did not work with database schemas at all, but just put tables, indexes, etc. in whatever schema was set by your database search path, usually public. In 1.4.0 we introduced schema support. If you set a schema specifically, that schema is used, otherwise osm2pgsql would fall back to the old behaviour. From 1.9.0 onwards, the public schema will be used by default instead. For most users nothing will change, but if you have been using a different schema and/or relying on the database search path, you might need some changes. The new rule is simple: If you want to use the public schema, don’t set any option. If you want to use any other schema, set it with one of the command line options and/or settings in the flex config file.

We added a new command line option --schema that sets the default schema for all the other places in osm2pgsql where schemas are used (--middle-schema, --output-pgsql-schema, and the flex and generalizer config). You can use that option and/or any of the more specific ones.

Properties

It has always been slightly annoying that you had to give the same command line options to osm2pgsql when doing updates that you had already given on import. We now store some of the settings from the command line options and some other meta data in a database table called osm2pgsql_properties. This makes use of osm2pgsql simpler, especially when doing updates. The osm2pgsql-replication script has also been updated to take advantage of this. The settings for -O|--output, -S|--style, --flat-nodes, -p|--prefix, -x|--extra-attributes are stored.

We also store some meta data from the input files needed for replication.

See the manual for the details.

Database Middle

To allow updates, osm2pgsql needs to store all raw OSM data from the input file in the database, in osm2pgsql speak we call that storage the “middle”. The format in which it does that was somewhat strange and cumbersome to use. This release introduces a new format which is much easier to use, it is documented and even makes the database a bit smaller. It is currently in “experimental” state, please try it out (with the command line option --middle-database-format=new) and report any problems. The plan is that this will become the new default format and we’ll then promise to keep it stable so users can built upon it. Later we’ll phase out the old format.

See the manual for details on the new format.

While we were working on the database format we also improved the way we access the database quite a bit. Updates, especially of larger change files, are now much faster. You can see up to 10x speedup for large updates!

For lots more detail about the new format and performance improvements see this recent news article.

Generalization

A major addition is support for generalization. The generalization functionality was developed in a half-year project funded by the Prototype Fund. Generalization allows efficient representation of OSM data for smaller zoom levels which is especially important for vector tiles. This functionality is currently marked as experimental, because we want to reserve the right to change anything. Some parts of the code are quite experimental but some parts already work quite well. Try it out and tell us what your experiences are. We need feedback from users to improve this.

More information on the project page and in the Generalization chapter in the manual.

Tile Expiry

As part of the support for generalization we also added the option to put lists of expired tiles not only in a file (which we have supported forever), but also in database tables. You can have several expire lists, for instance for different layers. If the rest of your tool chain supports this you can update only those layers of a tile that changed, for instance.

See the manual for the details.

Other Changes

You now need several new libraries to compile osm2pgsql, see the README.md for details.

As always there are lots of smaller changes:

• A new spherical_area() function is available in flex config files to calculate the area of a (multi)polygon on the sphere.
• If you are using the new database middle, the --middle-with-nodes option allows you to store all tagged nodes in the database (with their tags and location).
• Several improvements to osm2pgsql-replication to make it more flexible and better tested. Thanks to Amanda McCann and Jakob Miksch for supplying many of these.
• Don’t do multi-statement SQL queries. They are not supported by the PgPool-II connection pooler.
• Many small fixes and code cleanups.

Many thanks to the Prototype Fund, Thunderforest and Geofabrik who supported osm2pgsql development.