We have just released version 1.9.0 of osm2pgsql. It brings a lot of new features which will open up osm2pgsql for many more use cases.
First, a possibly breaking change for some users in the way we handle database
schemas. Historically osm2pgsql did not work with database schemas at all, but
just put tables, indexes, etc. in whatever schema was set by your database
search path, usually
public. In 1.4.0 we introduced schema support. If you
set a schema specifically, that schema is used, otherwise osm2pgsql would fall
back to the old behaviour. From 1.9.0 onwards, the
public schema will be used
by default instead. For most users nothing will change, but if you have been
using a different schema and/or relying on the database search path, you might
need some changes. The new rule is simple: If you want to use the
schema, don’t set any option. If you want to use any other schema, set it with
one of the command line options and/or settings in the flex config file.
We added a new command line option
--schema that sets the default schema for
all the other places in osm2pgsql where schemas are used (
--output-pgsql-schema, and the flex and generalizer config). You can use that
option and/or any of the more specific ones.
It has always been slightly annoying that you had to give the same command line
options to osm2pgsql when doing updates that you had already given on import.
We now store some of the settings from the command line options and some other
meta data in a database table called
osm2pgsql_properties. This makes use of
osm2pgsql simpler, especially when doing updates. The
script has also been updated to take advantage of this. The settings for
-x|--extra-attributes are stored.
We also store some meta data from the input files needed for replication.
See the manual for the details.
To allow updates, osm2pgsql needs to store all raw OSM data from the input file
in the database, in osm2pgsql speak we call that storage the “middle”. The
format in which it does that was somewhat strange and cumbersome to use. This
release introduces a new format which is much easier to use, it is
documented and even
makes the database a bit smaller. It is currently in “experimental” state,
please try it out (with the command line option
and report any problems. The plan is that this will become the new default
format and we’ll then promise to keep it stable so users can built upon it.
Later we’ll phase out the old format.
See the manual for details on the new format.
While we were working on the database format we also improved the way we access the database quite a bit. Updates, especially of larger change files, are now much faster. You can see up to 10x speedup for large updates!
For lots more detail about the new format and performance improvements see this recent news article.
A major addition is support for generalization. The generalization functionality was developed in a half-year project funded by the Prototype Fund. Generalization allows efficient representation of OSM data for smaller zoom levels which is especially important for vector tiles. This functionality is currently marked as experimental, because we want to reserve the right to change anything. Some parts of the code are quite experimental but some parts already work quite well. Try it out and tell us what your experiences are. We need feedback from users to improve this.
More information on the project page and in the Generalization chapter in the manual.
As part of the support for generalization we also added the option to put lists of expired tiles not only in a file (which we have supported forever), but also in database tables. You can have several expire lists, for instance for different layers. If the rest of your tool chain supports this you can update only those layers of a tile that changed, for instance.
See the manual for the details.
You now need several new libraries to compile osm2pgsql, see the README.md for details.
As always there are lots of smaller changes:
- A new
spherical_area()function is available in flex config files to calculate the area of a (multi)polygon on the sphere.
- If you are using the new database middle, the
--middle-with-nodesoption allows you to store all tagged nodes in the database (with their tags and location).
- Several improvements to osm2pgsql-replication to make it more flexible and better tested. Thanks to Amanda McCann and Jakob Miksch for supplying many of these.
- Don’t do multi-statement SQL queries. They are not supported by the PgPool-II connection pooler.
- Many small fixes and code cleanups.
Many thanks to the Prototype Fund, Thunderforest and Geofabrik who supported osm2pgsql development.