We have had tests for osm2pgsql, the software, for a long time. This includes tests that run osm2pgsql with certain inputs and parameters and check that osm2pgsql produces the correct output, including checks that make sure the correct information ends up in the correct database tables. These kinds of tests are really similar to what users need, but the tests were deeply integrated with the rest of the osm2pgsql testing system, they were hard to use, and they were not documented.
We have polished that testing framework, documented it and made it work on its own and are now making it available for osm2pgsql users. So you can now use the same testing engine used for internal testing to test your style sheets. The test code builds upon the Python BDD behave testing framework, and adds all sorts of useful features to the tests. OSM test data can be integrated in various ways, inline in the test file or as external data. And we have added various handy features for testing database content.
Style testing will be available in the next version of osm2pgsql, but you can already try it out. You only need to download a single file from the current master and can use that with any reasonably modern version of osm2pgsql. Find the full documentation in the manual.
We’d love it if you tried the new style tester and tell us what you think. We are sure there will be some teething problems you can help us find and fix before the next release.
]]>This release introduces a new feature: Locators. They help with a common problem, namely how to quickly find out in which region an OSM object is. The region can be a country or any other area. A locator is initialized with one or more regions, each region has a name and a polygon or bounding box. A geometry of an OSM object can then be checked against this region list to figure out in which region(s) it is located. This check is much faster than it would be to do this inside the database after import.
Locators can be used for all sorts of interesting features, for instance:
For details see the manual and look at the example config files provided in the flex-config/locator directory.
The Lua callback functions process_node(), process_way(), and process_relation() (and their untagged companions) only get called for new or changed OSM objects. Deleted objects are usually handled behind the scenes and for most use cases this is enough.
But sometimes it is useful to also know when an object was deleted. That’s why we have new process_deleted_node/way/relation() callbacks now. These open up a lot of new processing options for working with changes. Some users do advanced processing of OSM data inside the database after the data is imported with osm2pgsql. This is much easier now because you don’t need database triggers anymore to detect and process deleted objects.
We have added a osm2pgsql-expire command. It is currently marked as experimental so it might change without notice. You can use it to turn entries in expire files (that look like ZOOM/X/Y) into a GeoJSON file visualizing the tiles. When given an OSM data file it can generate the tile output (in expire file format or as GeoJSON). This tool is intended as a debugging aid.
process_node/way/relation() callback even if they had no tags. This is now fixed, the process_untagged_*() callbacks are called instead.--username and --user command line options in all commands, there was an inconsistency between osm2pgsql and osm2pgsql-replication before.The new Contributing Overview no longer focuses on coding because there are so many more ways to help the project than with code. Whether you are a user, have ideas on improving the documentation, or want to show your use case, everything helps to make osm2pgsql better.
We have also updated the road map with the most important milestones for getting to the next major release 3. If you are looking at a concrete project to work on, have a look at the updated project ideas.
]]>And then there are things that are not problems by themselves, they just make the code harder to read, possibly leading to more problems down the line. Or sometimes a tool can detect that you are using an “old-fashioned” way of doing things and there is a better, more modern or more efficient way, to achieve the same thing. Some of these issues are also detected by the compiler, but for most programming languages there are extra tools, so called “linters”, that find those issues. For C++ “clang-tidy” is such a tool.
We have been compiling with lots of warnings enabled for years now. And we have also been running “clang-tidy” regularly for a long time, chipping away at the problems mentioned. Some warnings are easily dealt with, but others are more complicated and need more code changes for a good solution. And in some cases, after careful consideration, warnings have to be disabled, either for the whole code or for specific places in the code. Osm2pgsql is 19 years old now, the first implementation was written in C, along the way we switched to C++98, then C++11, C++14 and then C++17. So the way things can and should be done changed over time. New versions of compilers and new versions of clang-tidy came up with new warnings. As part of our work on osm2pgsql we looked at those warnings whenever there was time and fixed something here or there. But there was always another warning, another sometimes very old and sometimes newly introduced problem. Until now.
For a long time now our code has compiled cleanly with the GCC and Clang compilers, today we reached the point where it also compiles cleanly on MSVC. And it passes all checks configured in clang-tidy. To “lock in” that achievement we have changed the CI settings so that any warnings will show up as errors blocking us from merging the code. This way we can be sure to keep the “zero warnings” achievement.
Full disclosure: For some legacy code we have disabled a few warnings. That part of osm2pgsql is already deprecated and will be removed, it just doesn’t make sense to spend time fixing code that’s more than a decade old when that code is going away anyway. Especially as fixing warnings can sometimes introduce new bugs in the process.
For clang-tidy we enable all checks by default and then selectively disable all checks that don’t make sense for us. The clang-tidy config file contains comments explaining why the checks are disabled. We will probably revisit some decisions on what checks to disable in the future as our code improves further.
This is not the end of our journey to improve the osm2pgsql code, there is plenty more to do. But it is a major milestone in that it makes sure we are not sliding back behind what we have already achieved, regardless of any new changes we make.
]]>PREPARE SQL commands and there is a new option to
explicitly create a primary key on flex tables.
Other fixes and features:
--schema parameter from osm2pgsql-replication to osm2pgsqlosm2pgsql_find_changed_ways function in the right schema (thanks @falko17)They are using some configuration options that are probably not great for general use, but if you know what you are doing they can give you that extra edge.
]]>This release marks a milestone in modernizing osm2pgsql. We removed a lot of accumulated cruft from more than a decade of development. This makes osm2pgsql easier to understand for users and for developers. And it has allowed us to solve some long-standing issues and will allow further improvements in the future.
Major breaking changes:
add_row() function in the flex Lua config has been removed, use insert() instead.Please see the Upgrading appendix in the manual for the details of these changes. We suggest you update to version 1.11.0 first and resolve any issues you see there before upgrading to version 2.0.0.
This is the first release that deprecates the “pgsql” output, please start moving towards the “flex” output instead. See this FAQ entry for the details.
New features in the flex output:
| New after_nodes | ways | relations() processing functions in flex Lua config files |
Changes in the generalization code:
Other fixes and features:
Many thanks to Thunderforest for their continued support.
]]>