From: Sarah Hoffmann Date: Thu, 29 Dec 2022 18:43:56 +0000 (+0100) Subject: Merge remote-tracking branch 'upstream/master' X-Git-Tag: deploy~81 X-Git-Url: https://git.openstreetmap.org./nominatim.git/commitdiff_plain/0cdbb97f6a4953971d60a46ddabc5df24be5f8ad?hp=11844313b4f420e13885714750d620ec41e75f36 Merge remote-tracking branch 'upstream/master' --- diff --git a/CMakeLists.txt b/CMakeLists.txt index d4c16126..c8a57521 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -292,11 +292,12 @@ endif() install(FILES settings/env.defaults settings/address-levels.json settings/phrase-settings.json - settings/import-admin.style - settings/import-street.style - settings/import-address.style - settings/import-full.style - settings/import-extratags.style + settings/import-admin.lua + settings/import-street.lua + settings/import-address.lua + settings/import-full.lua + settings/import-extratags.lua + settings/flex-base.lua settings/icu_tokenizer.yaml settings/country_settings.yaml DESTINATION ${NOMINATIM_CONFIGDIR}) diff --git a/VAGRANT.md b/VAGRANT.md index e00e0954..819c6071 100644 --- a/VAGRANT.md +++ b/VAGRANT.md @@ -1,6 +1,6 @@ # Install Nominatim in a virtual machine for development and testing -This document describes how you can install Nominatim inside a Ubuntu 16 +This document describes how you can install Nominatim inside a Ubuntu 22 virtual machine on your desktop/laptop (host machine). The goal is to give you a development environment to easily edit code and run the test suite without affecting the rest of your system. @@ -69,8 +69,7 @@ installation. PHP errors are written to `/var/log/apache2/error.log`. With `echo` and `var_dump()` you write into the output (HTML/XML/JSON) when -you either add `&debug=1` to the URL (preferred) or set -`@define('CONST_Debug', true);` in `settings/local.php`. +you either add `&debug=1` to the URL. In the Python BDD test you can use `logger.info()` for temporary debug statements. @@ -130,6 +129,10 @@ and then Yes, Vagrant and Virtualbox can be installed on MS Windows just fine. You need a 64bit version of Windows. +##### Will it run on Apple Silicon? + +You might need to replace Virtualbox with [Parallels](https://www.parallels.com/products/desktop/). +There is no free/open source version of Parallels. ##### Why Monaco, can I use another country? @@ -141,11 +144,12 @@ No. Long running Nominatim installations will differ once new import features (o bug fixes) get added since those usually only get applied to new/changed data. Also this document skips the optional Wikipedia data import which affects ranking -of search results. See [Nominatim installation](https://nominatim.org/release-docs/latest/admin/Installation) for details. +of search results. See [Nominatim installation](https://nominatim.org/release-docs/latest/admin/Installation) +for details. ##### Why Ubuntu? Can I test CentOS/Fedora/CoreOS/FreeBSD? -There is a Vagrant script for CentOS available, but the Nominatim directory +There used to be a Vagrant script for CentOS available, but the Nominatim directory isn't symlinked/mounted to the host which makes development trickier. We used it mainly for debugging installation with SELinux. @@ -154,14 +158,17 @@ are slightly different, e.g. the name of the package manager, Apache2 package name, location of files. We chose Ubuntu because that is closest to the nominatim.openstreetmap.org production environment. -You can configure/download other Vagrant boxes from [https://app.vagrantup.com/boxes/search](https://app.vagrantup.com/boxes/search). +You can configure/download other Vagrant boxes from +[https://app.vagrantup.com/boxes/search](https://app.vagrantup.com/boxes/search). ##### How can I connect to an existing database? -Let's say you have a Postgres database named `nominatim_it` on server `your-server.com` and port `5432`. The Postgres username is `postgres`. You can edit `settings/local.php` and point Nominatim to it. +Let's say you have a Postgres database named `nominatim_it` on server `your-server.com` +and port `5432`. The Postgres username is `postgres`. You can edit the `.env` in your +project directory and point Nominatim to it. + + NOMINATIM_DATABASE_DSN="pgsql:host=your-server.com;port=5432;user=postgres;dbname=nominatim_it - pgsql:host=your-server.com;port=5432;user=postgres;dbname=nominatim_it - No data import or restarting necessary. If the Postgres installation is behind a firewall, you can try @@ -169,11 +176,12 @@ If the Postgres installation is behind a firewall, you can try ssh -L 9999:localhost:5432 your-username@your-server.com inside the virtual machine. It will map the port to `localhost:9999` and then -you edit `settings/local.php` with +you edit `.env` file with - @define('CONST_Database_DSN', 'pgsql:host=localhost;port=9999;user=postgres;dbname=nominatim_it'); + NOMINATIM_DATABASE_DSN="pgsql:host=localhost;port=9999;user=postgres;dbname=nominatim_it" -To access postgres directly remember to specify the hostname, e.g. `psql --host localhost --port 9999 nominatim_it` +To access postgres directly remember to specify the hostname, +e.g. `psql --host localhost --port 9999 nominatim_it` ##### My computer is slow and the import takes too long. Can I start the virtual machine "in the cloud"? diff --git a/docs/admin/Import.md b/docs/admin/Import.md index 8b6d6baa..d84a2376 100644 --- a/docs/admin/Import.md +++ b/docs/admin/Import.md @@ -74,7 +74,7 @@ but it will improve the quality of the results if this is installed. This data is available as a binary download. Put it into your project directory: cd $PROJECT_DIR - wget https://www.nominatim.org/data/wikimedia-importance.sql.gz + wget https://nominatim.org/data/wikimedia-importance.sql.gz The file is about 400MB and adds around 4GB to the Nominatim database. @@ -92,8 +92,8 @@ and the UK (using the [CodePoint OpenData set](https://osdatahub.os.uk/downloads This data can be optionally downloaded into the project directory: cd $PROJECT_DIR - wget https://www.nominatim.org/data/gb_postcodes.csv.gz - wget https://www.nominatim.org/data/us_postcodes.csv.gz + wget https://nominatim.org/data/gb_postcodes.csv.gz + wget https://nominatim.org/data/us_postcodes.csv.gz You can also add your own custom postcode sources, see [Customization of postcodes](../customize/Postcodes.md). diff --git a/docs/admin/Installation.md b/docs/admin/Installation.md index bcc4524d..73fb3dae 100644 --- a/docs/admin/Installation.md +++ b/docs/admin/Installation.md @@ -135,7 +135,7 @@ git clone --recursive https://github.com/openstreetmap/Nominatim.git The development version does not include the country grid. Download it separately: ``` -wget -O Nominatim/data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz +wget -O Nominatim/data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz ``` ### Building Nominatim diff --git a/docs/customize/Import-Styles.md b/docs/customize/Import-Styles.md index fcd02ae1..6085d4e4 100644 --- a/docs/customize/Import-Styles.md +++ b/docs/customize/Import-Styles.md @@ -1,149 +1,439 @@ ## Configuring the Import -Which OSM objects are added to the database and which of the tags are used -can be configured via the import style configuration file. This -is a JSON file which contains a list of rules which are matched against every -tag of every object and then assign the tag its specific role. +In the very first step of a Nominatim import, OSM data is loaded into the +database. Nominatim uses [osm2pgsql](https://osm2pgsql.org) for this task. +It comes with a [flex style](https://osm2pgsql.org/doc/manual.html#the-flex-output) +specifically tailored to filter and convert OSM data into Nominatim's +internal data representation. -The style to use is given by the `NOMINATIM_IMPORT_STYLE` configuration -option. There are a number of default styles, which are explained in detail -in the [Import section](../admin/Import.md#filtering-imported-data). These -standard styles may be referenced by their name. +There are a number of default configurations for the flex style which +result in geocoding databases of different detail. The +[Import section](../admin/Import.md#filtering-imported-data) explains +these default configurations in detail. You can also create your own custom style. Put the style file into your project directory and then set `NOMINATIM_IMPORT_STYLE` to the name of the file. It is always recommended to start with one of the standard styles and customize -those. You find the standard styles under the name `import-.style` +those. You find the standard styles under the name `import-.lua` in the standard Nominatim configuration path (usually `/etc/nominatim` or `/usr/local/etc/nominatim`). -The remainder of the page describes the format of the file. +The remainder of the page describes how the flex style works and how to +customize it. -### Configuration Rules +### The `flex-base.lua` module -A single rule looks like this: +The core of Nominatim's flex import configuration is the `flex-base` module. +It defines the table layout used by Nominatim and provides standard +implementations for the import callbacks that make it easy to customize +how OSM tags are used by Nominatim. -```json -{ - "keys" : ["key1", "key2", ...], - "values" : { - "value1" : "prop", - "value2" : "prop1,prop2" - } -} -``` - -A rule first defines a list of keys to apply the rule to. This is always a list -of strings. The string may have four forms. An empty string matches against -any key. A string that ends in an asterisk `*` is a prefix match and accordingly -matches against any key that starts with the given string (minus the `*`). A -suffix match can be defined similarly with a string that starts with a `*`. Any -other string constitutes an exact match. +Every custom style should include this module to make sure that the correct +tables are created. Thus start your custom style as follows: -The second part of the rules defines a list of values and the properties that -apply to a successful match. Value strings may be either empty, which -means that they match any value, or describe an exact match. Prefix -or suffix matching of values is not possible. +``` lua +local flex = require('flex-base') -For a rule to match, it has to find a valid combination of keys and values. The -resulting property is that of the matched values. +``` -The rules in a configuration file are processed sequentially and the first -match for each tag wins. +The following sections explain how the module can be customized. + + +### Changing the recognized tags + +If you just want to change which OSM tags are recognized during import, +then there are a number of convenience functions to set the tag lists used +during the processing. + +!!! warning + There are no built-in defaults for the tag lists, so all the functions + need to be called from your style script to fully process the data. + Make sure you start from one of the default style and only modify + the data you are interested in. You can also derive your style from an + existing style by importing the appropriate module, e.g. + `local flex = require('import-street')`. + +Many of the following functions take _key match lists_. These lists can +contain three kinds of strings to match against tag keys: +A string that ends in an asterisk `*` is a prefix match and accordingly matches +against any key that starts with the given string (minus the `*`). +A suffix match can be defined similarly with a string that starts with a `*`. +Any other string is matched exactly against tag keys. + + +#### `set_main_tags()` - principal tags + +If a principal or main tag is found on an OSM object, then the object +is included in Nominatim's search index. A single object may also have +multiple main tags. In that case, the object will be included multiple +times in the index, once for each main tag. + +The flex script distinguishes between four types of main tags: + +* __always__: a main tag that is used unconditionally +* __named__: consider this main tag only, if the object has a proper name + (a reference is not enough, see below). +* __named_with_key__: consider this main tag only, when the object has + a proper name with a domain prefix. For example, if the main tag is + `bridge=yes`, then it will only be added as an extra row, if there is + a tag `bridge:name[:XXX]` for the same object. If this property is set, + all other names that are not domain-specific are ignored. +* __fallback__: use this main tag only, if there is no other main tag. + Fallback always implied `named`, i.e. fallbacks are only tried for + named objects. + +The `set_main_tags()` function takes exactly one table parameter which +defines the keys and key/value combinations to include and the kind of +main tag. Each lua table key defines an OSM tag key. The value may +be a string defining the kind of main key as described above. Then the tag will +be considered a main tag for any possible value. To further restrict +which values are acceptable, give a table with the permitted values +and their kind of main tag. If the table contains a simple value without +key, then this is used as default for values that are not listed. + +!!! example + ``` lua + local flex = require('import-full') + + flex.set_main_tags{ + boundary = {administrative = 'named'}, + highway = {'always', street_lamp = 'named'}, + landuse = 'fallback' + } + ``` + + In this example an object with a `boundary` tag will only be included + when it has a value of `administrative`. Objects with `highway` tags are + always included. However when the value is `street_lamp` then the object + must have a name, too. With any other value, the object is included + independently of the name. Finally, if a `landuse` tag is present then + it will be used independely of the concrete value if neither boundary + nor highway tags were found and the object is named. + + +#### `set_prefilters()` - ignoring tags + +Pre-filtering of tags allows to ignore them for any further processing. +Thus pre-filtering takes precedence over any other tag processing. This is +useful when some specific key/value combinations need to be excluded from +processing. When tags are filtered, they may either be deleted completely +or moved to `extratags`. Extra tags are saved with the object and returned +to the user when requested, but are not used otherwise. + +`set_prefilters()` takes a table with four optional fields: + +* __delete_keys__ is a _key match list_ for tags that should be deleted +* __delete_tags__ contains a table of tag keys pointing to a list of tag + values. Tags with matching key/value pairs are deleted. +* __extra_keys__ is a _key match list_ for tags which should be saved into + extratags +* __delete_tags__ contains a table of tag keys pointing to a list of tag + values. Tags with matching key/value pairs are moved to extratags. + +Key list may contain three kinds of strings: +A string that ends in an asterisk `*` is a prefix match and accordingly matches +against any key that starts with the given string (minus the `*`). +A suffix match can be defined similarly with a string that starts with a `*`. +Any other string is matched exactly against tag keys. + +!!! example + ``` lua + local flex = require('import-full') + + flex.set_prefilters{ + delete_keys = {'source', 'source:*'}, + extra_tags = {amenity = {'yes', 'no'}} + } + flex.set_main_tags{ + amenity = 'always' + } + ``` -A rule where key and value are the empty string is special. This defines the -fallback when none of the rules match. The fallback is always used as a last -resort when nothing else matches, no matter where the rule appears in the file. -Defining multiple fallback rules is not allowed. What happens in this case, -is undefined. + In this example any tags `source` and tags that begin with `source:` are + deleted before any other processing is done. Getting rid of frequent tags + this way can speed up the import. -### Tag Properties + Tags with `amenity=yes` or `amenity=no` are moved to extratags. Later + all tags with an `amenity` key are made a main tag. This effectively means + that Nominatim will use all amenity tags except for those with value + yes and no. -One or more of the following properties may be given for each tag: +#### `set_name_tags()` - defining names -* `main` +The flex script distinguishes between two kinds of names: - A principal tag. A new row will be added for the object with key and value - as `class` and `type`. +* __main__: the primary names make an object fully searchable. + Main tags of type _named_ will only cause the object to be included when + such a primary name is present. Primary names are usually those found + in the `name` tag and its variants. +* __extra__: extra names are still added to the search index but they are + alone not sufficient to make an object named. -* `with_name` +`set_name_tags()` takes a table with two optional fields `main` and `extra`. +They take _key match lists_ for main and extra names respectively. - When the tag is a principal tag (`main` property set): only really add a new - row, if there is any name tag found (a reference tag is not sufficient, see - below). +!!! example + ``` lua + local flex = require('flex-base') -* `with_name_key` + flex.set_main_tags{highway = {traffic_light = 'named'}} + flex.set_name_tags{main = {'name', 'name:*'}, + extra = {'ref'} + } + ``` - When the tag is a principal tag (`main` property set): only really add a new - row, if there is also a name tag that matches the key of the principal tag. - For example, if the main tag is `bridge=yes`, then it will only be added as - an extra row, if there is a tag `bridge:name[:XXX]` for the same object. - If this property is set, all other names that are not domain-specific are - ignored. + This example creates a search index over traffic lights but will + only include those that have a common name and not those which just + have some reference ID from the city. -* `fallback` +#### `set_address_tags()` - defining address parts - When the tag is a principal tag (`main` property set): only really add a new - row, when no other principal tags for this object have been found. Only one - fallback tag can win for an object. +Address tags will be used to build up the address of an object. -* `operator` +`set_address_tags()` takes a table with arbitrary fields pointing to +_key match lists_. To fields have a special meaning: - When the tag is a principal tag (`main` property set): also include the - `operator` tag in the list of names. This is a special construct for an - out-dated tagging practise in OSM. Fuel stations and chain restaurants - in particular used to have the name of the chain tagged as `operator`. - These days the chain can be more commonly found in the `brand` tag but - there is still enough old data around to warrant this special case. +__main__ defines +the tags that make a full address object out of the OSM object. This +is usually the housenumber or variants thereof. If a main address tag +appears, then the object will always be included, if necessary with a +fallback of `place=house`. If the key has a prefix of `addr:` or `is_in:` +this will be stripped. -* `name` +__extra__ defines all supplementary tags for addresses, tags like `addr:street`, `addr:city` etc. If the key has a prefix of `addr:` or `is_in:` this will be stripped. - Add tag to the list of names. +All other fields will be handled as summary fields. If a key matches the +key match list, then its value will be added to the address tags with the +name of the field as key. If multiple tags match, then an arbitrary one +wins. -* `ref` +Country tags are handled slightly special. Only tags with a two-letter code +are accepted, all other values are discarded. - Add tag to the list of names as a reference. At the moment this only means - that the object is not considered to be named for `with_name`. +!!! example + ``` lua + local flex = require('import-full') -* `address` + flex.set_address_tags{ + main = {'addr:housenumber'}, + extra = {'addr:*'}, + postcode = {'postal_code', 'postcode', 'addr:postcode'}, + country = {'country-code', 'ISO3166-1'} + } + ``` - Add tag to the list of address tags. If the tag starts with `addr:` or - `is_in:`, then this prefix is cut off before adding it to the list. + In this example all tags which begin with `addr:` will be saved in + the address tag list. If one of the tags is `addr:housenumber`, the + object will fall back to be entered as a `place=house` in the database + unless there is another interested main tag to be found. -* `postcode` + Tags with keys `country-code` and `ISO3166-1` are saved with their + value under `country` in the address tag list. The same thing happens + to postcodes, they will always be saved under the key `postcode` thus + normalizing the multitude of keys that are used in the OSM database. - Add the value as a postcode to the address tags. If multiple tags are - candidate for postcodes, one wins out and the others are dropped. -* `country` +#### `set_unused_handling()` - processing remaining tags - Add the value as a country code to the address tags. The value must be a - two letter country code, otherwise it is ignored. If there are multiple - tags that match, then one wins out and the others are dropped. +This function defines what to do with tags that remain after all tags +have been classified using the functions above. There are two ways in +which the function can be used: -* `house` +`set_unused_handling(delete_keys = ..., delete_tags = ...)` deletes all +keys that match the descriptions in the parameters and moves all remaining +tags into the extratags list. +`set_unused_handling(extra_keys = ..., extra_tags = ...)` moves all tags +matching the parameters into the extratags list and then deletes the remaining +tags. For the format of the parameters see the description in `set_prefilters()` +above. - If no principle tags can be found for the object, still add the object with - `class`=`place` and `type`=`house`. Use this for address nodes that have no - other function. +!!! example + ``` lua + local flex = require('import-full') -* `interpolation` + flex.set_address_tags{ + main = {'addr:housenumber'}, + extra = {'addr:*', 'tiger:county'} + } + flex.set_unused_handling{delete_keys = {'tiger:*'}} + ``` - Add this object as an address interpolation (appears as `class`=`place` and - `type`=`houses` in the database). + In this example all remaining tags except those beginning with `tiger:` + are moved to the extratags list. Note that it is not possible to + already delete the tiger tags with `set_prefilters()` because that + would remove tiger:county before the address tags are processed. -* `extra` +### Customizing osm2pgsql callbacks - Add tag to the list of extra tags. +osm2pgsql expects the flex style to implement three callbacks, one process +function per OSM type. If you want to implement special handling for +certain OSM types, you can override the default implementations provided +by the flex-base module. -* `skip` +#### Changing the relation types to be handled - Skip the tag completely. Useful when a custom default fallback is defined - or to define exceptions to rules. +The default scripts only allows relations of type `multipolygon`, `boundary` +and `waterway`. To add other types relations, set `RELATION_TYPES` for +the type to the kind of geometry that should be created. The following +kinds of geometries can be used: -A rule can define as many of these properties for one match as it likes. For -example, if the property is `"main,extra"` then the tag will open a new row -but also have the tag appear in the list of extra tags. +* __relation_as_multipolygon__ creates a (Multi)Polygon from the ways in + the relation. If the ways do not form a valid area, then the object is + silently discarded. +* __relation_as_multiline__ creates a (Mutli)LineString from the ways in + the relations. Ways are combined as much as possible without any regards + to their order in the relation. + +!!! Example + ``` lua + local flex = require('import-full') + + flex.RELATION_TYPES['site'] = flex.relation_as_multipolygon + ``` + + With this line relations of `type=site` will be included in the index + according to main tags found. This only works when the site relation + resolves to a valid area. Nodes in the site relation are not part of the + geometry. + + +#### Adding additional logic to processing functions + +The default processing functions are also exported by the flex-base module +as `process_node`, `process_way` and `process_relation`. These can be used +to implement your own processing functions with some additional processing +logic. + +!!! Example + ``` lua + local flex = require('import-full') + + function osm2pgsql.process_relation(object) + if object.tags.boundary ~= 'administrative' or object.tags.admin_level ~= '2' then + flex.process_relation(object) + end + end + ``` + + This example discards all country-level boundaries and uses standard + handling for everything else. This can be useful if you want to use + your own custom country boundaries. + + +### Customizing the main processing function + +The main processing function of the flex style can be found in the function +`process_tags`. This function is called for all OSM object kinds and is +responsible for filtering the tags and writing out the rows into Postgresql. + +!!! Example + ``` lua + local flex = require('import-full') + + local original_process_tags = flex.process_tags + + function flex.process_tags(o) + if o.object.tags.highway ~= nil and o.object.tags.access == 'no' then + return + end + + original_process_tags(o) + end + ``` + + This example shows the most simple customization of the process_tags function. + It simply adds some additional processing before running the original code. + To do that, first save the original function and then overwrite process_tags + from the module. In this example all highways which are not accessible + by anyone will be ignored. + + +#### The `Place` class + +The `process_tags` function receives a Lua object of `Place` type which comes +with some handy functions to collect the data necessary for geocoding and +writing it into the place table. Always use this object to fill the table. + +The Place class has some attributes which you may access read-only: + +* __object__ is the original OSM object data handed in by osm2pgsql +* __admin_level__ is the content of the admin_level tag, parsed into an + integer and normalized to a value between 0 and 15 +* __has_name__ is a boolean indicating if the object has a full name +* __names__ is a table with the collected list of name tags +* __address__ is a table with the collected list of address tags +* __extratags__ is a table with the collected list of additional tags to save + +There are a number of functions to fill these fields. All functions expect +a table parameter with fields as indicated in the description. +Many of these functions expect match functions which are described in detail +further below. + +* __delete{match=...}__ removes all tags that match the match function given + in _match_. +* __grab_extratags{match=...}__ moves all tags that match the match function + given in _match_ into extratags. Returns the number of tags moved. +* __clean{delete=..., extra=...}__ deletes all tags that match _delete_ and + moves the ones that match _extra_ into extratags +* __grab_address_parts{groups=...}__ moves matching tags into the address table. + _groups_ must be a group match function. Tags of the group `main` and + `extra` are added to the address table as is but with `addr:` and `is_in:` + prefixes removed from the tag key. All other groups are added with the + group name as key and the value from the tag. Multiple values of the same + group overwrite each other. The function returns the number of tags saved + from the main group. +* __grab_main_parts{groups=...}__ moves matching tags into the name table. + _groups_ must be a group match function. If a tags of the group `main` is + present, the object will be marked as having a name. Tags of group `house` + produce a fallback to `place=house`. This fallback is return by the function + if present. + +There are two functions to write a row into the place table. Both functions +expect the main tag (key and value) for the row and then use the collected +information from the name, address, extratags etc. fields to complete the row. +They also have a boolean parameter `save_extra_mains` which defines how any +unprocessed tags are handled: when True, the tags will be saved as extratags, +when False, they will be simply discarded. + +* __write_row(key, value, save_extra_mains)__ creates a new table row from + the current state of the Place object. +* __write_place(key, value, mtype, save_extra_mains)__ creates a new row + conditionally. When value is nil, the function will attempt to look up the + value in the object tags. If value is still nil or mtype is nil, the row + is ignored. An mtype of `always` will then always write out the row, + a mtype of `named` only, when the object has a full name. When mtype + is `named_with_key`, the function checks for a domain name, i.e. a name + tag prefixed with the name of the main key. Only if at least one is found, + the row will be written. The names are replaced with the domain names found. + +#### Match functions + +The Place functions usually expect either a _match function_ or a +_group match function_ to find the tags to apply their function to. + +The __match function__ is a Lua function which takes two parameters, +key and value, and returns a boolean to indicate that a tag matches. The +flex-base module has a convenience function `tag_match()` to create such a +function. It takes a table with two optional fields: `keys` takes a key match +list (see above), `tags` takes a table with keys that point to a list of +possible values, thus defining key/value matches. + +The __group match function__ is a Lua function which also takes two parameters, +key and value, and returns a string indicating to which group or type they +belong to. The `tag_group()` can be used to create such a function. It expects +a table where the group names are the keys and the values are a key match list. + + + +### Using the gazetteer output of osm2pgsql + +Nominatim still allows you to configure the gazetteer output to remain +backwards compatible with older imports. It will be automatically used +when the style file name ends in `.style`. For documentation of the +old import style, please refer to the documentation of older releases +of Nominatim. Do not use the gazetteer output for new imports. There is no +guarantee that new versions of Nominatim are fully compatible with the +gazetteer output. ### Changing the Style of Existing Databases diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index ab7dec30..eb624d8b 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -51,6 +51,7 @@ nav: markdown_extensions: - codehilite - admonition + - pymdownx.superfences - def_list - toc: permalink:  diff --git a/nominatim/config.py b/nominatim/config.py index 1728c291..e0f19b04 100644 --- a/nominatim/config.py +++ b/nominatim/config.py @@ -172,7 +172,7 @@ class Configuration: style = getattr(self, 'IMPORT_STYLE') if style in ('admin', 'street', 'address', 'full', 'extratags'): - return self.config_dir / f'import-{style}.style' + return self.config_dir / f'import-{style}.lua' return self.find_config_file('', 'IMPORT_STYLE') diff --git a/nominatim/tools/exec_utils.py b/nominatim/tools/exec_utils.py index 2e5c4801..ab2ccc7c 100644 --- a/nominatim/tools/exec_utils.py +++ b/nominatim/tools/exec_utils.py @@ -125,7 +125,7 @@ def run_osm2pgsql(options: Mapping[str, Any]) -> None: ] if str(options['osm2pgsql_style']).endswith('.lua'): - env['LUA_PATH'] = ';'.join((str(options['osm2pgsql_style_path'] / 'flex-base.lua'), + env['LUA_PATH'] = ';'.join((str(options['osm2pgsql_style_path'] / '?.lua'), os.environ.get('LUAPATH', ';'))) cmd.extend(('--output', 'flex')) else: diff --git a/osm2pgsql b/osm2pgsql index 6a5d2500..4facd1ae 160000 --- a/osm2pgsql +++ b/osm2pgsql @@ -1 +1 @@ -Subproject commit 6a5d2500e9689f55485d186306aadc55560085fd +Subproject commit 4facd1aea451cea220261c361698b8e5f18a9327 diff --git a/settings/flex-base.lua b/settings/flex-base.lua index fe3ce32a..0e112736 100644 --- a/settings/flex-base.lua +++ b/settings/flex-base.lua @@ -1,9 +1,19 @@ -- Core functions for Nominatim import flex style. -- +local module = {} + +local PRE_DELETE = nil +local PRE_EXTRAS = nil +local MAIN_KEYS = nil +local NAMES = nil +local ADDRESS_TAGS = nil +local SAVE_EXTRA_MAINS = false +local POSTCODE_FALLBACK = true + -- The single place table. -place_table = osm2pgsql.define_table{ +local place_table = osm2pgsql.define_table{ name = "place", ids = { type = 'any', id_column = 'osm_id', type_column = 'osm_type' }, columns = { @@ -14,7 +24,25 @@ place_table = osm2pgsql.define_table{ { column = 'address', type = 'hstore' }, { column = 'extratags', type = 'hstore' }, { column = 'geometry', type = 'geometry', projection = 'WGS84', not_null = true }, - } + }, + indexes = {} +} + +------------ Geometry functions for relations --------------------- + +function module.relation_as_multipolygon(o) + return o:as_multipolygon() +end + +function module.relation_as_multiline(o) + return o:as_multilinestring():line_merge() +end + + +module.RELATION_TYPES = { + multipolygon = module.relation_as_multipolygon, + boundary = module.relation_as_multipolygon, + waterway = module.relation_as_multiline } ------------- Place class ------------------------------------------ @@ -43,6 +71,17 @@ function Place.new(object, geom_func) return self end +function Place:clean(data) + for k, v in pairs(self.object.tags) do + if data.delete ~= nil and data.delete(k, v) then + self.object.tags[k] = nil + elseif data.extra ~= nil and data.extra(k, v) then + self.extratags[k] = v + self.object.tags[k] = nil + end + end +end + function Place:delete(data) if data.match ~= nil then for k, v in pairs(self.object.tags) do @@ -69,31 +108,37 @@ function Place:grab_extratags(data) return count end -function Place:grab_address(data) +local function strip_address_prefix(k) + if k:sub(1, 5) == 'addr:' then + return k:sub(6) + end + + if k:sub(1, 6) == 'is_in:' then + return k:sub(7) + end + + return k +end + + +function Place:grab_address_parts(data) local count = 0 - if data.match ~= nil then + if data.groups ~= nil then for k, v in pairs(self.object.tags) do - if data.match(k, v) then - self.object.tags[k] = nil + local atype = data.groups(k, v) - if data.include_on_name == true then + if atype ~= nil then + if atype == 'main' then self.has_name = true - end - - if data.out_key ~= nil then - self.address[data.out_key] = v - return 1 - end - - if k:sub(1, 5) == 'addr:' then - self.address[k:sub(6)] = v - elseif k:sub(1, 6) == 'is_in:' then - self.address[k:sub(7)] = v + self.address[strip_address_prefix(k)] = v + count = count + 1 + elseif atype == 'extra' then + self.address[strip_address_prefix(k)] = v else - self.address[k] = v + self.address[atype] = v end - count = count + 1 + self.object.tags[k] = nil end end end @@ -101,36 +146,30 @@ function Place:grab_address(data) return count end -function Place:set_address(key, value) - self.address[key] = value -end -function Place:grab_name(data) - local count = 0 +function Place:grab_name_parts(data) + local fallback = nil - if data.match ~= nil then + if data.groups ~= nil then for k, v in pairs(self.object.tags) do - if data.match(k, v) then - self.object.tags[k] = nil + local atype = data.groups(k, v) + + if atype ~= nil then self.names[k] = v - if data.include_on_name ~= false then + self.object.tags[k] = nil + if atype == 'main' then self.has_name = true + elseif atype == 'house' then + self.has_name = true + fallback = {'place', 'house', 'always'} end - count = count + 1 end end end - return count -end - -function Place:grab_tag(key) - return self.object:grab_tag(key) + return fallback end -function Place:tags() - return self.object.tags -end function Place:write_place(k, v, mtype, save_extra_mains) if mtype == nil then @@ -214,7 +253,7 @@ function Place:write_row(k, v, save_extra_mains) end -function tag_match(data) +function module.tag_match(data) if data == nil or next(data) == nil then return nil end @@ -276,17 +315,72 @@ function tag_match(data) end +function module.tag_group(data) + if data == nil or next(data) == nil then + return nil + end + + local fullmatches = {} + local key_prefixes = {} + local key_suffixes = {} + + for group, tags in pairs(data) do + for _, key in pairs(tags) do + if key:sub(1, 1) == '*' then + if #key > 1 then + if key_suffixes[#key - 1] == nil then + key_suffixes[#key - 1] = {} + end + key_suffixes[#key - 1][key:sub(2)] = group + end + elseif key:sub(#key, #key) == '*' then + if key_prefixes[#key - 1] == nil then + key_prefixes[#key - 1] = {} + end + key_prefixes[#key - 1][key:sub(1, #key - 1)] = group + else + fullmatches[key] = group + end + end + end + + return function (k, v) + local val = fullmatches[k] + if val ~= nil then + return val + end + + for slen, slist in pairs(key_suffixes) do + if #k >= slen then + val = slist[k:sub(-slen)] + if val ~= nil then + return val + end + end + end + + for slen, slist in pairs(key_prefixes) do + if #k >= slen then + val = slist[k:sub(1, slen)] + if val ~= nil then + return val + end + end + end + end +end + -- Process functions for all data types -function osm2pgsql.process_node(object) +function module.process_node(object) local function geom_func(o) return o:as_point() end - process_tags(Place.new(object, geom_func)) + module.process_tags(Place.new(object, geom_func)) end -function osm2pgsql.process_way(object) +function module.process_way(object) local function geom_func(o) local geom = o:as_polygon() @@ -298,30 +392,24 @@ function osm2pgsql.process_way(object) return geom end - process_tags(Place.new(object, geom_func)) -end - -function relation_as_multipolygon(o) - return o:as_multipolygon() -end - -function relation_as_multiline(o) - return o:as_multilinestring():line_merge() + module.process_tags(Place.new(object, geom_func)) end -function osm2pgsql.process_relation(object) - local geom_func = RELATION_TYPES[object.tags.type] +function module.process_relation(object) + local geom_func = module.RELATION_TYPES[object.tags.type] if geom_func ~= nil then - process_tags(Place.new(object, geom_func)) + module.process_tags(Place.new(object, geom_func)) end end -function process_tags(o) - local fallback +-- The process functions are used by default by osm2pgsql. +osm2pgsql.process_node = module.process_node +osm2pgsql.process_way = module.process_way +osm2pgsql.process_relation = module.process_relation - o:delete{match = PRE_DELETE} - o:grab_extratags{match = PRE_EXTRAS} +function module.process_tags(o) + o:clean{delete = PRE_DELETE, extra = PRE_EXTRAS} -- Exception for boundary/place double tagging if o.object.tags.boundary == 'administrative' then @@ -330,54 +418,93 @@ function process_tags(o) end} end + -- name keys + local fallback = o:grab_name_parts{groups=NAMES} + -- address keys - o:grab_address{match=COUNTRY_TAGS, out_key='country'} + if o:grab_address_parts{groups=ADDRESS_TAGS} > 0 and fallback == nil then + fallback = {'place', 'house', 'always'} + end if o.address.country ~= nil and #o.address.country ~= 2 then o.address['country'] = nil end - if o:grab_name{match=HOUSENAME_TAGS} > 0 then - fallback = {'place', 'house'} - end - if o:grab_address{match=HOUSENUMBER_TAGS, include_on_name = true} > 0 and fallback == nil then - fallback = {'place', 'house'} + if POSTCODE_FALLBACK and fallback == nil and o.address.postcode ~= nil then + fallback = {'place', 'postcode', 'always'} end - if o:grab_address{match=POSTCODES, out_key='postcode'} > 0 and fallback == nil then - fallback = {'place', 'postcode'} + + if o.address.interpolation ~= nil then + o:write_place('place', 'houses', 'always', SAVE_EXTRA_MAINS) + return end - local is_interpolation = o:grab_address{match=INTERPOLATION_TAGS} > 0 + o:clean{delete = POST_DELETE, extra = POST_EXTRAS} - o:grab_address{match=ADDRESS_TAGS} + -- collect main keys + for k, v in pairs(o.object.tags) do + local ktype = MAIN_KEYS[k] + if ktype == 'fallback' then + if o.has_name then + fallback = {k, v, 'named'} + end + elseif ktype ~= nil then + o:write_place(k, v, MAIN_KEYS[k], SAVE_EXTRA_MAINS) + end + end - if is_interpolation then - o:write_place('place', 'houses', 'always', SAVE_EXTRA_MAINS) - return + if fallback ~= nil and o.num_entries == 0 then + o:write_place(fallback[1], fallback[2], fallback[3], SAVE_EXTRA_MAINS) end +end - -- name keys - o:grab_name{match = NAMES} - o:grab_name{match = REFS, include_on_name = false} +--------- Convenience functions for simple style configuration ----------------- - o:delete{match = POST_DELETE} - o:grab_extratags{match = POST_EXTRAS} - -- collect main keys - local num_mains = 0 - for k, v in pairs(o:tags()) do - num_mains = num_mains + o:write_place(k, v, MAIN_KEYS[k], SAVE_EXTRA_MAINS) +function module.set_prefilters(data) + PRE_DELETE = module.tag_match{keys = data.delete_keys, tags = data.delete_tags} + PRE_EXTRAS = module.tag_match{keys = data.extra_keys, + tags = data.extra_tags} +end + +function module.set_main_tags(data) + MAIN_KEYS = data +end + +function module.set_name_tags(data) + NAMES = module.tag_group(data) +end + +function module.set_address_tags(data) + if data.postcode_fallback ~= nil then + POSTCODE_FALLBACK = data.postcode_fallback + data.postcode_fallback = nil end - if num_mains == 0 then - for tag, mtype in pairs(MAIN_FALLBACK_KEYS) do - if o:write_place(tag, nil, mtype, SAVE_EXTRA_MAINS) > 0 then - return - end - end + ADDRESS_TAGS = module.tag_group(data) +end - if fallback ~= nil then - o:write_place(fallback[1], fallback[2], 'always', SAVE_EXTRA_MAINS) - end +function module.set_unused_handling(data) + if data.extra_keys == nil and data.extra_tags == nil then + POST_DELETE = module.tag_match{keys = data.delete_keys, tags = data.delete_tags} + POST_EXTRAS = nil + SAVE_EXTRA_MAINS = true + elseif data.delete_keys == nil and data.delete_tags == nil then + POST_DELETE = nil + POST_EXTRAS = module.tag_match{keys = data.extra_keys, tags = data.extra_tags} + SAVE_EXTRA_MAINS = false + else + error("unused handler can have only 'extra_keys' or 'delete_keys' set.") end end +function set_relation_types(data) + module.RELATION_TYPES = {} + for k, v in data do + if v == 'multipolygon' then + module.RELATION_TYPES[k] = module.relation_as_multipolygon + elseif v == 'multiline' then + module.RELATION_TYPES[k] = module.relation_as_multiline + end + end +end +return module diff --git a/settings/import-address.lua b/settings/import-address.lua new file mode 100644 index 00000000..00d089cc --- /dev/null +++ b/settings/import-address.lua @@ -0,0 +1,69 @@ +local flex = require('flex-base') + +flex.set_main_tags{ + highway = {'always', + street_lamp = 'named', + traffic_signals = 'named', + service = 'named', + cycleway = 'named', + path = 'named', + footway = 'named', + steps = 'named', + bridleway = 'named', + track = 'named', + motorway_link = 'named', + trunk_link = 'named', + primary_link = 'named', + secondary_link = 'named', + tertiary_link = 'named'}, + boundary = {administrative = 'named', + postal_code = 'named'}, + landuse = 'fallback', + place = 'always' +} + +flex.set_prefilters{delete_keys = {'building', 'source', + 'source', '*source', 'type', + 'is_in:postcode', '*:wikidata', + '*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*', + 'name:etymology', 'name:signed', 'name:botanical', + 'addr:street:name', 'addr:street:type'}, + delete_tags = {highway = {'no', 'turning_circle', 'mini_roundabout', + 'noexit', 'crossing', 'give_way', 'stop'}, + landuse = {'cemetry', 'no'}, + boundary = {'place'}}, + extra_keys = {'wikipedia', 'wikipedia:*', 'wikidata', 'capital', 'area'} + } + +flex.set_name_tags{main = {'name', 'name:*', + 'int_name', 'int_name:*', + 'nat_name', 'nat_name:*', + 'reg_name', 'reg_name:*', + 'loc_name', 'loc_name:*', + 'old_name', 'old_name:*', + 'alt_name', 'alt_name:*', 'alt_name_*', + 'official_name', 'official_name:*', + 'place_name', 'place_name:*', + 'short_name', 'short_name:*', 'brand'}, + extra = {'ref', 'int_ref', 'nat_ref', 'reg_ref', + 'loc_ref', 'old_ref', + 'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'}, + house = {'addr:housename'} + } + +flex.set_address_tags{main = {'addr:housenumber', + 'addr:conscriptionnumber', + 'addr:streetnumber'}, + extra = {'addr:*', 'is_in:*', 'tiger:county'}, + postcode = {'postal_code', 'postcode', 'addr:postcode', + 'tiger:zip_left', 'tiger:zip_right'}, + country = {'country_code', 'ISO3166-1', + 'addr:country_code', 'is_in:country_code', + 'addr:country', 'is_in:country'}, + interpolation = {'addr:interpolation'} + } + + +flex.set_unused_handling{extra_keys = {'place'}} + +return flex diff --git a/settings/import-address.style b/settings/import-address.style deleted file mode 100644 index b4457fa5..00000000 --- a/settings/import-address.style +++ /dev/null @@ -1,124 +0,0 @@ -[ -{ - "keys" : [ "" ], - "values" : { - "no" : "skip" - } -}, -{ "keys" : ["wikipedia", "wikipedia:*", "wikidata", "area"], - "values" : { - "" : "extra" - } -}, -{ - "keys" : ["*:prefix", "*:suffix", "name:prefix:*", "name:suffix:*", - "name:etymology", "name:signed", "name:botanical", "*:wikidata", - "addr:street:name", "addr:street:type"], - "values" : { - "" : "skip" - } -}, -{ - "keys" : ["ref", "int_ref", "nat_ref", "reg_ref", "loc_ref", "old_ref", - "iata", "icao", "pcode", "ISO3166-2"], - "values" : { - "" : "ref" - } -}, -{ - "keys" : ["name", "name:*", "int_name", "int_name:*", "nat_name", "nat_name:*", - "reg_name", "reg_name:*", "loc_name", "loc_name:*", - "old_name", "old_name:*", "alt_name", "alt_name:*", "alt_name_*", - "official_name", "official_name:*", "place_name", "place_name:*", - "short_name", "short_name:*", "brand"], - "values" : { - "" : "name" - } -}, -{ - "keys" : ["landuse"], - "values" : { - "cemetry" : "skip", - "" : "fallback,with_name" - } -}, -{ - "keys" : ["boundary"], - "values" : { - "administrative" : "main", - "postal_code" : "main" - } -}, -{ - "keys" : ["place"], - "values" : { - "" : "main" - } -}, -{ - "keys" : ["addr:housename"], - "values" : { - "" : "name,house" - } -}, -{ - "keys" : ["addr:housenumber", "addr:conscriptionnumber", "addr:streetnumber"], - "values" : { - "" : "address,house" - } -}, -{ - "keys" : ["addr:interpolation"], - "values" : { - "" : "interpolation,address" - } -}, -{ - "keys" : ["postal_code", "postcode", "addr:postcode", - "tiger:zip_left", "tiger:zip_right"], - "values" : { - "" : "postcode,fallback" - } -}, -{ - "keys" : ["country_code", "ISO3166-1", "is_in:country_code", "is_in:country", - "addr:country", "addr:country_code"], - "values" : { - "" : "country" - } -}, -{ - "keys" : ["addr:*", "is_in:*", "tiger:county"], - "values" : { - "" : "address" - } -}, -{ - "keys" : ["highway"], - "values" : { - "motorway" : "main", - "trunk" : "main", - "primary" : "main", - "secondary" : "main", - "tertiary" : "main", - "unclassified" : "main", - "residential" : "main", - "living_street" : "main", - "pedestrian" : "main", - "road" : "main", - "service" : "main,with_name", - "cycleway" : "main,with_name", - "path" : "main,with_name", - "footway" : "main,with_name", - "steps" : "main,with_name", - "bridleway" : "main,with_name", - "track" : "main,with_name", - "byway": "main,with_name", - "motorway_link" : "main,with_name", - "trunk_link" : "main,with_name", - "primary_link" : "main,with_name", - "secondary_link" : "main,with_name", - "tertiary_link" : "main,with_name" - } -} -] diff --git a/settings/import-admin.lua b/settings/import-admin.lua new file mode 100644 index 00000000..a1164acc --- /dev/null +++ b/settings/import-admin.lua @@ -0,0 +1,46 @@ +local flex = require('flex-base') + +flex.set_main_tags{ + boundary = {administrative = 'named'}, + landuse = 'fallback', + place = 'always' +} + +flex.set_prefilters{delete_keys = {'building', 'source', 'highway', + 'addr:housenumber', 'addr:street', 'addr:city', + 'source', '*source', 'type', + 'is_in:postcode', '*:wikidata', + '*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*', + 'name:etymology', 'name:signed', 'name:botanical', + 'addr:street:name', 'addr:street:type'}, + delete_tags = {landuse = {'cemetry', 'no'}, + boundary = {'place'}}, + extra_keys = {'wikipedia', 'wikipedia:*', 'wikidata', 'capital'} + } + +flex.set_name_tags{main = {'name', 'name:*', + 'int_name', 'int_name:*', + 'nat_name', 'nat_name:*', + 'reg_name', 'reg_name:*', + 'loc_name', 'loc_name:*', + 'old_name', 'old_name:*', + 'alt_name', 'alt_name:*', 'alt_name_*', + 'official_name', 'official_name:*', + 'place_name', 'place_name:*', + 'short_name', 'short_name:*', 'brand'}, + extra = {'ref', 'int_ref', 'nat_ref', 'reg_ref', + 'loc_ref', 'old_ref', + 'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'} + } + +flex.set_address_tags{extra = {'addr:*', 'is_in:*'}, + postcode = {'postal_code', 'postcode', 'addr:postcode'}, + country = {'country_code', 'ISO3166-1', + 'addr:country_code', 'is_in:country_code', + 'addr:country', 'is_in:country'}, + postcode_fallback = false + } + +flex.set_unused_handling{extra_keys = {'place'}} + +return flex diff --git a/settings/import-admin.style b/settings/import-admin.style deleted file mode 100644 index 37d03a83..00000000 --- a/settings/import-admin.style +++ /dev/null @@ -1,77 +0,0 @@ -[ -{ "keys" : ["wikipedia", "wikipedia:*", "wikidata"], - "values" : { - "" : "extra" - } -}, -{ - "keys" : ["*:prefix", "*:suffix", "name:prefix:*", "name:suffix:*", - "name:etymology", "name:signed", "name:botanical", "*:wikidata", - "addr:street:name", "addr:street:type"], - "values" : { - "" : "skip" - } -}, -{ - "keys" : ["ref", "int_ref", "nat_ref", "reg_ref", "loc_ref", "old_ref", - "iata", "icao", "pcode", "ISO3166-2"], - "values" : { - "" : "ref" - } -}, -{ - "keys" : ["name", "name:*", "int_name", "int_name:*", "nat_name", "nat_name:*", - "reg_name", "reg_name:*", "loc_name", "loc_name:*", - "old_name", "old_name:*", "alt_name", "alt_name:*", "alt_name_*", - "official_name", "official_name:*", "place_name", "place_name:*", - "short_name", "short_name:*", "brand"], - "values" : { - "" : "name" - } -}, -{ - "keys" : ["landuse"], - "values" : { - "cemetry" : "skip", - "" : "fallback,with_name" - } -}, -{ - "keys" : ["boundary"], - "values" : { - "administrative" : "main" - } -}, -{ - "keys" : ["place"], - "values" : { - "" : "main" - } -}, -{ - "keys" : ["country_code", "ISO3166-1", "is_in:country_code", "is_in:country", - "addr:country", "addr:country_code"], - "values" : { - "" : "country" - } -}, -{ - "keys" : ["addr:*", "is_in:*", "tiger:county"], - "values" : { - "" : "address" - } -}, -{ - "keys" : ["postal_code", "postcode", "addr:postcode", - "tiger:zip_left", "tiger:zip_right"], - "values" : { - "" : "postcode" - } -}, -{ - "keys" : ["capital"], - "values" : { - "" : "extra" - } -} -] diff --git a/settings/import-extratags.lua b/settings/import-extratags.lua index 7b1880ef..d634d4a1 100644 --- a/settings/import-extratags.lua +++ b/settings/import-extratags.lua @@ -1,13 +1,9 @@ -require('flex-base') +local flex = require('flex-base') -RELATION_TYPES = { - multipolygon = relation_as_multipolygon, - boundary = relation_as_multipolygon, - waterway = relation_as_multiline -} - -MAIN_KEYS = { +flex.set_main_tags{ + building = 'fallback', emergency = 'always', + healthcare = 'fallback', historic = 'always', military = 'always', natural = 'named', @@ -36,6 +32,8 @@ MAIN_KEYS = { amenity = 'always', club = 'always', craft = 'always', + junction = 'fallback', + landuse = 'fallback', leisure = 'always', office = 'always', mountain_pass = 'always', @@ -47,55 +45,42 @@ MAIN_KEYS = { place = 'always' } -MAIN_FALLBACK_KEYS = { - building = 'named', - landuse = 'named', - junction = 'named', - healthcare = 'named' -} - - -PRE_DELETE = tag_match{keys = {'note', 'note:*', 'source', 'source*', 'attribution', - 'comment', 'fixme', 'FIXME', 'created_by', 'NHD:*', - 'nhd:*', 'gnis:*', 'geobase:*', 'KSJ2:*', 'yh:*', - 'osak:*', 'naptan:*', 'CLC:*', 'import', 'it:fvg:*', - 'type', 'lacounty:*', 'ref:ruian:*', 'building:ruian:type', - 'ref:linz:*', 'is_in:postcode'}, - tags = {emergency = {'yes', 'no', 'fire_hydrant'}, - historic = {'yes', 'no'}, - military = {'yes', 'no'}, - natural = {'yes', 'no', 'coastline'}, - highway = {'no', 'turning_circle', 'mini_roundabout', - 'noexit', 'crossing', 'give_way', 'stop'}, - railway = {'level_crossing', 'no', 'rail'}, - man_made = {'survey_point', 'cutline'}, - aerialway = {'pylon', 'no'}, - aeroway = {'no'}, - amenity = {'no'}, - club = {'no'}, - craft = {'no'}, - leisure = {'no'}, - office = {'no'}, - mountain_pass = {'no'}, - shop = {'no'}, - tourism = {'yes', 'no'}, - bridge = {'no'}, - tunnel = {'no'}, - waterway = {'riverbank'}, - building = {'no'}, - boundary = {'place'}} - } - -POST_DELETE = tag_match{keys = {'tiger:*'}} - -PRE_EXTRAS = tag_match{keys = {'*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*', +flex.set_prefilters{delete_keys = {'note', 'note:*', 'source', '*source', 'attribution', + 'comment', 'fixme', 'FIXME', 'created_by', 'NHD:*', + 'nhd:*', 'gnis:*', 'geobase:*', 'KSJ2:*', 'yh:*', + 'osak:*', 'naptan:*', 'CLC:*', 'import', 'it:fvg:*', + 'type', 'lacounty:*', 'ref:ruian:*', 'building:ruian:type', + 'ref:linz:*', 'is_in:postcode'}, + delete_tags = {emergency = {'yes', 'no', 'fire_hydrant'}, + historic = {'yes', 'no'}, + military = {'yes', 'no'}, + natural = {'yes', 'no', 'coastline'}, + highway = {'no', 'turning_circle', 'mini_roundabout', + 'noexit', 'crossing', 'give_way', 'stop'}, + railway = {'level_crossing', 'no', 'rail'}, + man_made = {'survey_point', 'cutline'}, + aerialway = {'pylon', 'no'}, + aeroway = {'no'}, + amenity = {'no'}, + club = {'no'}, + craft = {'no'}, + leisure = {'no'}, + office = {'no'}, + mountain_pass = {'no'}, + shop = {'no'}, + tourism = {'yes', 'no'}, + bridge = {'no'}, + tunnel = {'no'}, + waterway = {'riverbank'}, + building = {'no'}, + boundary = {'place'}}, + extra_keys = {'*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*', 'name:etymology', 'name:signed', 'name:botanical', 'wikidata', '*:wikidata', 'addr:street:name', 'addr:street:type'} - } + } - -NAMES = tag_match{keys = {'name', 'name:*', +flex.set_name_tags{main = {'name', 'name:*', 'int_name', 'int_name:*', 'nat_name', 'nat_name:*', 'reg_name', 'reg_name:*', @@ -104,26 +89,26 @@ NAMES = tag_match{keys = {'name', 'name:*', 'alt_name', 'alt_name:*', 'alt_name_*', 'official_name', 'official_name:*', 'place_name', 'place_name:*', - 'short_name', 'short_name:*', 'brand'}} - -REFS = tag_match{keys = {'ref', 'int_ref', 'nat_ref', 'reg_ref', 'loc_ref', 'old_ref', - 'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'}} - -POSTCODES = tag_match{keys = {'postal_code', 'postcode', 'addr:postcode', - 'tiger:zip_left', 'tiger:zip_right'}} - -COUNTRY_TAGS = tag_match{keys = {'country_code', 'ISO3166-1', + 'short_name', 'short_name:*', 'brand'}, + extra = {'ref', 'int_ref', 'nat_ref', 'reg_ref', + 'loc_ref', 'old_ref', + 'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'}, + house = {'addr:housename'} + } + +flex.set_address_tags{main = {'addr:housenumber', + 'addr:conscriptionnumber', + 'addr:streetnumber'}, + extra = {'addr:*', 'is_in:*', 'tiger:county'}, + postcode = {'postal_code', 'postcode', 'addr:postcode', + 'tiger:zip_left', 'tiger:zip_right'}, + country = {'country_code', 'ISO3166-1', 'addr:country_code', 'is_in:country_code', - 'addr:country', 'is_in:country'}} - -HOUSENAME_TAGS = tag_match{keys = {'addr:housename'}} - -HOUSENUMBER_TAGS = tag_match{keys = {'addr:housenumber', 'addr:conscriptionnumber', - 'addr:streetnumber'}} - -INTERPOLATION_TAGS = tag_match{keys = {'addr:interpolation'}} + 'addr:country', 'is_in:country'}, + interpolation = {'addr:interpolation'} + } -ADDRESS_TAGS = tag_match{keys = {'addr:*', 'is_in:*', 'tiger:county'}} -SAVE_EXTRA_MAINS = true +flex.set_unused_handling{delete_keys = {'tiger:*'}} +return flex diff --git a/settings/import-extratags.style b/settings/import-extratags.style deleted file mode 100644 index 76146de2..00000000 --- a/settings/import-extratags.style +++ /dev/null @@ -1,237 +0,0 @@ -[ -{ - "keys" : ["*source"], - "values" : { - "" : "skip" - } -}, -{ - "keys" : ["*:prefix", "*:suffix", "name:prefix:*", "name:suffix:*", - "name:etymology", "name:signed", "name:botanical", "wikidata", "*:wikidata", - "addr:street:name", "addr:street:type"], - "values" : { - "" : "extra" - } -}, -{ - "keys" : ["ref", "int_ref", "nat_ref", "reg_ref", "loc_ref", "old_ref", - "iata", "icao", "pcode", "pcode:*", "ISO3166-2"], - "values" : { - "" : "ref" - } -}, -{ - "keys" : ["name", "name:*", "int_name", "int_name:*", "nat_name", "nat_name:*", - "reg_name", "reg_name:*", "loc_name", "loc_name:*", - "old_name", "old_name:*", "alt_name", "alt_name:*", "alt_name_*", - "official_name", "official_name:*", "place_name", "place_name:*", - "short_name", "short_name:*", "brand"], - "values" : { - "" : "name" - } -}, -{ - "keys" : ["addr:housename"], - "values" : { - "" : "name,house" - } -}, -{ - "keys" : ["emergency"], - "values" : { - "fire_hydrant" : "skip", - "yes" : "skip", - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["historic", "military"], - "values" : { - "no" : "skip", - "yes" : "skip", - "" : "main" - } -}, -{ - "keys" : ["natural"], - "values" : { - "yes" : "skip", - "no" : "skip", - "coastline" : "skip", - "" : "main,with_name" - } -}, -{ - "keys" : ["landuse"], - "values" : { - "cemetry" : "main,with_name", - "" : "main,fallback,with_name" - } -}, -{ - "keys" : ["highway"], - "values" : { - "no" : "skip", - "turning_circle" : "skip", - "mini_roundabout" : "skip", - "noexit" : "skip", - "crossing" : "skip", - "give_way" : "skip", - "stop" : "skip", - "street_lamp" : "main,with_name", - "traffic_signals" : "main,with_name", - "service" : "main,with_name", - "cycleway" : "main,with_name", - "path" : "main,with_name", - "footway" : "main,with_name", - "steps" : "main,with_name", - "bridleway" : "main,with_name", - "track" : "main,with_name", - "byway": "main,with_name", - "motorway_link" : "main,with_name", - "trunk_link" : "main,with_name", - "primary_link" : "main,with_name", - "secondary_link" : "main,with_name", - "tertiary_link" : "main,with_name", - "" : "main" - } -}, -{ - "keys" : ["railway"], - "values" : { - "level_crossing" : "skip", - "no" : "skip", - "rail" : "extra", - "" : "main,with_name" - } -}, -{ - "keys" : ["man_made"], - "values" : { - "survey_point" : "skip", - "cutline" : "skip", - "" : "main" - } -}, -{ - "keys" : ["aerialway"], - "values" : { - "pylon" : "skip", - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["boundary"], - "values" : { - "place" : "skip", - "postal_code" : "main", - "" : "main,with_name" - } -}, -{ - "keys" : ["aeroway", "amenity", "club", "craft", "leisure", - "office", "mountain_pass"], - "values" : { - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["shop"], - "values" : { - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["tourism"], - "values" : { - "yes" : "skip", - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["bridge", "tunnel"], - "values" : { - "" : "main,with_name_key" - } -}, -{ - "keys" : ["waterway"], - "values" : { - "riverbank" : "skip", - "" : "main,with_name" - } -}, -{ - "keys" : ["place"], - "values" : { - "" : "main" - } -}, -{ - "keys" : ["junction", "healthcare"], - "values" : { - "" : "main,fallback,with_name" - } -}, -{ - "keys" : ["postal_code", "postcode", "addr:postcode", - "tiger:zip_left", "tiger:zip_right"], - "values" : { - "" : "postcode,fallback" - } -}, -{ - "keys" : ["country_code", "ISO3166-1", "is_in:country_code", "is_in:country", - "addr:country", "addr:country_code"], - "values" : { - "" : "country" - } -}, -{ - "keys" : ["addr:housenumber", "addr:conscriptionnumber", "addr:streetnumber"], - "values" : { - "" : "address,house" - } -}, -{ - "keys" : ["addr:interpolation"], - "values" : { - "" : "interpolation,address" - } -}, -{ - "keys" : ["addr:*", "is_in:*", "tiger:county"], - "values" : { - "" : "address" - } -}, -{ - "keys" : ["building"], - "values" : { - "no" : "skip", - "" : "main,fallback,with_name" - } -}, -{ - "keys" : ["note", "note:*", "source", "source*", "attribution", - "comment", "fixme", "FIXME", "created_by", "tiger:*", "NHD:*", - "nhd:*", "gnis:*", "geobase:*", "KSJ2:*", "yh:*", - "osak:*", "naptan:*", "CLC:*", "import", "it:fvg:*", - "type", "lacounty:*", "ref:ruian:*", "building:ruian:type", - "ref:linz:*"], - "values" : { - "" : "skip" - } -}, -{ - "keys" : [""], - "values" : { - "" : "extra" - } -} -] diff --git a/settings/import-full.lua b/settings/import-full.lua new file mode 100644 index 00000000..a932fa50 --- /dev/null +++ b/settings/import-full.lua @@ -0,0 +1,114 @@ +local flex = require('flex-base') + +flex.set_main_tags{ + building = 'fallback', + emergency = 'always', + healthcare = 'fallback', + historic = 'always', + military = 'always', + natural = 'named', + landuse = 'named', + highway = {'always', + street_lamp = 'named', + traffic_signals = 'named', + service = 'named', + cycleway = 'named', + path = 'named', + footway = 'named', + steps = 'named', + bridleway = 'named', + track = 'named', + motorway_link = 'named', + trunk_link = 'named', + primary_link = 'named', + secondary_link = 'named', + tertiary_link = 'named'}, + railway = 'named', + man_made = 'always', + aerialway = 'always', + boundary = {'named', + postal_code = 'named'}, + aeroway = 'always', + amenity = 'always', + club = 'always', + craft = 'always', + junction = 'fallback', + landuse = 'fallback', + leisure = 'always', + office = 'always', + mountain_pass = 'always', + shop = 'always', + tourism = 'always', + bridge = 'named_with_key', + tunnel = 'named_with_key', + waterway = 'named', + place = 'always' +} + +flex.set_prefilters{delete_keys = {'note', 'note:*', 'source', '*source', 'attribution', + 'comment', 'fixme', 'FIXME', 'created_by', 'NHD:*', + 'nhd:*', 'gnis:*', 'geobase:*', 'KSJ2:*', 'yh:*', + 'osak:*', 'naptan:*', 'CLC:*', 'import', 'it:fvg:*', + 'type', 'lacounty:*', 'ref:ruian:*', 'building:ruian:type', + 'ref:linz:*', 'is_in:postcode'}, + delete_tags = {emergency = {'yes', 'no', 'fire_hydrant'}, + historic = {'yes', 'no'}, + military = {'yes', 'no'}, + natural = {'yes', 'no', 'coastline'}, + highway = {'no', 'turning_circle', 'mini_roundabout', + 'noexit', 'crossing', 'give_way', 'stop'}, + railway = {'level_crossing', 'no', 'rail'}, + man_made = {'survey_point', 'cutline'}, + aerialway = {'pylon', 'no'}, + aeroway = {'no'}, + amenity = {'no'}, + club = {'no'}, + craft = {'no'}, + leisure = {'no'}, + office = {'no'}, + mountain_pass = {'no'}, + shop = {'no'}, + tourism = {'yes', 'no'}, + bridge = {'no'}, + tunnel = {'no'}, + waterway = {'riverbank'}, + building = {'no'}, + boundary = {'place'}}, + extra_keys = {'*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*', + 'name:etymology', 'name:signed', 'name:botanical', + 'wikidata', '*:wikidata', + 'addr:street:name', 'addr:street:type'} + } + +flex.set_name_tags{main = {'name', 'name:*', + 'int_name', 'int_name:*', + 'nat_name', 'nat_name:*', + 'reg_name', 'reg_name:*', + 'loc_name', 'loc_name:*', + 'old_name', 'old_name:*', + 'alt_name', 'alt_name:*', 'alt_name_*', + 'official_name', 'official_name:*', + 'place_name', 'place_name:*', + 'short_name', 'short_name:*', 'brand'}, + extra = {'ref', 'int_ref', 'nat_ref', 'reg_ref', + 'loc_ref', 'old_ref', + 'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'}, + house = {'addr:housename'} + } + +flex.set_address_tags{main = {'addr:housenumber', + 'addr:conscriptionnumber', + 'addr:streetnumber'}, + extra = {'addr:*', 'is_in:*', 'tiger:county'}, + postcode = {'postal_code', 'postcode', 'addr:postcode', + 'tiger:zip_left', 'tiger:zip_right'}, + country = {'country_code', 'ISO3166-1', + 'addr:country_code', 'is_in:country_code', + 'addr:country', 'is_in:country'}, + interpolation = {'addr:interpolation'} + } + + +flex.set_unused_handling{extra_keys = {'place'}} + +return flex diff --git a/settings/import-full.style b/settings/import-full.style deleted file mode 100644 index ed874a17..00000000 --- a/settings/import-full.style +++ /dev/null @@ -1,240 +0,0 @@ -[ -{ - "keys" : ["*source"], - "values" : { - "" : "skip" - } -}, -{ - "keys" : ["*:prefix", "*:suffix", "name:prefix:*", "name:suffix:*", - "name:etymology", "name:signed", "name:botanical", "wikidata", "*:wikidata", - "addr:street:name", "addr:street:type"], - "values" : { - "" : "extra" - } -}, -{ - "keys" : ["ref", "int_ref", "nat_ref", "reg_ref", "loc_ref", "old_ref", - "iata", "icao", "pcode", "pcode:*", "ISO3166-2"], - "values" : { - "" : "ref" - } -}, -{ - "keys" : ["name", "name:*", "int_name", "int_name:*", "nat_name", "nat_name:*", - "reg_name", "reg_name:*", "loc_name", "loc_name:*", - "old_name", "old_name:*", "alt_name", "alt_name:*", "alt_name_*", - "official_name", "official_name:*", "place_name", "place_name:*", - "short_name", "short_name:*", "brand"], - "values" : { - "" : "name" - } -}, -{ - "keys" : ["addr:housename"], - "values" : { - "" : "name,house" - } -}, -{ - "keys" : ["emergency"], - "values" : { - "fire_hydrant" : "skip", - "yes" : "skip", - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["historic", "military"], - "values" : { - "no" : "skip", - "yes" : "skip", - "" : "main" - } -}, -{ - "keys" : ["natural"], - "values" : { - "yes" : "skip", - "no" : "skip", - "coastline" : "skip", - "" : "main,with_name" - } -}, -{ - "keys" : ["landuse"], - "values" : { - "cemetry" : "main,with_name", - "" : "main,fallback,with_name" - } -}, -{ - "keys" : ["highway"], - "values" : { - "no" : "skip", - "turning_circle" : "skip", - "mini_roundabout" : "skip", - "noexit" : "skip", - "crossing" : "skip", - "give_way" : "skip", - "stop" : "skip", - "street_lamp" : "main,with_name", - "traffic_signals" : "main,with_name", - "service" : "main,with_name", - "cycleway" : "main,with_name", - "path" : "main,with_name", - "footway" : "main,with_name", - "steps" : "main,with_name", - "bridleway" : "main,with_name", - "track" : "main,with_name", - "byway": "main,with_name", - "motorway_link" : "main,with_name", - "trunk_link" : "main,with_name", - "primary_link" : "main,with_name", - "secondary_link" : "main,with_name", - "tertiary_link" : "main,with_name", - "" : "main" - } -}, -{ - "keys" : ["railway"], - "values" : { - "level_crossing" : "skip", - "no" : "skip", - "rail" : "skip", - "" : "main,with_name" - } -}, -{ - "keys" : ["man_made"], - "values" : { - "survey_point" : "skip", - "cutline" : "skip", - "" : "main" - } -}, -{ - "keys" : ["aerialway"], - "values" : { - "pylon" : "skip", - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["boundary"], - "values" : { - "place" : "skip", - "postal_code" : "main", - "" : "main,with_name" - } -}, -{ - "keys" : ["aeroway", "amenity", "club", "craft", "leisure", - "office", "mountain_pass"], - "values" : { - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["shop"], - "values" : { - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["tourism"], - "values" : { - "yes" : "skip", - "no" : "skip", - "" : "main" - } -}, -{ - "keys" : ["bridge", "tunnel"], - "values" : { - "" : "main,with_name_key" - } -}, -{ - "keys" : ["waterway"], - "values" : { - "riverbank" : "skip", - "" : "main,with_name" - } -}, -{ - "keys" : ["place"], - "values" : { - "" : "main" - } -}, -{ - "keys" : ["junction", "healthcare"], - "values" : { - "" : "main,fallback,with_name" - } -}, -{ - "keys" : ["postal_code", "postcode", "addr:postcode", - "tiger:zip_left", "tiger:zip_right"], - "values" : { - "" : "postcode,fallback" - } -}, -{ - "keys" : ["country_code", "ISO3166-1", "is_in:country_code", "is_in:country", - "addr:country", "addr:country_code"], - "values" : { - "" : "country" - } -}, -{ - "keys" : ["addr:housenumber", "addr:conscriptionnumber", "addr:streetnumber"], - "values" : { - "" : "address,house" - } -}, -{ - "keys" : ["addr:interpolation"], - "values" : { - "" : "interpolation,address" - } -}, -{ - "keys" : ["addr:*", "is_in:*", "tiger:county"], - "values" : { - "" : "address" - } -}, -{ - "keys" : ["building"], - "values" : { - "no" : "skip", - "" : "main,fallback,with_name" - } -}, -{ - "keys" : ["tracktype", "traffic_calming", "service", "cuisine", "capital", - "dispensing", "religion", "denomination", "sport", - "internet_access", "lanes", "surface", "smoothness", "width", - "est_width", "incline", "opening_hours", "collection_times", - "service_times", "disused", "wheelchair", "sac_scale", - "trail_visibility", "mtb:scale", "mtb:description", "wood", - "drive_through", "drive_in", "access", "vehicle", "bicyle", - "foot", "goods", "hgv", "motor_vehicle", "motor_car", "oneway", - "date_on", "date_off", "day_on", "day_off", "hour_on", "hour_off", - "maxweight", "maxheight", "maxspeed", "fee", "toll", "charge", - "population", "description", "image", "attribution", "fax", - "email", "url", "website", "phone", "real_ale", "smoking", - "food", "camera", "brewery", "locality", "wikipedia", - "wikipedia:*", "access:*", "contact:*", "drink:*", "toll:*", - "area"], - "values" : { - "" : "extra" - } -} -] diff --git a/settings/import-street.lua b/settings/import-street.lua new file mode 100644 index 00000000..acadf01e --- /dev/null +++ b/settings/import-street.lua @@ -0,0 +1,68 @@ +local flex = require('flex-base') + +flex.set_main_tags{ + highway = {'always', + street_lamp = 'named', + traffic_signals = 'named', + service = 'named', + cycleway = 'named', + path = 'named', + footway = 'named', + steps = 'named', + bridleway = 'named', + track = 'named', + motorway_link = 'named', + trunk_link = 'named', + primary_link = 'named', + secondary_link = 'named', + tertiary_link = 'named'}, + boundary = {administrative = 'named'}, + landuse = 'fallback', + place = 'always' +} + +flex.set_prefilters{delete_keys = {'building', 'source', + 'addr:housenumber', 'addr:street', + 'source', '*source', 'type', + 'is_in:postcode', '*:wikidata', + '*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*', + 'name:etymology', 'name:signed', 'name:botanical', + 'addr:street:name', 'addr:street:type'}, + delete_tags = {highway = {'no', 'turning_circle', 'mini_roundabout', + 'noexit', 'crossing', 'give_way', 'stop'}, + landuse = {'cemetry', 'no'}, + boundary = {'place'}}, + extra_keys = {'wikipedia', 'wikipedia:*', 'wikidata', 'capital', 'area'} + } + +flex.set_name_tags{main = {'name', 'name:*', + 'int_name', 'int_name:*', + 'nat_name', 'nat_name:*', + 'reg_name', 'reg_name:*', + 'loc_name', 'loc_name:*', + 'old_name', 'old_name:*', + 'alt_name', 'alt_name:*', 'alt_name_*', + 'official_name', 'official_name:*', + 'place_name', 'place_name:*', + 'short_name', 'short_name:*', 'brand'}, + extra = {'ref', 'int_ref', 'nat_ref', 'reg_ref', + 'loc_ref', 'old_ref', + 'iata', 'icao', 'pcode', 'pcode:*', 'ISO3166-2'} + } + +flex.set_address_tags{main = {'addr:housenumber', + 'addr:conscriptionnumber', + 'addr:streetnumber'}, + extra = {'addr:*', 'is_in:*', 'tiger:county'}, + postcode = {'postal_code', 'postcode', 'addr:postcode', + 'tiger:zip_left', 'tiger:zip_right'}, + country = {'country_code', 'ISO3166-1', + 'addr:country_code', 'is_in:country_code', + 'addr:country', 'is_in:country'}, + interpolation = {'addr:interpolation'}, + postcode_fallback = false + } + +flex.set_unused_handling{extra_keys = {'place'}} + +return flex diff --git a/settings/import-street.style b/settings/import-street.style deleted file mode 100644 index a1b0e8d6..00000000 --- a/settings/import-street.style +++ /dev/null @@ -1,92 +0,0 @@ -[ -{ "keys" : ["wikipedia", "wikipedia:*", "wikidata", "area"], - "values" : { - "" : "extra" - } -}, -{ - "keys" : ["*:prefix", "*:suffix", "name:prefix:*", "name:suffix:*", - "name:etymology", "name:signed", "name:botanical", "*:wikidata", - "addr:street:name", "addr:street:type"], - "values" : { - "" : "skip" - } -}, -{ - "keys" : ["ref", "int_ref", "nat_ref", "reg_ref", "loc_ref", "old_ref", - "iata", "icao", "pcode", "ISO3166-2"], - "values" : { - "" : "ref" - } -}, -{ - "keys" : ["name", "name:*", "int_name", "int_name:*", "nat_name", "nat_name:*", - "reg_name", "reg_name:*", "loc_name", "loc_name:*", - "old_name", "old_name:*", "alt_name", "alt_name:*", "alt_name_*", - "official_name", "official_name:*", "place_name", "place_name:*", - "short_name", "short_name:*", "brand"], - "values" : { - "" : "name" - } -}, -{ - "keys" : ["landuse"], - "values" : { - "cemetry" : "skip", - "" : "fallback,with_name" - } -}, -{ - "keys" : ["boundary"], - "values" : { - "administrative" : "main" - } -}, -{ - "keys" : ["place"], - "values" : { - "" : "main" - } -}, -{ - "keys" : ["country_code", "ISO3166-1", "is_in:country_code", "is_in:country", - "addr:country", "addr:country_code"], - "values" : { - "" : "country" - } -}, -{ - "keys" : ["addr:*", "is_in:*", "tiger:county"], - "values" : { - "" : "address" - } -}, -{ - "keys" : ["highway"], - "values" : { - "motorway" : "main", - "trunk" : "main", - "primary" : "main", - "secondary" : "main", - "tertiary" : "main", - "unclassified" : "main", - "residential" : "main", - "living_street" : "main", - "pedestrian" : "main", - "road" : "main", - "service" : "main,with_name", - "cycleway" : "main,with_name", - "path" : "main,with_name", - "footway" : "main,with_name", - "steps" : "main,with_name", - "bridleway" : "main,with_name", - "track" : "main,with_name", - "byway": "main,with_name", - "motorway_link" : "main,with_name", - "trunk_link" : "main,with_name", - "primary_link" : "main,with_name", - "secondary_link" : "main,with_name", - "tertiary_link" : "main,with_name" - } -} -] diff --git a/test/bdd/osm2pgsql/import/custom_style.feature b/test/bdd/osm2pgsql/import/custom_style.feature new file mode 100644 index 00000000..2ca95c91 --- /dev/null +++ b/test/bdd/osm2pgsql/import/custom_style.feature @@ -0,0 +1,200 @@ +@DB +Feature: Import with custom styles by osm2pgsql + Tests for the example customizations given in the documentation. + + Scenario: Custom main tags + Given the lua style file + """ + local flex = require('import-full') + + flex.set_main_tags{ + boundary = {administrative = 'named'}, + highway = {'always', street_lamp = 'named'}, + landuse = 'fallback' + } + """ + When loading osm data + """ + n10 Tboundary=administrative x0 y0 + n11 Tboundary=administrative,name=Foo x0 y0 + n12 Tboundary=electoral x0 y0 + n13 Thighway=primary x0 y0 + n14 Thighway=street_lamp x0 y0 + n15 Thighway=primary,landuse=street x0 y0 + """ + Then place contains exactly + | object | class | type | + | N11 | boundary | administrative | + | N13 | highway | primary | + | N15 | highway | primary | + + Scenario: Prefiltering tags + Given the lua style file + """ + local flex = require('import-full') + + flex.set_prefilters{ + delete_keys = {'source', 'source:*'}, + extra_tags = {amenity = {'yes', 'no'}} + } + flex.set_main_tags{ + amenity = 'always', + tourism = 'always' + } + """ + When loading osm data + """ + n1 Tamenity=yes x0 y6 + n2 Tamenity=hospital,source=survey x3 y6 + n3 Ttourism=hotel,amenity=yes x0 y0 + n4 Ttourism=hotel,amenity=telephone x0 y0 + """ + Then place contains exactly + | object | extratags | + | N2:amenity | - | + | N3:tourism | 'amenity': 'yes' | + | N4:tourism | - | + | N4:amenity | - | + + Scenario: Name tags + Given the lua style file + """ + local flex = require('flex-base') + + flex.set_main_tags{highway = {traffic_light = 'named'}} + flex.set_name_tags{main = {'name', 'name:*'}, + extra = {'ref'} + } + """ + When loading osm data + """ + n1 Thighway=stop,name=Something x0 y0 + n2 Thighway=traffic_light,ref=453-4 x0 y0 + n3 Thighway=traffic_light,name=Greens x0 y0 + n4 Thighway=traffic_light,name=Red,ref=45 x0 y0 + """ + Then place contains exactly + | object | name | + | N3:highway | 'name': 'Greens' | + | N4:highway | 'name': 'Red', 'ref': '45' | + + Scenario: Address tags + Given the lua style file + """ + local flex = require('import-full') + + flex.set_address_tags{ + main = {'addr:housenumber'}, + extra = {'addr:*'}, + postcode = {'postal_code', 'postcode', 'addr:postcode'}, + country = {'country-code', 'ISO3166-1'} + } + """ + When loading osm data + """ + n1 Ttourism=hotel,addr:street=Foo x0 y0 + n2 Taddr:housenumber=23,addr:street=Budd,postal_code=5567 x0 y0 + n3 Taddr:street=None,addr:city=Where x0 y0 + """ + Then place contains exactly + | object | type | address | + | N1:tourism | hotel | 'street': 'Foo' | + | N2:place | house | 'housenumber': '23', 'street': 'Budd', 'postcode': '5567' | + + Scenario: Unused handling + Given the lua style file + """ + local flex = require('import-full') + + flex.set_address_tags{ + main = {'addr:housenumber'}, + extra = {'addr:*', 'tiger:county'} + } + flex.set_unused_handling{delete_keys = {'tiger:*'}} + """ + When loading osm data + """ + n1 Ttourism=hotel,tiger:county=Fargo x0 y0 + n2 Ttourism=hotel,tiger:xxd=56,else=other x0 y0 + """ + Then place contains exactly + | object | type | address | extratags | + | N1:tourism | hotel | 'tiger:county': 'Fargo' | - | + | N2:tourism | hotel | - | 'else': 'other' | + + Scenario: Additional relation types + Given the lua style file + """ + local flex = require('import-full') + + flex.RELATION_TYPES['site'] = flex.relation_as_multipolygon + """ + And the grid + | 1 | 2 | + | 4 | 3 | + When loading osm data + """ + n1 + n2 + n3 + n4 + w1 Nn1,n2,n3,n4,n1 + r1 Ttype=multipolygon,amenity=school Mw1@ + r2 Ttype=site,amenity=school Mw1@ + """ + Then place contains exactly + | object | type | + | R1:amenity | school | + | R2:amenity | school | + + Scenario: Exclude country relations + Given the lua style file + """ + local flex = require('import-full') + + function osm2pgsql.process_relation(object) + if object.tags.boundary ~= 'administrative' or object.tags.admin_level ~= '2' then + flex.process_relation(object) + end + end + """ + And the grid + | 1 | 2 | + | 4 | 3 | + When loading osm data + """ + n1 + n2 + n3 + n4 + w1 Nn1,n2,n3,n4,n1 + r1 Ttype=multipolygon,boundary=administrative,admin_level=4,name=Small Mw1@ + r2 Ttype=multipolygon,boundary=administrative,admin_level=2,name=Big Mw1@ + """ + Then place contains exactly + | object | type | + | R1:boundary | administrative | + + Scenario: Customize processing functions + Given the lua style file + """ + local flex = require('import-full') + + local original_process_tags = flex.process_tags + + function flex.process_tags(o) + if o.object.tags.highway ~= nil and o.object.tags.access == 'no' then + return + end + + original_process_tags(o) + end + """ + When loading osm data + """ + n1 Thighway=residential x0 y0 + n2 Thighway=residential,access=no x0 y0 + """ + Then place contains exactly + | object | type | + | N1:highway | residential | diff --git a/test/bdd/osm2pgsql/import/tags.feature b/test/bdd/osm2pgsql/import/tags.feature index 60d241fe..c2ab736a 100644 --- a/test/bdd/osm2pgsql/import/tags.feature +++ b/test/bdd/osm2pgsql/import/tags.feature @@ -122,10 +122,10 @@ Feature: Tag evaluation n8003 Tshop=shoes,name:source=survey """ Then place contains exactly - | object | class | extratags | - | N8001 | shop | 'xx': 'yy' | - | N8002 | shop | 'ele': '234' | - | N8003 | shop | - | + | object | class | name | extratags | + | N8001 | shop | - | 'xx': 'yy' | + | N8002 | shop | - | 'ele': '234' | + | N8003 | shop | - | - | Scenario: Admin levels diff --git a/test/bdd/steps/nominatim_environment.py b/test/bdd/steps/nominatim_environment.py index 238081c0..6179ca34 100644 --- a/test/bdd/steps/nominatim_environment.py +++ b/test/bdd/steps/nominatim_environment.py @@ -89,11 +89,6 @@ class NominatimEnvironment: if self.db_pass: dsn += ';password=' + self.db_pass - if self.website_dir is not None \ - and self.test_env is not None \ - and dsn == self.test_env['NOMINATIM_DATABASE_DSN']: - return # environment already set uo - self.test_env = dict(self.default_config) self.test_env['NOMINATIM_DATABASE_DSN'] = dsn self.test_env['NOMINATIM_LANGUAGES'] = 'en,de,fr,ja' diff --git a/test/bdd/steps/steps_osm_data.py b/test/bdd/steps/steps_osm_data.py index 7590b17c..336fb707 100644 --- a/test/bdd/steps/steps_osm_data.py +++ b/test/bdd/steps/steps_osm_data.py @@ -49,6 +49,15 @@ def write_opl_file(opl, grid): return fd.name +@given('the lua style file') +def lua_style_file(context): + """ Define a custom style file to use for the import. + """ + style = Path(context.nominatim.website_dir.name) / 'custom.lua' + style.write_text(context.text) + context.nominatim.test_env['NOMINATIM_IMPORT_STYLE'] = str(style) + + @given(u'the ([0-9.]+ )?grid(?: with origin (?P.*))?') def define_node_grid(context, grid_step, origin): """ diff --git a/test/python/config/test_config.py b/test/python/config/test_config.py index a003065d..ff8b587d 100644 --- a/test/python/config/test_config.py +++ b/test/python/config/test_config.py @@ -222,7 +222,7 @@ def test_get_import_style_intern(make_config, src_dir, monkeypatch): monkeypatch.setenv('NOMINATIM_IMPORT_STYLE', 'street') - expected = src_dir / 'settings' / 'import-street.style' + expected = src_dir / 'settings' / 'import-street.lua' assert config.get_import_style_file() == expected diff --git a/vagrant/Install-on-Ubuntu-18.sh b/vagrant/Install-on-Ubuntu-18.sh index 621e4473..09de9747 100755 --- a/vagrant/Install-on-Ubuntu-18.sh +++ b/vagrant/Install-on-Ubuntu-18.sh @@ -115,7 +115,7 @@ fi #DOCS: # download the country grid: if [ ! -f data/country_osm_grid.sql.gz ]; then #DOCS: :::sh - wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz + wget -O data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz fi #DOCS: # The code must be built in a separate directory. Create this directory, diff --git a/vagrant/Install-on-Ubuntu-20.sh b/vagrant/Install-on-Ubuntu-20.sh index 87aad5ef..2d4eaa71 100755 --- a/vagrant/Install-on-Ubuntu-20.sh +++ b/vagrant/Install-on-Ubuntu-20.sh @@ -109,7 +109,7 @@ fi #DOCS: # download the country grid: if [ ! -f data/country_osm_grid.sql.gz ]; then #DOCS: :::sh - wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz + wget -O data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz fi #DOCS: # The code must be built in a separate directory. Create this directory, diff --git a/vagrant/Install-on-Ubuntu-22.sh b/vagrant/Install-on-Ubuntu-22.sh index 83cd5350..c45ac55e 100755 --- a/vagrant/Install-on-Ubuntu-22.sh +++ b/vagrant/Install-on-Ubuntu-22.sh @@ -109,7 +109,7 @@ fi #DOCS: # download the country grid: if [ ! -f data/country_osm_grid.sql.gz ]; then #DOCS: :::sh - wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz + wget -O data/country_osm_grid.sql.gz https://nominatim.org/data/country_grid.sql.gz fi #DOCS: # The code must be built in a separate directory. Create this directory,