# 'too-many-ancestors' is triggered already by deriving from UserDict
# 'not-context-manager' disabled because it causes false positives once
# typed Python is enabled. See also https://github.com/PyCQA/pylint/issues/5273
-disable=too-few-public-methods,duplicate-code,too-many-ancestors,bad-option-value,no-self-use,not-context-manager,use-dict-literal,chained-comparison,attribute-defined-outside-init,too-many-boolean-expressions
+disable=too-few-public-methods,duplicate-code,too-many-ancestors,bad-option-value,no-self-use,not-context-manager,use-dict-literal,chained-comparison,attribute-defined-outside-init,too-many-boolean-expressions,contextmanager-generator-missing-cleanup
good-names=i,j,x,y,m,t,fd,db,cc,x1,x2,y1,y2,pt,k,v,nr
cd $PROJECT_DIR
wget https://nominatim.org/data/wikimedia-importance.sql.gz
+ wget -O secondary_importance.sql.gz https://nominatim.org/data/wikimedia-secondary-importance.sql.gz
-The file is about 400MB and adds around 4GB to the Nominatim database.
+The files are about 400MB and add around 4GB to the Nominatim database. For
+more information about importance,
+see [Importance Customization](../customize/Importance.md).
!!! tip
If you forgot to download the wikipedia rankings, then you can
also add importances after the import. Download the SQL files, then
- run `nominatim refresh --wiki-data --importance`. Updating
- importances for a planet will take a couple of hours.
+ run `nominatim refresh --wiki-data --secondary-importance --importance`.
+ Updating importances for a planet will take a couple of hours.
### External postcodes
The main value for importance is derived from page ranking values for Wikipedia
pages for a place. For places that do not have their own
Wikipedia page, a formula is used that derives a static importance from the
-places [search rank](../customize/Ranking.md#search-rank).
+place's [search rank](../customize/Ranking.md#search-rank).
In a second step, a secondary importance value is added which is meant to
represent how well-known the general area is where the place is located. It
nominatim.org has preprocessed importance tables for the
[primary Wikipedia rankings](https://nominatim.org/data/wikimedia-importance.sql.gz)
-and for a secondary importance based on the number of tile views on openstreetmap.org.
+and for [secondary importance](https://nominatim.org/data/wikimedia-secondary-importance.sql.gz)
+based on Wikipedia importance of the administrative areas.
+
+The source code for creating these files is avaible in the Github projects
+[osm-search/wikipedia-wikidata](https://github.com/osm-search/wikipedia-wikidata)
+and
+[osm-search/secondary-importance](https://github.com/osm-search/secondary-importance).
### Customizing secondary importance
CREATE INDEX ON secondary_importance USING gist(ST_ConvexHull(gist))
```
-The following raster2pgsql command will create a table that conforms to
-the requirements:
+The following raster2pgsql command will create a table from a tiff file
+that conforms to the requirements:
```
raster2pgsql -I -C -Y -d -t 128x128 input.tiff public.secondary_importance
## Importing custom special phrases
-But, it is also possible to import some phrases from a csv file.
-To do so, you have access to the following command:
+Special phrases may also be imported from any custom CSV file. The file needs
+to have a header line, use comma as delimiter and define the following
+columns:
+
+ * **phrase**: the keyword to look for
+ * **class**: key of the main tag of the place to find
+ (see [principal tags in import style](../Import-Styles.md#set_main_tags-principal-tags)
+ * **type**: value of the main tag
+ * **operator**: type of special phrase, may be one of:
+ * *in*: place is within the place defined by the search term (e.g. "_Hotels in_ Berlin")
+ * *near*: place is near the place defined by the search term (e.g. "_bus stops near_ Big Ben")
+ * *named*: special phrase is a classifier (e.g. "_hotel_ California")
+ * *-*: unspecified, can be any of the above
+
+If the file contains any other columns, then they are silently ignored
+
+To import the CSV file, use the following command:
```sh
nominatim special-phrases --import-from-csv <csv file>
```
Note that the two previous import commands will update the phrases from your database.
-This means that if you import some phrases from a csv file, only the phrases
-present in the csv file will be kept into the database. All other phrases will
+This means that if you import some phrases from a CSV file, only the phrases
+present in the CSV file will be kept in the database. All other phrases will
be removed.
If you want to only add new phrases and not update the other ones you can add
nominatim special-phrases --import-from-csv <csv file> --no-replace
```
-This will add the phrases present in the csv file into the database without
+This will add the phrases present in the CSV file into the database without
removing the other ones.
Any configuration found in the `.env` file in this directory will automatically
used.
-Yo may also configure Nominatim be setting environment variables.
+You may also configure Nominatim by setting environment variables.
Normally, Nominatim will check the operating system environment. This can be
overwritten by giving the constructor a dictionary of configuration parameters.
END LOOP;
END IF;
- IF parent_place_id is null THEN
- RETURN 0;
- END IF;
-
RETURN parent_place_id;
END;
$$
ST_PointOnSurface(NEW.linegeo),
NEW.linegeo);
+ -- Cannot find a parent street. We will not be able to display a reliable
+ -- address, so drop entire interpolation.
+ IF NEW.parent_place_id is NULL THEN
+ DELETE FROM location_property_osmline where place_id = OLD.place_id;
+ RETURN NULL;
+ END IF;
+
NEW.token_info := token_strip_info(NEW.token_info);
IF NEW.address ? '_inherited' THEN
NEW.address := hstore('interpolation', NEW.address->'interpolation');
{% if debug %}RAISE WARNING 'placex_delete:12 % %',OLD.osm_type,OLD.osm_id;{% endif %}
+ UPDATE location_postcode SET indexed_status = 2 WHERE parent_place_id = OLD.place_id;
+
RETURN OLD;
END;
type TEXT,
deferred BOOLEAN
);
+---
+ CREATE INDEX IF NOT EXISTS idx_location_postcode_parent_place_id
+ ON location_postcode USING BTREE (parent_place_id) {{db.tablespace.address_index}};
{% endif %}
-- Indices only needed for search.
indexer.index_boundaries(args.minrank, args.maxrank)
if not args.boundaries_only:
indexer.index_by_rank(args.minrank, args.maxrank)
+ indexer.index_postcodes()
if not args.no_boundaries and not args.boundaries_only \
and args.minrank == 0 and args.maxrank == 30:
if not args.do_index:
LOG.fatal("Indexing cannot be disabled when running updates continuously.")
raise UsageError("Bad argument '--no-index'.")
- recheck_interval = args.config.get_int('REPLICATION_RECHECK_INTERVAL')
+ recheck_interval = args.config.get_int('REPLICATION_RECHECK_INTERVAL')
tokenizer = tokenizer_factory.get_tokenizer_for_db(args.config)
indexer = Indexer(args.config.get_libpq_dsn(), tokenizer, args.threads or 1)
""")
@_migration(4, 4, 99, 0)
-def create_postcode_ara_lookup_index(conn: Connection, **_: Any) -> None:
+def create_postcode_area_lookup_index(conn: Connection, **_: Any) -> None:
""" Create index needed for looking up postcode areas from postocde points.
"""
with conn.cursor() as cur:
ON placex USING BTREE (country_code, postcode)
WHERE osm_type = 'R' AND class = 'boundary' AND type = 'postal_code'
""")
+
+
+@_migration(4, 4, 99, 1)
+def create_postcode_parent_index(conn: Connection, **_: Any) -> None:
+ """ Create index needed for updating postcodes when a parent changes.
+ """
+ if conn.table_exists('planet_osm_ways'):
+ with conn.cursor() as cur:
+ cur.execute("""CREATE INDEX IF NOT EXISTS
+ idx_location_postcode_parent_place_id
+ ON location_postcode USING BTREE (parent_place_id)""")
return f"{self.major}.{self.minor}.{self.patch_level}-{self.db_patch_level}"
-NOMINATIM_VERSION = NominatimVersion(4, 4, 99, 0)
+NOMINATIM_VERSION = NominatimVersion(4, 4, 99, 1)
POSTGRESQL_REQUIRED_VERSION = (9, 6)
POSTGIS_REQUIRED_VERSION = (2, 2)
local flex = require('flex-base')
flex.set_main_tags{
- highway = {'always',
- street_lamp = 'named',
- traffic_signals = 'named',
+ highway = {motorway = 'always',
+ trunk = 'always',
+ primary = 'always',
+ secondary = 'always',
+ tertiary = 'always',
+ unclassified = 'always',
+ residential = 'always',
+ road = 'always',
+ living_street = 'always',
+ pedestrian = 'always',
service = 'named',
cycleway = 'named',
path = 'named',
'*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*',
'name:etymology', 'name:signed', 'name:botanical',
'addr:street:name', 'addr:street:type'},
- delete_tags = {highway = {'no', 'turning_circle', 'mini_roundabout',
- 'noexit', 'crossing', 'give_way', 'stop'},
- landuse = {'cemetry', 'no'},
+ delete_tags = {landuse = {'cemetry', 'no'},
boundary = {'place'}},
extra_keys = {'wikipedia', 'wikipedia:*', 'wikidata', 'capital', 'area'}
}
local flex = require('flex-base')
flex.set_main_tags{
- highway = {'always',
- street_lamp = 'named',
- traffic_signals = 'named',
+ highway = {motorway = 'always',
+ trunk = 'always',
+ primary = 'always',
+ secondary = 'always',
+ tertiary = 'always',
+ unclassified = 'always',
+ residential = 'always',
+ road = 'always',
+ living_street = 'always',
+ pedestrian = 'always',
service = 'named',
cycleway = 'named',
path = 'named',
'*:prefix', '*:suffix', 'name:prefix:*', 'name:suffix:*',
'name:etymology', 'name:signed', 'name:botanical',
'addr:street:name', 'addr:street:type'},
- delete_tags = {highway = {'no', 'turning_circle', 'mini_roundabout',
- 'noexit', 'crossing', 'give_way', 'stop'},
- landuse = {'cemetry', 'no'},
+ delete_tags = {landuse = {'cemetry', 'no'},
boundary = {'place'}},
extra_keys = {'wikipedia', 'wikipedia:*', 'wikidata', 'capital', 'area'}
}
Feature: Import of address interpolations
Tests that interpolated addresses are added correctly
+ Scenario: Simple even interpolation line with two points and no street nearby
+ Given the grid with origin 1,1
+ | 1 | | 9 | | 2 |
+ Given the places
+ | osm | class | type | housenr |
+ | N1 | place | house | 2 |
+ | N2 | place | house | 6 |
+ And the places
+ | osm | class | type | addr+interpolation | geometry |
+ | W1 | place | houses | even | 1,2 |
+ And the ways
+ | id | nodes |
+ | 1 | 1,2 |
+ When importing
+ Then W1 expands to no interpolation
+
Scenario: Simple even interpolation line with two points
Given the grid with origin 1,1
| 1 | | 9 | | 2 |
+ | 4 | | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,2 |
Scenario: Backwards even two point interpolation line
Given the grid with origin 1,1
| 1 | 8 | 9 | 2 |
+ | 4 | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 2,1 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 2,1 |
Scenario: Simple odd two point interpolation
Given the grid with origin 1,1
| 1 | 8 | | | 9 | 2 |
+ | 4 | | | | 5 | |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 1 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | odd | 1,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,2 |
Scenario: Simple all two point interpolation
Given the grid with origin 1,1
| 1 | 8 | 9 | 2 |
+ | 4 | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 1 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | all | 1,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,2 |
Scenario: Even two point interpolation line with intermediate empty node
Given the grid
| 1 | 8 | | 3 | 9 | 2 |
+ | 4 | | | | 5 | |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,3,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,3,2 |
Scenario: Even two point interpolation line with intermediate duplicated empty node
Given the grid
+ | 4 | | | | 5 |
| 1 | 8 | 3 | 9 | 2 |
Given the places
| osm | class | type | housenr |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,3,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,3,3,2 |
Scenario: Simple even three point interpolation line
Given the grid
+ | 4 | | | | | | 5 |
| 1 | 8 | | 9 | 3 | 7 | 2 |
Given the places
| osm | class | type | housenr |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,3,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,3,2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,3,2,4 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 1,3,2,4 |
And the ways
| id | nodes |
| 1 | 1,3,2,4 |
Scenario: Reverse simple even three point interpolation line
Given the grid
| 1 | 8 | | 9 | 3 | 7 | 2 |
+ | 4 | | | | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 2,3,1 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 2,3,1 |
Scenario: Even three point interpolation line with odd center point
Given the grid
| 1 | | 10 | | | 11 | 3 | 2 |
+ | 4 | | | | | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,3,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,3,2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,2,3,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 1,2,3 |
And the ways
| id | nodes |
| 1 | 1,2,3,2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,2,3,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 1,2,3 |
And the ways
| id | nodes |
| 1 | 1,2,3,2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 144.9632341 -37.76163,144.9630541 -37.7628172,144.9629794 -37.7630755 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 144.9632341 -37.76163,144.9629794 -37.7630755 |
And the ways
| id | nodes |
| 1 | 1,2,3 |
Scenario: Place with missing address information
Given the grid
| 1 | | 2 | | | 3 |
+ | 4 | | | | | 5 |
And the places
| osm | class | type | housenr |
| N1 | place | house | 23 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | odd | 1,2,3 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,2,3 |
Given the places
| osm | class | type | housenr | geometry |
| W1 | place | houses | even | 1 1, 1 1.001 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 1 1, 1 1.001 |
When importing
Then W1 expands to no interpolation
Scenario: Ways with nodes without housenumbers are ignored
Given the grid
- | 1 | | 2 |
+ | 1 | | 2 |
+ | 4 | | 5 |
Given the places
| osm | class | type |
| N1 | place | house |
Given the places
| osm | class | type | housenr | geometry |
| W1 | place | houses | even | 1,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
When importing
Then W1 expands to no interpolation
Scenario: Two point interpolation starting at 0
Given the grid with origin 1,1
| 1 | 10 | | | 11 | 2 |
+ | 4 | | | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 0 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,2 |
+ And the places
+ | osm | class | type | name | geometry |
+ | W10 | highway | residential | London Road |4,5 |
And the ways
| id | nodes |
| 1 | 1,2 |
When sending v1/reverse at 1,1
Then results contain
| ID | osm | type | display_name |
- | 0 | N1 | house | 0 |
+ | 0 | N1 | house | 0, London Road |
Scenario: Parenting of interpolation with additional tags
Given the grid
Scenario Outline: Bad interpolation values are ignored
Given the grid with origin 1,1
| 1 | | 9 | | 2 |
+ | 4 | | | | 5 |
Given the places
| osm | class | type | housenr |
| N1 | place | house | 2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | <value> | 1,2 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,2 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 1,2,3,4 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 1,4 |
And the ways
| id | nodes |
| 1 | 1,2,3,4 |
Scenario: Interpolation line with duplicated points
Given the grid
| 7 | 10 | 8 | 11 | 9 |
+ | 4 | | | | 5 |
Given the places
| osm | class | type | housenr | geometry |
| N1 | place | house | 2 | 7 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 7,8,8,9 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 4,5 |
And the ways
| id | nodes |
| 1 | 1,2,3,4 |
And the places
| osm | class | type | addr+interpolation | geometry |
| W1 | place | houses | even | 8,9 |
+ And the named places
+ | osm | class | type | geometry |
+ | W10 | highway | residential | 1,4 |
And the ways
| id | nodes |
| 1 | 1,8,9,2,3,4 |
| country | postcode | geometry |
| de | 01982 | country:de |
And there are word tokens for postcodes 01982
+
+ Scenario: When a parent is deleted, the postcode gets a new parent
+ Given the grid with origin DE
+ | 1 | | 3 | 4 |
+ | | 9 | | |
+ | 2 | | 5 | 6 |
+ Given the places
+ | osm | class | type | name | admin | geometry |
+ | R1 | boundary | administrative | Big | 6 | (1,4,6,2,1) |
+ | R2 | boundary | administrative | Small | 6 | (1,3,5,2,1) |
+ Given the named places
+ | osm | class | type | addr+postcode | geometry |
+ | N9 | place | postcode | 12345 | 9 |
+ When importing
+ And updating postcodes
+ Then location_postcode contains exactly
+ | country | postcode | geometry | parent_place_id |
+ | de | 12345 | 9 | R2 |
+ When marking for delete R2
+ Then location_postcode contains exactly
+ | country | postcode | geometry | parent_place_id |
+ | de | 12345 | 9 | R1 |
"""
n1 Taddr:housenumber=3
n2 Taddr:housenumber=17
+ w33 Thighway=residential,name=Tao Nn1,n2
"""
Then place contains
| object | type |
| object | type |
| N1:place | house |
| N2:place | house |
+ | W33:highway | residential |
Then location_property_osmline contains exactly
| object |
| 99:5 |
"""
n1 Taddr:housenumber=3
n2 Taddr:housenumber=17
+ w33 Thighway=residential Nn1,n2
w99 Thighway=residential Nn1,n2
"""
Then place contains
| object | type |
| N1:place | house |
| N2:place | house |
+ | W33:highway | residential |
And location_property_osmline contains exactly
| object |
| 99:5 |
"""
n1 Taddr:housenumber=3
n2 Taddr:housenumber=17
+ w33 Thighway=residential Nn1,n2
w34 Tpostcode=4456 Nn1,n2
"""
Then place contains exactly
| object | type |
| N1:place | house |
| N2:place | house |
+ | W33:highway | residential |
| W34:place | postcode |
When updating osm data
| object | type |
| N1:place | house |
| N2:place | house |
+ | W33:highway | residential |
| W34:place | houses |
When indexing
Then location_property_osmline contains exactly
| object | type |
| N1:place | house |
| N2:place | house |
+ | W33:highway | residential |
table_factory('import_status', 'indexed bool')
bnd_mock = mock_func_factory(nominatim.indexer.indexer.Indexer, 'index_boundaries')
rank_mock = mock_func_factory(nominatim.indexer.indexer.Indexer, 'index_by_rank')
+ postcode_mock = mock_func_factory(nominatim.indexer.indexer.Indexer, 'index_postcodes')
assert self.call_nominatim('index', *params) == 0
assert bnd_mock.called == do_bnds
assert rank_mock.called == do_ranks
+ assert postcode_mock.called == do_ranks
def test_special_phrases_wiki_command(self, mock_func_factory):
projects / nominatim.git / commitdiff