]> git.openstreetmap.org Git - nominatim.git/log
nominatim.git
3 years agoremove some odd varaints of addr:street from the styles
Sarah Hoffmann [Mon, 6 Dec 2021 14:17:00 +0000 (15:17 +0100)]
remove some odd varaints of addr:street from the styles

Some import has added names in partial tags which confuse the
street name matching.

3 years agoskip most addr: tags with suffixes
Sarah Hoffmann [Mon, 6 Dec 2021 13:46:40 +0000 (14:46 +0100)]
skip most addr: tags with suffixes

Only one addr: tag can be processed currently, so make
sure it is the one without suffixes to not get odd data.
addr:street is the exception because it uses a different
matching mechanism.

3 years agoICU: matching any street name will do again
Sarah Hoffmann [Mon, 6 Dec 2021 13:26:08 +0000 (14:26 +0100)]
ICU: matching any street name will do again

3 years agorevert to using full names for street name matching
Sarah Hoffmann [Mon, 6 Dec 2021 10:38:38 +0000 (11:38 +0100)]
revert to using full names for street name matching

Using partial names turned out to not work well because there are
often similarly named streets next to each other. It also
prevents us from being able to take into account all addr:street:*
tags.

This change gets all the full term tokens for the addr:street tags
from the DB. As they are used for matching only, we can assume that
the term must already be there or there will be no match. This
avoid creating unused full name tags.

3 years agoMerge pull request #2539 from lonvia/clean-up-python-tests
Sarah Hoffmann [Fri, 3 Dec 2021 16:08:25 +0000 (17:08 +0100)]
Merge pull request #2539 from lonvia/clean-up-python-tests

Restructure and extend python unit tests

3 years agospecify text type in test SQL
Sarah Hoffmann [Fri, 3 Dec 2021 11:01:53 +0000 (12:01 +0100)]
specify text type in test SQL

Older version of postgres fail otherwise.

3 years agosplit cli tests by subcommand and extend coverage
Sarah Hoffmann [Thu, 2 Dec 2021 22:45:48 +0000 (23:45 +0100)]
split cli tests by subcommand and extend coverage

3 years agoremove unnecessary pass statements
Sarah Hoffmann [Thu, 2 Dec 2021 14:54:24 +0000 (15:54 +0100)]
remove unnecessary pass statements

3 years agomore unit tests for tokenizers
Sarah Hoffmann [Thu, 2 Dec 2021 14:46:36 +0000 (15:46 +0100)]
more unit tests for tokenizers

3 years agoextend API unit tests
Sarah Hoffmann [Wed, 1 Dec 2021 19:48:29 +0000 (20:48 +0100)]
extend API unit tests

3 years agoadd tests for migration
Sarah Hoffmann [Wed, 1 Dec 2021 19:27:40 +0000 (20:27 +0100)]
add tests for migration

3 years agomore testing for refresh functions
Sarah Hoffmann [Wed, 1 Dec 2021 13:58:54 +0000 (14:58 +0100)]
more testing for refresh functions

3 years agomore tests for exec utilities
Sarah Hoffmann [Wed, 1 Dec 2021 13:23:51 +0000 (14:23 +0100)]
more tests for exec utilities

3 years agoadd more tests for database import
Sarah Hoffmann [Wed, 1 Dec 2021 10:54:58 +0000 (11:54 +0100)]
add more tests for database import

3 years agoadd tests for adding additional data
Sarah Hoffmann [Wed, 1 Dec 2021 10:22:46 +0000 (11:22 +0100)]
add tests for adding additional data

Also adds checks that parameters for osm2pgsql are set
as expected.

3 years agoadd tests for flatten_config_file and other than yaml formats
Sarah Hoffmann [Wed, 1 Dec 2021 09:24:11 +0000 (10:24 +0100)]
add tests for flatten_config_file and other than yaml formats

3 years agotests: add fixture for making test project directory
Sarah Hoffmann [Tue, 30 Nov 2021 17:01:46 +0000 (18:01 +0100)]
tests: add fixture for making test project directory

3 years agogeneralize fixtures for cli tests
Sarah Hoffmann [Tue, 30 Nov 2021 13:07:39 +0000 (14:07 +0100)]
generalize fixtures for cli tests

3 years agopython test: move single-use fixtures to subdirectories
Sarah Hoffmann [Tue, 30 Nov 2021 11:03:16 +0000 (12:03 +0100)]
python test: move single-use fixtures to subdirectories

3 years agoremove unused test files
Sarah Hoffmann [Tue, 30 Nov 2021 10:23:00 +0000 (11:23 +0100)]
remove unused test files

3 years agoorganise python tests in subdirectories
Sarah Hoffmann [Tue, 30 Nov 2021 10:10:47 +0000 (11:10 +0100)]
organise python tests in subdirectories

The directories follow the same structure as the modules in
nominatim/.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Thu, 25 Nov 2021 07:41:57 +0000 (08:41 +0100)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2530 from lonvia/declassify-highway
Sarah Hoffmann [Thu, 25 Nov 2021 07:41:25 +0000 (08:41 +0100)]
Merge pull request #2530 from lonvia/declassify-highway

Change default rank for highway objects to 30

3 years agochange default rank for highway objects to 30
Sarah Hoffmann [Wed, 24 Nov 2021 13:40:23 +0000 (14:40 +0100)]
change default rank for highway objects to 30

The highway key is being used more and more for non-ways these
days. This clashes with Nominatim's assumption that essentially
everything that has a highway tag can be used as the street part
of the address.

Change the default rank of highway objects to 30 to avoid this.
Only the known values for streets keep the rank 26 and are now
listed explicitly.

3 years agoMerge pull request #2529 from lonvia/sort-street-results-by-tiger-housenumber
Sarah Hoffmann [Wed, 24 Nov 2021 15:23:41 +0000 (16:23 +0100)]
Merge pull request #2529 from lonvia/sort-street-results-by-tiger-housenumber

Take tiger housenumber into account when ranking street results

3 years agoadd migration for inclusive housenumber Tiger index
Sarah Hoffmann [Wed, 24 Nov 2021 10:05:04 +0000 (11:05 +0100)]
add migration for inclusive housenumber Tiger index

3 years agoadd index for Tiger housenumber queries
Sarah Hoffmann [Tue, 23 Nov 2021 19:24:08 +0000 (20:24 +0100)]
add index for Tiger housenumber queries

3 years agotake Tiger housenumbers into account when ranking street results
Sarah Hoffmann [Tue, 23 Nov 2021 19:04:50 +0000 (20:04 +0100)]
take Tiger housenumbers into account when ranking street results

Queries with a housenumber need to rank streets higher that
have the requested housenumber attached. We already do that for
ordinary housenumber objects and for interpolations. This
adds support for Tiger housenumbers as well.

Fixes #2501.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Sun, 21 Nov 2021 09:53:42 +0000 (10:53 +0100)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2528 from lonvia/allow-french-extra-housenumbers
Sarah Hoffmann [Sun, 21 Nov 2021 09:53:20 +0000 (10:53 +0100)]
Merge pull request #2528 from lonvia/allow-french-extra-housenumbers

Don't penalize French 'bis' housenumbers

3 years agoMerge pull request #2526 from lonvia/docs-moving-database
Sarah Hoffmann [Fri, 19 Nov 2021 20:14:53 +0000 (21:14 +0100)]
Merge pull request #2526 from lonvia/docs-moving-database

Add a section about moving the database to another machine

3 years agodon't penalize French 'bis' housenumbers
Sarah Hoffmann [Fri, 19 Nov 2021 20:12:17 +0000 (21:12 +0100)]
don't penalize French 'bis' housenumbers

House numbers of the form '9 bis' are usual in France. So
be a bit more lenient before adding penalties to house numbers
with letters in them.

Fixes #2527.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Fri, 19 Nov 2021 15:16:53 +0000 (16:16 +0100)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2525 from lonvia/fix-replication-indexer
Sarah Hoffmann [Fri, 19 Nov 2021 15:16:30 +0000 (16:16 +0100)]
Merge pull request #2525 from lonvia/fix-replication-indexer

Fix instantiation of indexer for replication

3 years agoadd a section about moving the database to another machine
Sarah Hoffmann [Fri, 19 Nov 2021 15:11:32 +0000 (16:11 +0100)]
add a section about moving the database to another machine

3 years agoonly instantiate indexer once for replication
Sarah Hoffmann [Fri, 19 Nov 2021 13:47:00 +0000 (14:47 +0100)]
only instantiate indexer once for replication

Also makes sure that indexer object exists everywhere were needed.

See #2518.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Thu, 11 Nov 2021 06:43:10 +0000 (07:43 +0100)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2517 from lonvia/transliteration-special-chars
Sarah Hoffmann [Thu, 11 Nov 2021 06:42:42 +0000 (07:42 +0100)]
Merge pull request #2517 from lonvia/transliteration-special-chars

ICU: avoid non-alphanumerical characters in transliteration

3 years agomake sure housenumbers are properly quoted
Sarah Hoffmann [Wed, 10 Nov 2021 16:15:34 +0000 (17:15 +0100)]
make sure housenumbers are properly quoted

3 years agoavoid special characters in word tokens
Sarah Hoffmann [Wed, 10 Nov 2021 16:14:13 +0000 (17:14 +0100)]
avoid special characters in word tokens

Transliteration should only consist of ASCII letters
and numbers. Avoid any other characters.

3 years agoMerge pull request #2516 from lonvia/test-for-website-dir
Sarah Hoffmann [Wed, 10 Nov 2021 12:27:09 +0000 (13:27 +0100)]
Merge pull request #2516 from lonvia/test-for-website-dir

Better error reporting when API script does not exist

3 years agobetter error reporting when API script does not exist
Sarah Hoffmann [Wed, 10 Nov 2021 08:42:49 +0000 (09:42 +0100)]
better error reporting when API script does not exist

Check if the API script exists on the expected location before
running php-cli. This way we can add a useful hint about the
project directory.

Fixes #2513.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Sat, 6 Nov 2021 11:12:16 +0000 (12:12 +0100)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2511 from lonvia/fix-combination-error-needs-address
Sarah Hoffmann [Sat, 6 Nov 2021 11:11:55 +0000 (12:11 +0100)]
Merge pull request #2511 from lonvia/fix-combination-error-needs-address

Fix boolean combination of NeedsAddress flag

3 years agofix combination of NeedsAddress flag
Sarah Hoffmann [Fri, 5 Nov 2021 21:18:37 +0000 (22:18 +0100)]
fix combination of NeedsAddress flag

When dealing with multiple partial terms, only keep the
flag, when all partial terms are so frequent as to need
an address.

Fixes #2510.

3 years agoincrease number of search descriptions tried
Sarah Hoffmann [Thu, 4 Nov 2021 10:16:05 +0000 (11:16 +0100)]
increase number of search descriptions tried

3 years agoprepare release 4.0.0 v4.0.0
Sarah Hoffmann [Mon, 1 Nov 2021 11:14:53 +0000 (12:14 +0100)]
prepare release 4.0.0

3 years agofix typo
Sarah Hoffmann [Tue, 2 Nov 2021 10:09:17 +0000 (11:09 +0100)]
fix typo

3 years agoMerge pull request #2502 from lonvia/improve-development-documentation
Sarah Hoffmann [Mon, 1 Nov 2021 15:12:23 +0000 (16:12 +0100)]
Merge pull request #2502 from lonvia/improve-development-documentation

Extend developer's documentation

3 years agodocs: add overview over indexing
Sarah Hoffmann [Mon, 1 Nov 2021 10:04:03 +0000 (11:04 +0100)]
docs: add overview over indexing

3 years agodocs: section about database layout
Sarah Hoffmann [Fri, 29 Oct 2021 10:03:22 +0000 (12:03 +0200)]
docs: section about database layout

Replaces the import description which basically was
table layout only now.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Thu, 28 Oct 2021 13:30:20 +0000 (15:30 +0200)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2498 from lonvia/ordering-for-unlisted-place-results
Sarah Hoffmann [Thu, 28 Oct 2021 13:28:47 +0000 (15:28 +0200)]
Merge pull request #2498 from lonvia/ordering-for-unlisted-place-results

Include unlisted places in ordering by housenumber

3 years agoMerge pull request #2497 from lonvia/docs-maintenance
Sarah Hoffmann [Thu, 28 Oct 2021 09:33:34 +0000 (11:33 +0200)]
Merge pull request #2497 from lonvia/docs-maintenance

docs: add new maintenance section

3 years agoinclude unlisted places in ordering by housenumber
Sarah Hoffmann [Thu, 28 Oct 2021 09:27:31 +0000 (11:27 +0200)]
include unlisted places in ordering by housenumber

When ordering results by the fact that they have a housenumber,
also take cases into account where the housenumber is on the
place itself. This may happen when the search includes the name
of the place and the housenumber or for addr:place addresses
where the place is unlisted.

3 years agodocs: add new maintenance section
Sarah Hoffmann [Wed, 27 Oct 2021 18:59:45 +0000 (20:59 +0200)]
docs: add new maintenance section

currently used for postcode updates, word count updates and
deleted relations.

3 years agoMerge pull request #2495 from lonvia/fix-normalization-in-php
Sarah Hoffmann [Wed, 27 Oct 2021 12:40:42 +0000 (14:40 +0200)]
Merge pull request #2495 from lonvia/fix-normalization-in-php

ICU: use correct normalization during search

3 years agoICU: use normalization from config in PHP
Sarah Hoffmann [Wed, 27 Oct 2021 08:07:19 +0000 (10:07 +0200)]
ICU: use normalization from config in PHP

The TERM_NORMALIZATION config option is no longer applicable.
That was already documented but not yet implemented.

3 years agobdd: add tests for non-latin scripts
Sarah Hoffmann [Tue, 26 Oct 2021 15:29:03 +0000 (17:29 +0200)]
bdd: add tests for non-latin scripts

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Tue, 26 Oct 2021 15:24:33 +0000 (17:24 +0200)]
Merge remote-tracking branch 'upstream/master'

3 years agoRevert "ICU: additional ranking by matching of normalised term"
Sarah Hoffmann [Tue, 26 Oct 2021 15:02:16 +0000 (17:02 +0200)]
Revert "ICU: additional ranking by matching of normalised term"

This reverts commit b6a831443c8f46bfe1be59ecd1cb7a6847fdf948.

3 years agoMerge pull request #2493 from lonvia/handle-frequent-partials
Sarah Hoffmann [Tue, 26 Oct 2021 15:00:43 +0000 (17:00 +0200)]
Merge pull request #2493 from lonvia/handle-frequent-partials

Tune search queries with frequent partial words

3 years agoadapt BDD tests to stricter partial search
Sarah Hoffmann [Tue, 26 Oct 2021 10:07:13 +0000 (12:07 +0200)]
adapt BDD tests to stricter partial search

3 years agodo not count words when in reverse-only mode
Sarah Hoffmann [Tue, 26 Oct 2021 09:42:42 +0000 (11:42 +0200)]
do not count words when in reverse-only mode

3 years agofurther refactor setup to keep function small
Sarah Hoffmann [Tue, 26 Oct 2021 08:57:51 +0000 (10:57 +0200)]
further refactor setup to keep function small

3 years agosearches for house numbers must have an address
Sarah Hoffmann [Tue, 26 Oct 2021 08:28:28 +0000 (10:28 +0200)]
searches for house numbers must have an address

3 years agodisallow search for partials without address
Sarah Hoffmann [Tue, 26 Oct 2021 08:23:55 +0000 (10:23 +0200)]
disallow search for partials without address

Very frequent partial terms take too long to look up and
do not return any valuable results unless the search is
further narrowed down by an address.

3 years agomake word count computation part of the import
Sarah Hoffmann [Tue, 26 Oct 2021 07:37:57 +0000 (09:37 +0200)]
make word count computation part of the import

Accurate word counts are now essential when using
the ICU tokenizer and don't hurt for the legacy one.

Adds about an hour import time.

3 years agoactions: move ICU tests into its own run
Sarah Hoffmann [Tue, 26 Oct 2021 08:32:43 +0000 (10:32 +0200)]
actions: move ICU tests into its own run

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Mon, 25 Oct 2021 19:46:18 +0000 (21:46 +0200)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2486 from lonvia/fix-special-phrases
Sarah Hoffmann [Mon, 25 Oct 2021 19:45:08 +0000 (21:45 +0200)]
Merge pull request #2486 from lonvia/fix-special-phrases

Fix parsing of operator in special phrases

3 years agoICU: add an index over word_ids
Sarah Hoffmann [Mon, 25 Oct 2021 19:33:27 +0000 (21:33 +0200)]
ICU: add an index over word_ids

Needed for keyword lookup in the details response.

3 years agoICU: additional ranking by matching of normalised term
Sarah Hoffmann [Mon, 25 Oct 2021 18:46:01 +0000 (20:46 +0200)]
ICU: additional ranking by matching of normalised term

Keep track of normalised word for tokens and then recheck
against normalized form in database to exclude non-matching
script.

3 years agobe case-insensitve about special phrase operator
Sarah Hoffmann [Mon, 25 Oct 2021 17:51:20 +0000 (19:51 +0200)]
be case-insensitve about special phrase operator

3 years agofix parsing of operator in special phrases
Sarah Hoffmann [Mon, 25 Oct 2021 17:46:30 +0000 (19:46 +0200)]
fix parsing of operator in special phrases

Because of unstripped input, the operators wouldn't match.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Mon, 25 Oct 2021 15:21:10 +0000 (17:21 +0200)]
Merge remote-tracking branch 'upstream/master'

3 years agoMerge pull request #2484 from lonvia/fix-index-use
Sarah Hoffmann [Mon, 25 Oct 2021 15:20:42 +0000 (17:20 +0200)]
Merge pull request #2484 from lonvia/fix-index-use

Reverse: add index hints

3 years agoMerge pull request #2483 from lonvia/fix-warming
Sarah Hoffmann [Mon, 25 Oct 2021 14:21:36 +0000 (16:21 +0200)]
Merge pull request #2483 from lonvia/fix-warming

Fix warming for ICU tokenizer

3 years agoreverse: add index hints
Sarah Hoffmann [Mon, 25 Oct 2021 12:55:15 +0000 (14:55 +0200)]
reverse: add index hints

The fairly complex where condition of idx_placex_geometry_placenode
won't always be matched by the query planner if the condition
part doesn't appear verbatim in the query.

Fixes #2480.

3 years agofix warming for ICU tokenizer
Sarah Hoffmann [Mon, 25 Oct 2021 11:08:16 +0000 (13:08 +0200)]
fix warming for ICU tokenizer

Running the warm-up search requests requires querying
the most frequent words. This must be done via the tokenizer
to honor the different formats of the word table.

3 years agoallow relative paths for log files
Sarah Hoffmann [Mon, 25 Oct 2021 08:13:11 +0000 (10:13 +0200)]
allow relative paths for log files

3 years agoMerge pull request #2476 from lonvia/harmonize-configuration-file-settings
Sarah Hoffmann [Sun, 24 Oct 2021 08:57:48 +0000 (10:57 +0200)]
Merge pull request #2476 from lonvia/harmonize-configuration-file-settings

Standardize handling of file names in configuration values

3 years agoallow relative paths for flatnode file
Sarah Hoffmann [Fri, 22 Oct 2021 15:32:51 +0000 (17:32 +0200)]
allow relative paths for flatnode file

3 years agoswitch IMPORT_STYLE to use generic file search
Sarah Hoffmann [Fri, 22 Oct 2021 14:49:57 +0000 (16:49 +0200)]
switch IMPORT_STYLE to use generic file search

Allows relative paths wrt project directory.

3 years agohave ADDRESS_LEVEL_CONFIG use load_sub_configuration
Sarah Hoffmann [Fri, 22 Oct 2021 14:31:33 +0000 (16:31 +0200)]
have ADDRESS_LEVEL_CONFIG use load_sub_configuration

This means that relative paths now are looked up in the
project directory.

3 years agoreplace NOMINATIM_PHRASE_CONFIG with command line option
Sarah Hoffmann [Fri, 22 Oct 2021 12:41:14 +0000 (14:41 +0200)]
replace NOMINATIM_PHRASE_CONFIG with command line option

3 years agodoc: clarify relative paths for tokenizer config
Sarah Hoffmann [Thu, 21 Oct 2021 14:38:06 +0000 (16:38 +0200)]
doc: clarify relative paths for tokenizer config

3 years agoMerge pull request #2475 from lonvia/catchup-mode
Sarah Hoffmann [Thu, 21 Oct 2021 14:21:58 +0000 (16:21 +0200)]
Merge pull request #2475 from lonvia/catchup-mode

Add catch-up mode to replication and extend documentation for updating

3 years agoextend documentation for updating database
Sarah Hoffmann [Thu, 21 Oct 2021 10:14:47 +0000 (12:14 +0200)]
extend documentation for updating database

Explains the different modes and adds hints for
setting up a systemd job.

3 years agoadd new replication mode catch-up
Sarah Hoffmann [Wed, 20 Oct 2021 20:05:15 +0000 (22:05 +0200)]
add new replication mode catch-up

This mode gets updates until the server reports no new diffs
anymore.

Also adds additional indexing, when the main indexing step left
a couple of objects to process. This happens only when the
next update is expected to be more than 40min away.

3 years agoMerge remote-tracking branch 'upstream/master'
Sarah Hoffmann [Tue, 19 Oct 2021 13:07:17 +0000 (15:07 +0200)]
Merge remote-tracking branch 'upstream/master'

3 years agorun Tiger import with parallel threads per default
Sarah Hoffmann [Tue, 19 Oct 2021 13:00:26 +0000 (15:00 +0200)]
run Tiger import with parallel threads per default

3 years agoMerge pull request #2472 from lonvia/word-count-computation
Sarah Hoffmann [Tue, 19 Oct 2021 12:58:57 +0000 (14:58 +0200)]
Merge pull request #2472 from lonvia/word-count-computation

Fix word count computation for ICU tokenizer

3 years agoadapt tests for new word count mechanism
Sarah Hoffmann [Tue, 19 Oct 2021 10:03:48 +0000 (12:03 +0200)]
adapt tests for new word count mechanism

3 years agoicu: no longer precompute terms
Sarah Hoffmann [Tue, 19 Oct 2021 09:50:06 +0000 (11:50 +0200)]
icu: no longer precompute terms

The ICU analyzer no longer drops frequent partials, so it is no
longer necessary to know the frequencies in advance.

3 years agomake word recount a tokenizer-specific function
Sarah Hoffmann [Tue, 19 Oct 2021 09:21:16 +0000 (11:21 +0200)]
make word recount a tokenizer-specific function

3 years agoMerge pull request #2471 from lonvia/update-install-rules
Sarah Hoffmann [Tue, 19 Oct 2021 07:11:16 +0000 (09:11 +0200)]
Merge pull request #2471 from lonvia/update-install-rules

Reorganise, update and extend documentation

3 years agodocs: fix more links
Sarah Hoffmann [Mon, 18 Oct 2021 15:26:14 +0000 (17:26 +0200)]
docs: fix more links

3 years agodocs: refer to our new Settings chapter in the import instruchtions
Sarah Hoffmann [Mon, 18 Oct 2021 15:02:52 +0000 (17:02 +0200)]
docs: refer to our new Settings chapter in the import instruchtions

3 years agocheck and fix all liks in documentation
Sarah Hoffmann [Mon, 18 Oct 2021 14:53:24 +0000 (16:53 +0200)]
check and fix all liks in documentation