Sarah Hoffmann [Wed, 26 May 2021 09:04:02 +0000 (11:04 +0200)]
always compute guessed postcode for POIs from centroid
When guessing postcodes from the area, only postcodes within
that area are accepted. For POIs that is usually not what we
want as the postcode would have to be within a house for
example.
Sarah Hoffmann [Sun, 23 May 2021 21:58:58 +0000 (23:58 +0200)]
reorganize keyword creation for legacy tokenizer
- only save partial words without internal spaces
- consider comma and semicolon a separator of full words
- consider parts before an opening bracket a full word
(but not the part after the bracket)
Sarah Hoffmann [Tue, 18 May 2021 14:28:21 +0000 (16:28 +0200)]
do not hide errors when importing tokenizer
Explicitly check for the tokenizer source file to check that
the name is correct. We can't use the import error for that
because it hides other import errors like a missing
library.
Sarah Hoffmann [Mon, 17 May 2021 14:36:32 +0000 (16:36 +0200)]
always use object type for details keywords
When name and address is empty, the keywords field in the response
of the details API would be an array because that is what PHP's
json_encode defaults to with empty array(). This default can only
be changed globally per json_encode call and that might cause
unintended colleteral damage. Work around the issue by making
name and address an empty array instead of keywords.
Sarah Hoffmann [Wed, 12 May 2021 17:57:48 +0000 (19:57 +0200)]
move filling of postcode table to python
The Python code now takes care of reading postcodes from placex,
enhancing them with potentially existing external postcodes and
updating location_postcodes accordingly. The initial setup and
updates use exactly the same function.
External postcode handling has been generalized. External postcodes
for any country are now accepted. The format of the external postcode
file has changed. We now expect CSV, potentially gzipped. The
postcodes are no longer saved in the database.
Sarah Hoffmann [Wed, 5 May 2021 19:16:55 +0000 (21:16 +0200)]
add missing transliterations
The ICU library only offers transliterations for a limited set of
script. Add transliterations for missing scripts from the PostgreSQL
module. These means that the same selection of scripts is supported
as with the old module.
Sarah Hoffmann [Wed, 5 May 2021 08:00:34 +0000 (10:00 +0200)]
enable BDD tests for different tokenizers
The tokenizer to be used can be choosen with -DTOKENIZER.
Adapt all tests, so that they work with legacy_icu tokenizer.
Move lookup in word table to a function in the tokenizer.
Special phrases are temporarily imported from the wiki until
we have an implementation that can import from file. TIGER
tests do not work yet.
Sarah Hoffmann [Fri, 30 Apr 2021 14:17:28 +0000 (16:17 +0200)]
indexer: fetch extra place data asynchronously
The indexer now fetches any extra data besides the place_id
asynchronously while processing the places from the last batch.
This also means that more places are now fetched at once.
Sarah Hoffmann [Wed, 28 Apr 2021 08:59:07 +0000 (10:59 +0200)]
boilerplate for PHP code of tokenizer
This adds an installation step for PHP code for the tokenizer. The
PHP code is split in two parts. The updateable code is found in
lib-php. The tokenizer installs an additional script in the
project directory which then includes the code from lib-php and
defines all settings that are static to the database. The website
code then always includes the PHP from the project directory.
Sarah Hoffmann [Tue, 27 Apr 2021 19:50:35 +0000 (21:50 +0200)]
move amenity creation to tokenizer
The BDD tests still use the old-style amenity creation scripts
because we don't have simple means to import a hand-crafted
test file of special phrases right now.
Sarah Hoffmann [Sun, 25 Apr 2021 09:47:29 +0000 (11:47 +0200)]
move houseunumber handling to tokenizer
Normalization and token computation are now done in the tokenizer.
The tokenizer keeps a cache to the hundred most used house numbers
to keep the numbers of calls to the database low.