X-Git-Url: https://git.openstreetmap.org./nominatim.git/blobdiff_plain/b694a97edfa5b25a9c4b0e06b2423a5b1ad666a4..f535340d5af0b59c1bbdfde885b62dd483a9e38e:/docs/customize/Tokenizers.md diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md index d849eb48..49e86a50 100644 --- a/docs/customize/Tokenizers.md +++ b/docs/customize/Tokenizers.md @@ -17,9 +17,29 @@ they can be configured. ## Legacy tokenizer +!!! danger + The Legacy tokenizer is deprecated and will be removed in Nominatim 5.0. + If you still use a database with the legacy tokenizer, you must reimport + it using the ICU tokenizer below. + The legacy tokenizer implements the analysis algorithms of older Nominatim versions. It uses a special Postgresql module to normalize names and queries. -This tokenizer is currently the default. +This tokenizer is automatically installed and used when upgrading an older +database. It should not be used for new installations anymore. + +### Compiling the PostgreSQL module + +The tokeinzer needs a special C module for PostgreSQL which is not compiled +by default. If you need the legacy tokenizer, compile Nominatim as follows: + +``` +mkdir build +cd build +cmake -DBUILD_MODULE=on +make +``` + +### Enabling the tokenizer To enable the tokenizer add the following line to your project configuration: @@ -37,7 +57,7 @@ NOMINATIM_DATABASE_MODULE_PATH= ``` This is in particular useful when the database runs on a different server. -See [Advanced installations](../admin/Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details. +See [Advanced installations](../admin/Advanced-Installations.md#using-an-external-postgresql-database) for details. There are no other configuration options for the legacy tokenizer. All normalization functions are hard-coded. @@ -47,6 +67,7 @@ normalization functions are hard-coded. The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to normalize names and queries. It also offers configurable decomposition and abbreviation handling. +This tokenizer is currently the default. To enable the tokenizer add the following line to your project configuration: @@ -86,7 +107,7 @@ Here is an example configuration file: ``` yaml normalization: - ":: lower ()" - - "ß > 'ss'" # German szet is unimbigiously equal to double ss + - "ß > 'ss'" # German szet is unambiguously equal to double ss transliteration: - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml - ":: Ascii ()" @@ -112,7 +133,7 @@ The configuration file contains four sections: The normalization and transliteration sections each define a set of ICU rules that are applied to the names. -The **normalisation** rules are applied after sanitation. They should remove +The **normalization** rules are applied after sanitation. They should remove any information that is not relevant for search at all. Usual rules to be applied here are: lower-casing, removing of special characters, cleanup of spaces. @@ -159,36 +180,67 @@ The following is a list of sanitizers that are shipped with Nominatim. ##### split-name-list -::: nominatim.tokenizer.sanitizers.split_name_list - selection: +::: nominatim_db.tokenizer.sanitizers.split_name_list + options: members: False - rendering: heading_level: 6 + docstring_section_style: spacy ##### strip-brace-terms -::: nominatim.tokenizer.sanitizers.strip_brace_terms - selection: +::: nominatim_db.tokenizer.sanitizers.strip_brace_terms + options: members: False - rendering: heading_level: 6 + docstring_section_style: spacy ##### tag-analyzer-by-language -::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language - selection: +::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language + options: members: False - rendering: heading_level: 6 + docstring_section_style: spacy ##### clean-housenumbers -::: nominatim.tokenizer.sanitizers.clean_housenumbers - selection: +::: nominatim_db.tokenizer.sanitizers.clean_housenumbers + options: members: False - rendering: heading_level: 6 + docstring_section_style: spacy + +##### clean-postcodes +::: nominatim_db.tokenizer.sanitizers.clean_postcodes + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +##### clean-tiger-tags + +::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +#### delete-tags + +::: nominatim_db.tokenizer.sanitizers.delete_tags + options: + members: False + heading_level: 6 + docstring_section_style: spacy + +#### tag-japanese + +::: nominatim_db.tokenizer.sanitizers.tag_japanese + options: + members: False + heading_level: 6 + docstring_section_style: spacy #### Token Analysis @@ -206,8 +258,12 @@ by a sanitizer (see for example the The token-analysis section contains the list of configured analyzers. Each analyzer must have an `id` parameter that uniquely identifies the analyzer. The only exception is the default analyzer that is used when no special -analyzer was selected. There is one special id '@housenumber'. If an analyzer -with that name is present, it is used for normalization of house numbers. +analyzer was selected. There are analysers with special ids: + + * '@housenumber'. If an analyzer with that name is present, it is used + for normalization of house numbers. + * '@potcode'. If an analyzer with that name is present, it is used + for normalization of postcodes. Different analyzer implementations may exist. To select the implementation, the `analyzer` parameter must be set. The different implementations are @@ -340,6 +396,14 @@ house numbers of the form '3 a', '3A', '3-A' etc. are all considered equivalent. The analyzer cannot be customized. +##### Postcode token analyzer + +The analyzer `postcodes` is pupose-made to analyze postcodes. It supports +a 'lookup' variant of the token, which produces variants with optional +spaces. Use together with the clean-postcodes sanitizer. + +The analyzer cannot be customized. + ### Reconfiguration Changing the configuration after the import is currently not possible, although