Merge remote-tracking branch 'upstream/master'

[nominatim.git] / docs / customize / Tokenizers.md

diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md
index 4d5fbb150c8bb02612943a7c0ab02772a4f89f50..2c7b687834ba7cd53308833d6179586813ba29aa 100644 (file)
--- a/docs/customize/Tokenizers.md
+++ b/docs/customize/Tokenizers.md
@@ -19,7 +19,22 @@ they can be configured.
 
 The legacy tokenizer implements the analysis algorithms of older Nominatim
 versions. It uses a special Postgresql module to normalize names and queries.
 
 The legacy tokenizer implements the analysis algorithms of older Nominatim
 versions. It uses a special Postgresql module to normalize names and queries.

-This tokenizer is currently the default.
+This tokenizer is automatically installed and used when upgrading an older
+database. It should not be used for new installations anymore.
+
+### Compiling the PostgreSQL module
+
+The tokeinzer needs a special C module for PostgreSQL which is not compiled
+by default. If you need the legacy tokenizer, compile Nominatim as follows:
+
+```
+mkdir build
+cd build
+cmake -DBUILD_MODULE=on
+make
+```
+
+### Enabling the tokenizer

 
 To enable the tokenizer add the following line to your project configuration:
 
 
 To enable the tokenizer add the following line to your project configuration:
 


@@ -47,6 +62,7 @@ normalization functions are hard-coded.
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
 normalize names and queries. It also offers configurable decomposition and
 abbreviation handling.
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
 normalize names and queries. It also offers configurable decomposition and
 abbreviation handling.

+This tokenizer is currently the default.

 
 To enable the tokenizer add the following line to your project configuration:
 
 
 To enable the tokenizer add the following line to your project configuration:
 


@@ -86,7 +102,7 @@ Here is an example configuration file:
 ``` yaml
 normalization:
     - ":: lower ()"
 ``` yaml
 normalization:
     - ":: lower ()"

-    - "ß > 'ss'" # German szet is unimbigiously equal to double ss
+    - "ß > 'ss'" # German szet is unambiguously equal to double ss

 transliteration:
     - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml
     - ":: Ascii ()"
 transliteration:
     - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml
     - ":: Ascii ()"


@@ -99,6 +115,9 @@ token-analysis:
           - words:
               - road -> rd
               - bridge -> bdge,br,brdg,bri,brg
           - words:
               - road -> rd
               - bridge -> bdge,br,brdg,bri,brg

+      mutations:
+          - pattern: 'ä'
+            replacements: ['ä', 'ae']

 ```
 
 The configuration file contains four sections:
 ```
 
 The configuration file contains four sections:


@@ -109,7 +128,7 @@ The configuration file contains four sections:
 The normalization and transliteration sections each define a set of
 ICU rules that are applied to the names.
 
 The normalization and transliteration sections each define a set of
 ICU rules that are applied to the names.
 

-The **normalisation** rules are applied after sanitation. They should remove
+The **normalization** rules are applied after sanitation. They should remove

 any information that is not relevant for search at all. Usual rules to be
 applied here are: lower-casing, removing of special characters, cleanup of
 spaces.
 any information that is not relevant for search at all. Usual rules to be
 applied here are: lower-casing, removing of special characters, cleanup of
 spaces.


@@ -157,28 +176,66 @@ The following is a list of sanitizers that are shipped with Nominatim.
 ##### split-name-list
 
 ::: nominatim.tokenizer.sanitizers.split_name_list
 ##### split-name-list
 
 ::: nominatim.tokenizer.sanitizers.split_name_list

-    selection:
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### strip-brace-terms
 
 ::: nominatim.tokenizer.sanitizers.strip_brace_terms
 
 ##### strip-brace-terms
 
 ::: nominatim.tokenizer.sanitizers.strip_brace_terms

-    selection:
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### tag-analyzer-by-language
 
 ::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
 
 ##### tag-analyzer-by-language
 
 ::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language

-    selection:
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+##### clean-housenumbers
+
+::: nominatim.tokenizer.sanitizers.clean_housenumbers
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+##### clean-postcodes
+
+::: nominatim.tokenizer.sanitizers.clean_postcodes
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 

+##### clean-tiger-tags

 
 

+::: nominatim.tokenizer.sanitizers.clean_tiger_tags
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+#### delete-tags
+
+::: nominatim.tokenizer.sanitizers.delete_tags
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+#### tag-japanese
+
+::: nominatim.tokenizer.sanitizers.tag_japanese
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy

 
 #### Token Analysis
 
 
 #### Token Analysis
 


@@ -196,21 +253,25 @@ by a sanitizer (see for example the
 The token-analysis section contains the list of configured analyzers. Each
 analyzer must have an `id` parameter that uniquely identifies the analyzer.
 The only exception is the default analyzer that is used when no special
 The token-analysis section contains the list of configured analyzers. Each
 analyzer must have an `id` parameter that uniquely identifies the analyzer.
 The only exception is the default analyzer that is used when no special

-analyzer was selected.
+analyzer was selected. There are analysers with special ids:
+
+ * '@housenumber'. If an analyzer with that name is present, it is used
+   for normalization of house numbers.
+ * '@potcode'. If an analyzer with that name is present, it is used
+   for normalization of postcodes.

 
 Different analyzer implementations may exist. To select the implementation,
 
 Different analyzer implementations may exist. To select the implementation,

-the `analyzer` parameter must be set. Currently there is only one implementation
-`generic` which is described in the following.
+the `analyzer` parameter must be set. The different implementations are
+described in the following.

 
 ##### Generic token analyzer
 
 
 ##### Generic token analyzer
 

-The generic analyzer is able to create variants from a list of given
-abbreviation and decomposition replacements. It takes one optional parameter
-`variants` which lists the replacements to apply. If the section is
-omitted, then the generic analyzer becomes a simple analyzer that only
-applies the transliteration.
+The generic analyzer `generic` is able to create variants from a list of given
+abbreviation and decomposition replacements and introduce spelling variations.
+
+###### Variants

 
 

-The variants section defines lists of replacements which create alternative
+The optional 'variants' section defines lists of replacements which create alternative

 spellings of a name. To create the variants, a name is scanned from left to
 right and the longest matching replacement is applied until the end of the
 string is reached.
 spellings of a name. To create the variants, a name is scanned from left to
 right and the longest matching replacement is applied until the end of the
 string is reached.


@@ -296,6 +357,48 @@ decomposition has an effect here on the source as well. So a rule
 means that for a word like `hauptstrasse` four variants are created:
 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
 
 means that for a word like `hauptstrasse` four variants are created:
 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
 

+###### Mutations
+
+The 'mutation' section in the configuration describes an additional set of
+replacements to be applied after the variants have been computed.
+
+Each mutation is described by two parameters: `pattern` and `replacements`.
+The pattern must contain a single regular expression to search for in the
+variant name. The regular expressions need to follow the syntax for
+[Python regular expressions](file:///usr/share/doc/python3-doc/html/library/re.html#regular-expression-syntax).
+Capturing groups are not permitted.
+`replacements` must contain a list of strings that the pattern
+should be replaced with. Each occurrence of the pattern is replaced with
+all given replacements. Be mindful of combinatorial explosion of variants.
+
+###### Modes
+
+The generic analyser supports a special mode `variant-only`. When configured
+then it consumes the input token and emits only variants (if any exist). Enable
+the mode by adding:
+
+```
+  mode: variant-only
+```
+
+to the analyser configuration.
+
+##### Housenumber token analyzer
+
+The analyzer `housenumbers` is purpose-made to analyze house numbers. It
+creates variants with optional spaces between numbers and letters. Thus,
+house numbers of the form '3 a', '3A', '3-A' etc. are all considered equivalent.
+
+The analyzer cannot be customized.
+
+##### Postcode token analyzer
+
+The analyzer `postcodes` is pupose-made to analyze postcodes. It supports
+a 'lookup' varaint of the token, which produces variants with optional
+spaces. Use together with the clean-postcodes sanitizer.
+
+The analyzer cannot be customized.
+

 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although
 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although