fix linting error

[nominatim.git] / docs / customize / Tokenizers.md

diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md
index d3d04502c00c3d9f1a79a27822418722d1c5adb0..d849eb48c0d457c7c57b27e2807ff55675e2fa33 100644 (file)
--- a/docs/customize/Tokenizers.md
+++ b/docs/customize/Tokenizers.md
@@ -37,7 +37,7 @@ NOMINATIM_DATABASE_MODULE_PATH=<path to directory where nominatim.so resides>
 ```
 
 This is in particular useful when the database runs on a different server.
 ```
 
 This is in particular useful when the database runs on a different server.

-See [Advanced installations](Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details.
+See [Advanced installations](../admin/Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details.

 
 There are no other configuration options for the legacy tokenizer. All
 normalization functions are hard-coded.
 
 There are no other configuration options for the legacy tokenizer. All
 normalization functions are hard-coded.


@@ -99,6 +99,9 @@ token-analysis:
           - words:
               - road -> rd
               - bridge -> bdge,br,brdg,bri,brg
           - words:
               - road -> rd
               - bridge -> bdge,br,brdg,bri,brg

+      mutations:
+          - pattern: 'ä'
+            replacements: ['ä', 'ae']

 ```
 
 The configuration file contains four sections:
 ```
 
 The configuration file contains four sections:


@@ -178,6 +181,13 @@ The following is a list of sanitizers that are shipped with Nominatim.
     rendering:
         heading_level: 6
 
     rendering:
         heading_level: 6
 

+##### clean-housenumbers
+
+::: nominatim.tokenizer.sanitizers.clean_housenumbers
+    selection:
+        members: False
+    rendering:
+        heading_level: 6

 
 
 #### Token Analysis
 
 
 #### Token Analysis


@@ -196,21 +206,21 @@ by a sanitizer (see for example the
 The token-analysis section contains the list of configured analyzers. Each
 analyzer must have an `id` parameter that uniquely identifies the analyzer.
 The only exception is the default analyzer that is used when no special
 The token-analysis section contains the list of configured analyzers. Each
 analyzer must have an `id` parameter that uniquely identifies the analyzer.
 The only exception is the default analyzer that is used when no special

-analyzer was selected.
+analyzer was selected. There is one special id '@housenumber'. If an analyzer
+with that name is present, it is used for normalization of house numbers.

 
 Different analyzer implementations may exist. To select the implementation,
 
 Different analyzer implementations may exist. To select the implementation,

-the `analyzer` parameter must be set. Currently there is only one implementation
-`generic` which is described in the following.
+the `analyzer` parameter must be set. The different implementations are
+described in the following.

 
 ##### Generic token analyzer
 
 
 ##### Generic token analyzer
 

-The generic analyzer is able to create variants from a list of given
-abbreviation and decomposition replacements. It takes one optional parameter
-`variants` which lists the replacements to apply. If the section is
-omitted, then the generic analyzer becomes a simple analyzer that only
-applies the transliteration.
+The generic analyzer `generic` is able to create variants from a list of given
+abbreviation and decomposition replacements and introduce spelling variations.

 
 

-The variants section defines lists of replacements which create alternative
+###### Variants
+
+The optional 'variants' section defines lists of replacements which create alternative

 spellings of a name. To create the variants, a name is scanned from left to
 right and the longest matching replacement is applied until the end of the
 string is reached.
 spellings of a name. To create the variants, a name is scanned from left to
 right and the longest matching replacement is applied until the end of the
 string is reached.


@@ -296,6 +306,40 @@ decomposition has an effect here on the source as well. So a rule
 means that for a word like `hauptstrasse` four variants are created:
 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
 
 means that for a word like `hauptstrasse` four variants are created:
 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
 

+###### Mutations
+
+The 'mutation' section in the configuration describes an additional set of
+replacements to be applied after the variants have been computed.
+
+Each mutation is described by two parameters: `pattern` and `replacements`.
+The pattern must contain a single regular expression to search for in the
+variant name. The regular expressions need to follow the syntax for
+[Python regular expressions](file:///usr/share/doc/python3-doc/html/library/re.html#regular-expression-syntax).
+Capturing groups are not permitted.
+`replacements` must contain a list of strings that the pattern
+should be replaced with. Each occurrence of the pattern is replaced with
+all given replacements. Be mindful of combinatorial explosion of variants.
+
+###### Modes
+
+The generic analyser supports a special mode `variant-only`. When configured
+then it consumes the input token and emits only variants (if any exist). Enable
+the mode by adding:
+
+```
+  mode: variant-only
+```
+
+to the analyser configuration.
+
+##### Housenumber token analyzer
+
+The analyzer `housenumbers` is purpose-made to analyze house numbers. It
+creates variants with optional spaces between numbers and letters. Thus,
+house numbers of the form '3 a', '3A', '3-A' etc. are all considered equivalent.
+
+The analyzer cannot be customized.
+

 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although
 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although