complete documentation for new clean-houseunubmers sanatizer

[nominatim.git] / docs / customize / Tokenizers.md

diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md
index 4d5fbb150c8bb02612943a7c0ab02772a4f89f50..f75bc6a5c9da88fad8b92bc046337d2fcebff863 100644 (file)
--- a/docs/customize/Tokenizers.md
+++ b/docs/customize/Tokenizers.md
@@ -99,6 +99,9 @@ token-analysis:
           - words:
               - road -> rd
               - bridge -> bdge,br,brdg,bri,brg
           - words:
               - road -> rd
               - bridge -> bdge,br,brdg,bri,brg

+      mutations:
+          - pattern: 'ä'
+            replacements: ['ä', 'ae']

 ```
 
 The configuration file contains four sections:
 ```
 
 The configuration file contains four sections:


@@ -178,6 +181,13 @@ The following is a list of sanitizers that are shipped with Nominatim.
     rendering:
         heading_level: 6
 
     rendering:
         heading_level: 6
 

+##### clean-housenumbers
+
+::: nominatim.tokenizer.sanitizers.clean_housenumbers
+    selection:
+        members: False
+    rendering:
+        heading_level: 6

 
 
 #### Token Analysis
 
 
 #### Token Analysis


@@ -205,12 +215,11 @@ the `analyzer` parameter must be set. Currently there is only one implementation
 ##### Generic token analyzer
 
 The generic analyzer is able to create variants from a list of given
 ##### Generic token analyzer
 
 The generic analyzer is able to create variants from a list of given

-abbreviation and decomposition replacements. It takes one optional parameter
-`variants` which lists the replacements to apply. If the section is
-omitted, then the generic analyzer becomes a simple analyzer that only
-applies the transliteration.
+abbreviation and decomposition replacements and introduce spelling variations.
+
+###### Variants

 
 

-The variants section defines lists of replacements which create alternative
+The optional 'variants' section defines lists of replacements which create alternative

 spellings of a name. To create the variants, a name is scanned from left to
 right and the longest matching replacement is applied until the end of the
 string is reached.
 spellings of a name. To create the variants, a name is scanned from left to
 right and the longest matching replacement is applied until the end of the
 string is reached.


@@ -296,6 +305,32 @@ decomposition has an effect here on the source as well. So a rule
 means that for a word like `hauptstrasse` four variants are created:
 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
 
 means that for a word like `hauptstrasse` four variants are created:
 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
 

+###### Mutations
+
+The 'mutation' section in the configuration describes an additional set of
+replacements to be applied after the variants have been computed.
+
+Each mutation is described by two parameters: `pattern` and `replacements`.
+The pattern must contain a single regular expression to search for in the
+variant name. The regular expressions need to follow the syntax for
+[Python regular expressions](file:///usr/share/doc/python3-doc/html/library/re.html#regular-expression-syntax).
+Capturing groups are not permitted.
+`replacements` must contain a list of strings that the pattern
+should be replaced with. Each occurrence of the pattern is replaced with
+all given replacements. Be mindful of combinatorial explosion of variants.
+
+###### Modes
+
+The generic analyser supports a special mode `variant-only`. When configured
+then it consumes the input token and emits only variants (if any exist). Enable
+the mode by adding:
+
+```
+  mode: variant-only
+```
+
+to the analyser configuration.
+

 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although
 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although