remove documentation around legacy tokenizer

[nominatim.git] / docs / customize / Tokenizers.md

diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md
index f75bc6a5c9da88fad8b92bc046337d2fcebff863..30be170edea91babfa271d47364d123d84f9fdf0 100644 (file)
--- a/docs/customize/Tokenizers.md
+++ b/docs/customize/Tokenizers.md
@@ -15,38 +15,12 @@ they can be configured.
     chosen tokenizer is very limited as well. See the comments in each tokenizer
     section.
 
     chosen tokenizer is very limited as well. See the comments in each tokenizer
     section.
 

-## Legacy tokenizer
-
-The legacy tokenizer implements the analysis algorithms of older Nominatim
-versions. It uses a special Postgresql module to normalize names and queries.
-This tokenizer is currently the default.
-
-To enable the tokenizer add the following line to your project configuration:
-
-```
-NOMINATIM_TOKENIZER=legacy
-```
-
-The Postgresql module for the tokenizer is available in the `module` directory
-and also installed with the remainder of the software under
-`lib/nominatim/module/nominatim.so`. You can specify a custom location for
-the module with
-
-```
-NOMINATIM_DATABASE_MODULE_PATH=<path to directory where nominatim.so resides>
-```
-
-This is in particular useful when the database runs on a different server.
-See [Advanced installations](../admin/Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details.
-
-There are no other configuration options for the legacy tokenizer. All
-normalization functions are hard-coded.
-

 ## ICU tokenizer
 
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
 normalize names and queries. It also offers configurable decomposition and
 abbreviation handling.
 ## ICU tokenizer
 
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
 normalize names and queries. It also offers configurable decomposition and
 abbreviation handling.

+This tokenizer is currently the default.

 
 To enable the tokenizer add the following line to your project configuration:
 
 
 To enable the tokenizer add the following line to your project configuration:
 


@@ -86,7 +60,7 @@ Here is an example configuration file:
 ``` yaml
 normalization:
     - ":: lower ()"
 ``` yaml
 normalization:
     - ":: lower ()"

-    - "ß > 'ss'" # German szet is unimbigiously equal to double ss
+    - "ß > 'ss'" # German szet is unambiguously equal to double ss

 transliteration:
     - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml
     - ":: Ascii ()"
 transliteration:
     - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml
     - ":: Ascii ()"


@@ -112,7 +86,7 @@ The configuration file contains four sections:
 The normalization and transliteration sections each define a set of
 ICU rules that are applied to the names.
 
 The normalization and transliteration sections each define a set of
 ICU rules that are applied to the names.
 

-The **normalisation** rules are applied after sanitation. They should remove
+The **normalization** rules are applied after sanitation. They should remove

 any information that is not relevant for search at all. Usual rules to be
 applied here are: lower-casing, removing of special characters, cleanup of
 spaces.
 any information that is not relevant for search at all. Usual rules to be
 applied here are: lower-casing, removing of special characters, cleanup of
 spaces.


@@ -159,36 +133,67 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### split-name-list
 
 
 ##### split-name-list
 

-::: nominatim.tokenizer.sanitizers.split_name_list
-    selection:
+::: nominatim_db.tokenizer.sanitizers.split_name_list
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### strip-brace-terms
 
 
 ##### strip-brace-terms
 

-::: nominatim.tokenizer.sanitizers.strip_brace_terms
-    selection:
+::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### tag-analyzer-by-language
 
 
 ##### tag-analyzer-by-language
 

-::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
-    selection:
+::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### clean-housenumbers
 
 
 ##### clean-housenumbers
 

-::: nominatim.tokenizer.sanitizers.clean_housenumbers
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+##### clean-postcodes
+
+::: nominatim_db.tokenizer.sanitizers.clean_postcodes
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 

+##### clean-tiger-tags
+
+::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+#### delete-tags
+
+::: nominatim_db.tokenizer.sanitizers.delete_tags
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+#### tag-japanese
+
+::: nominatim_db.tokenizer.sanitizers.tag_japanese
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy

 
 #### Token Analysis
 
 
 #### Token Analysis
 


@@ -206,15 +211,20 @@ by a sanitizer (see for example the
 The token-analysis section contains the list of configured analyzers. Each
 analyzer must have an `id` parameter that uniquely identifies the analyzer.
 The only exception is the default analyzer that is used when no special
 The token-analysis section contains the list of configured analyzers. Each
 analyzer must have an `id` parameter that uniquely identifies the analyzer.
 The only exception is the default analyzer that is used when no special

-analyzer was selected.
+analyzer was selected. There are analysers with special ids:
+
+ * '@housenumber'. If an analyzer with that name is present, it is used
+   for normalization of house numbers.
+ * '@potcode'. If an analyzer with that name is present, it is used
+   for normalization of postcodes.

 
 Different analyzer implementations may exist. To select the implementation,
 
 Different analyzer implementations may exist. To select the implementation,

-the `analyzer` parameter must be set. Currently there is only one implementation
-`generic` which is described in the following.
+the `analyzer` parameter must be set. The different implementations are
+described in the following.

 
 ##### Generic token analyzer
 
 
 ##### Generic token analyzer
 

-The generic analyzer is able to create variants from a list of given
+The generic analyzer `generic` is able to create variants from a list of given

 abbreviation and decomposition replacements and introduce spelling variations.
 
 ###### Variants
 abbreviation and decomposition replacements and introduce spelling variations.
 
 ###### Variants


@@ -331,6 +341,22 @@ the mode by adding:
 
 to the analyser configuration.
 
 
 to the analyser configuration.
 

+##### Housenumber token analyzer
+
+The analyzer `housenumbers` is purpose-made to analyze house numbers. It
+creates variants with optional spaces between numbers and letters. Thus,
+house numbers of the form '3 a', '3A', '3-A' etc. are all considered equivalent.
+
+The analyzer cannot be customized.
+
+##### Postcode token analyzer
+
+The analyzer `postcodes` is pupose-made to analyze postcodes. It supports
+a 'lookup' variant of the token, which produces variants with optional
+spaces. Use together with the clean-postcodes sanitizer.
+
+The analyzer cannot be customized.
+

 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although
 ### Reconfiguration
 
 Changing the configuration after the import is currently not possible, although