release 4.5.0.post1

[nominatim.git] / docs / customize / Tokenizers.md

diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md
index 11c27e38b903ae0683ace099f417ec16b1077bc8..30be170edea91babfa271d47364d123d84f9fdf0 100644 (file)
--- a/docs/customize/Tokenizers.md
+++ b/docs/customize/Tokenizers.md
@@ -15,48 +15,6 @@ they can be configured.
     chosen tokenizer is very limited as well. See the comments in each tokenizer
     section.
 
-## Legacy tokenizer
-
-The legacy tokenizer implements the analysis algorithms of older Nominatim
-versions. It uses a special Postgresql module to normalize names and queries.
-This tokenizer is automatically installed and used when upgrading an older
-database. It should not be used for new installations anymore.
-
-### Compiling the PostgreSQL module
-
-The tokeinzer needs a special C module for PostgreSQL which is not compiled
-by default. If you need the legacy tokenizer, compile Nominatim as follows:
-
-```
-mkdir build
-cd build
-cmake -DBUILD_MODULE=on
-make
-```
-
-### Enabling the tokenizer
-
-To enable the tokenizer add the following line to your project configuration:
-
-```
-NOMINATIM_TOKENIZER=legacy
-```
-
-The Postgresql module for the tokenizer is available in the `module` directory
-and also installed with the remainder of the software under
-`lib/nominatim/module/nominatim.so`. You can specify a custom location for
-the module with
-
-```
-NOMINATIM_DATABASE_MODULE_PATH=<path to directory where nominatim.so resides>
-```
-
-This is in particular useful when the database runs on a different server.
-See [Advanced installations](../admin/Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details.
-
-There are no other configuration options for the legacy tokenizer. All
-normalization functions are hard-coded.
-
 ## ICU tokenizer
 
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
@@ -175,59 +133,67 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### split-name-list
 
-::: nominatim.tokenizer.sanitizers.split_name_list
-    selection:
+::: nominatim_db.tokenizer.sanitizers.split_name_list
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 ##### strip-brace-terms
 
-::: nominatim.tokenizer.sanitizers.strip_brace_terms
-    selection:
+::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 ##### tag-analyzer-by-language
 
-::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
-    selection:
+::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 ##### clean-housenumbers
 
-::: nominatim.tokenizer.sanitizers.clean_housenumbers
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 ##### clean-postcodes
 
-::: nominatim.tokenizer.sanitizers.clean_postcodes
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_postcodes
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 ##### clean-tiger-tags
 
-::: nominatim.tokenizer.sanitizers.clean_tiger_tags
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 #### delete-tags
 
-::: nominatim.tokenizer.sanitizers.delete_tags
-    selection:
+::: nominatim_db.tokenizer.sanitizers.delete_tags
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+#### tag-japanese
+
+::: nominatim_db.tokenizer.sanitizers.tag_japanese
+    options:
         members: False
-    rendering:
         heading_level: 6
+        docstring_section_style: spacy
 
 #### Token Analysis
 
@@ -386,7 +352,7 @@ The analyzer cannot be customized.
 ##### Postcode token analyzer
 
 The analyzer `postcodes` is pupose-made to analyze postcodes. It supports
-a 'lookup' varaint of the token, which produces variants with optional
+a 'lookup' variant of the token, which produces variants with optional
 spaces. Use together with the clean-postcodes sanitizer.
 
 The analyzer cannot be customized.