remove documentation around legacy tokenizer

[nominatim.git] / docs / customize / Tokenizers.md

diff --git a/docs/customize/Tokenizers.md b/docs/customize/Tokenizers.md
index 11c27e38b903ae0683ace099f417ec16b1077bc8..30be170edea91babfa271d47364d123d84f9fdf0 100644 (file)
--- a/docs/customize/Tokenizers.md
+++ b/docs/customize/Tokenizers.md
@@ -15,48 +15,6 @@ they can be configured.
     chosen tokenizer is very limited as well. See the comments in each tokenizer
     section.
 
     chosen tokenizer is very limited as well. See the comments in each tokenizer
     section.
 

-## Legacy tokenizer
-
-The legacy tokenizer implements the analysis algorithms of older Nominatim
-versions. It uses a special Postgresql module to normalize names and queries.
-This tokenizer is automatically installed and used when upgrading an older
-database. It should not be used for new installations anymore.
-
-### Compiling the PostgreSQL module
-
-The tokeinzer needs a special C module for PostgreSQL which is not compiled
-by default. If you need the legacy tokenizer, compile Nominatim as follows:
-
-```
-mkdir build
-cd build
-cmake -DBUILD_MODULE=on
-make
-```
-
-### Enabling the tokenizer
-
-To enable the tokenizer add the following line to your project configuration:
-
-```
-NOMINATIM_TOKENIZER=legacy
-```
-
-The Postgresql module for the tokenizer is available in the `module` directory
-and also installed with the remainder of the software under
-`lib/nominatim/module/nominatim.so`. You can specify a custom location for
-the module with
-
-```
-NOMINATIM_DATABASE_MODULE_PATH=<path to directory where nominatim.so resides>
-```
-
-This is in particular useful when the database runs on a different server.
-See [Advanced installations](../admin/Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details.
-
-There are no other configuration options for the legacy tokenizer. All
-normalization functions are hard-coded.
-

 ## ICU tokenizer
 
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
 ## ICU tokenizer
 
 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to


@@ -175,59 +133,67 @@ The following is a list of sanitizers that are shipped with Nominatim.
 
 ##### split-name-list
 
 
 ##### split-name-list
 

-::: nominatim.tokenizer.sanitizers.split_name_list
-    selection:
+::: nominatim_db.tokenizer.sanitizers.split_name_list
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### strip-brace-terms
 
 
 ##### strip-brace-terms
 

-::: nominatim.tokenizer.sanitizers.strip_brace_terms
-    selection:
+::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### tag-analyzer-by-language
 
 
 ##### tag-analyzer-by-language
 

-::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
-    selection:
+::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### clean-housenumbers
 
 
 ##### clean-housenumbers
 

-::: nominatim.tokenizer.sanitizers.clean_housenumbers
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### clean-postcodes
 
 
 ##### clean-postcodes
 

-::: nominatim.tokenizer.sanitizers.clean_postcodes
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_postcodes
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 ##### clean-tiger-tags
 
 
 ##### clean-tiger-tags
 

-::: nominatim.tokenizer.sanitizers.clean_tiger_tags
-    selection:
+::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 #### delete-tags
 
 
 #### delete-tags
 

-::: nominatim.tokenizer.sanitizers.delete_tags
-    selection:
+::: nominatim_db.tokenizer.sanitizers.delete_tags
+    options:
+        members: False
+        heading_level: 6
+        docstring_section_style: spacy
+
+#### tag-japanese
+
+::: nominatim_db.tokenizer.sanitizers.tag_japanese
+    options:

         members: False
         members: False

-    rendering:

         heading_level: 6
         heading_level: 6

+        docstring_section_style: spacy

 
 #### Token Analysis
 
 
 #### Token Analysis
 


@@ -386,7 +352,7 @@ The analyzer cannot be customized.
 ##### Postcode token analyzer
 
 The analyzer `postcodes` is pupose-made to analyze postcodes. It supports
 ##### Postcode token analyzer
 
 The analyzer `postcodes` is pupose-made to analyze postcodes. It supports

-a 'lookup' varaint of the token, which produces variants with optional
+a 'lookup' variant of the token, which produces variants with optional

 spaces. Use together with the clean-postcodes sanitizer.
 
 The analyzer cannot be customized.
 spaces. Use together with the clean-postcodes sanitizer.
 
 The analyzer cannot be customized.