## Legacy tokenizer
+!!! danger
+ The Legacy tokenizer is deprecated and will be removed in Nominatim 5.0.
+ If you still use a database with the legacy tokenizer, you must reimport
+ it using the ICU tokenizer below.
+
The legacy tokenizer implements the analysis algorithms of older Nominatim
versions. It uses a special Postgresql module to normalize names and queries.
This tokenizer is automatically installed and used when upgrading an older
```
This is in particular useful when the database runs on a different server.
-See [Advanced installations](../admin/Advanced-Installations.md#importing-nominatim-to-an-external-postgresql-database) for details.
+See [Advanced installations](../admin/Advanced-Installations.md#using-an-external-postgresql-database) for details.
There are no other configuration options for the legacy tokenizer. All
normalization functions are hard-coded.
##### split-name-list
-::: nominatim.tokenizer.sanitizers.split_name_list
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.split_name_list
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
##### strip-brace-terms
-::: nominatim.tokenizer.sanitizers.strip_brace_terms
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
##### tag-analyzer-by-language
-::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
##### clean-housenumbers
-::: nominatim.tokenizer.sanitizers.clean_housenumbers
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
##### clean-postcodes
-::: nominatim.tokenizer.sanitizers.clean_postcodes
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.clean_postcodes
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
##### clean-tiger-tags
-::: nominatim.tokenizer.sanitizers.clean_tiger_tags
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
#### delete-tags
-::: nominatim.tokenizer.sanitizers.delete_tags
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.delete_tags
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
#### tag-japanese
-::: nominatim.tokenizer.sanitizers.tag_japanese
- selection:
- members: False
+::: nominatim_db.tokenizer.sanitizers.tag_japanese
options:
+ members: False
heading_level: 6
docstring_section_style: spacy
##### Postcode token analyzer
The analyzer `postcodes` is pupose-made to analyze postcodes. It supports
-a 'lookup' varaint of the token, which produces variants with optional
+a 'lookup' variant of the token, which produces variants with optional
spaces. Use together with the clean-postcodes sanitizer.
The analyzer cannot be customized.