Merge pull request #3333 from lonvia/set-empty-extratags-to-null

[nominatim.git] / docs / develop / Tokenizers.md

diff --git a/docs/develop/Tokenizers.md b/docs/develop/Tokenizers.md
index 5282db1ae73a1ad375232fedc647df130ad71a9e..050371771c27eb21ffa7158efbc261e8ae35154d 100644 (file)
--- a/docs/develop/Tokenizers.md
+++ b/docs/develop/Tokenizers.md
@@ -6,7 +6,7 @@ tokenizers that use different strategies for normalisation. This page describes
 how tokenizers are expected to work and the public API that needs to be
 implemented when creating a new tokenizer. For information on how to configure
 a specific tokenizer for a database see the
-[tokenizer chapter in the administration guide](../admin/Tokenizers.md).
+[tokenizer chapter in the Customization Guide](../customize/Tokenizers.md).
 
 ## Generic Architecture
 
@@ -93,7 +93,7 @@ for a custom tokenizer implementation.
 
 Nominatim expects two files for a tokenizer:
 
-* `nominiatim/tokenizer/<NAME>_tokenizer.py` containing the Python part of the
+* `nominatim/tokenizer/<NAME>_tokenizer.py` containing the Python part of the
   implementation
 * `lib-php/tokenizer/<NAME>_tokenizer.php` with the PHP part of the
   implementation
@@ -105,7 +105,7 @@ functions. By convention, these should be placed in `lib-sql/tokenizer`.
 If the tokenizer has a default configuration file, this should be saved in
 the `settings/<NAME>_tokenizer.<SUFFIX>`.
 
-### Configuration and Persistance
+### Configuration and Persistence
 
 Tokenizers may define custom settings for their configuration. All settings
 must be prefixed with `NOMINATIM_TOKENIZER_`. Settings may be transient or
@@ -134,14 +134,14 @@ All tokenizers must inherit from `nominatim.tokenizer.base.AbstractTokenizer`
 and implement the abstract functions defined there.
 
 ::: nominatim.tokenizer.base.AbstractTokenizer
-    rendering:
-        heading_level: 4
+    options:
+        heading_level: 6
 
 ### Python Analyzer Class
 
 ::: nominatim.tokenizer.base.AbstractAnalyzer
-    rendering:
-        heading_level: 4
+    options:
+        heading_level: 6
 
 ### PL/pgSQL Functions
 
@@ -189,6 +189,28 @@ a house number token text. If a place has multiple house numbers they must
 be listed with a semicolon as delimiter. Must be NULL when the place has no
 house numbers.
 
+```sql
+FUNCTION token_is_street_address(info JSONB) RETURNS BOOLEAN
+```
+
+Return true if this is an object that should be parented against a street.
+Only relevant for objects with address rank 30.
+
+```sql
+FUNCTION token_has_addr_street(info JSONB) RETURNS BOOLEAN
+```
+
+Return true if there are street names to match against for finding the
+parent of the object.
+
+
+```sql
+FUNCTION token_has_addr_place(info JSONB) RETURNS BOOLEAN
+```
+
+Return true if there are place names to match against for finding the
+parent of the object.
+
 ```sql
 FUNCTION token_matches_street(info JSONB, street_tokens INTEGER[]) RETURNS BOOLEAN
 ```
@@ -245,11 +267,11 @@ Currently, tokenizers are encouraged to make sure that matching works against
 both the search token list and the match token list.
 
 ```sql
-FUNCTION token_normalized_postcode(postcode TEXT) RETURNS TEXT
+FUNCTION token_get_postcode(info JSONB) RETURNS TEXT
 ```
 
-Return the normalized version of the given postcode. This function must return
-the same value as the Python function `AbstractAnalyzer->normalize_postcode()`.
+Return the postcode for the object, if any exists. The postcode must be in
+the form that should also be presented to the end-user.
 
 ```sql
 FUNCTION token_strip_info(info JSONB) RETURNS JSONB