adapt documentation for PhraseType type

[nominatim.git] / docs / develop / ICU-Tokenizer-Modules.md

diff --git a/docs/develop/ICU-Tokenizer-Modules.md b/docs/develop/ICU-Tokenizer-Modules.md
index 2cf30a5699f7863db7db1f7eb04f7fa2ae3bf1b5..30b85ac74fb6595163f48cf277a0954b820beb2b 100644 (file)
--- a/docs/develop/ICU-Tokenizer-Modules.md
+++ b/docs/develop/ICU-Tokenizer-Modules.md
@@ -14,10 +14,11 @@ of sanitizers and token analysis.
     implemented, it is not guaranteed to be stable at the moment.
 
 
     implemented, it is not guaranteed to be stable at the moment.
 
 

-## Using non-standard sanitizers and token analyzers
+## Using non-standard modules

 
 

-Sanitizer names (in the `step` property) and token analysis names (in the
-`analyzer`) may refer to externally supplied modules. There are two ways
+Sanitizer names (in the `step` property), token analysis names (in the
+`analyzer`) and query preprocessor names (in the `step` property)
+may refer to externally supplied modules. There are two ways

 to include external modules: through a library or from the project directory.
 
 To include a module from a library, use the absolute import path as name and
 to include external modules: through a library or from the project directory.
 
 To include a module from a library, use the absolute import path as name and


@@ -27,6 +28,53 @@ To use a custom module without creating a library, you can put the module
 somewhere in your project directory and then use the relative path to the
 file. Include the whole name of the file including the `.py` ending.
 
 somewhere in your project directory and then use the relative path to the
 file. Include the whole name of the file including the `.py` ending.
 

+## Custom query preprocessors
+
+A query preprocessor must export a single factory function `create` with
+the following signature:
+
+``` python
+create(self, config: QueryConfig) -> Callable[[list[Phrase]], list[Phrase]]
+```
+
+The function receives the custom configuration for the preprocessor and
+returns a callable (function or class) with the actual preprocessing
+code. When a query comes in, then the callable gets a list of phrases
+and needs to return the transformed list of phrases. The list and phrases
+may be changed in place or a completely new list may be generated.
+
+The `QueryConfig` is a simple dictionary which contains all configuration
+options given in the yaml configuration of the ICU tokenizer. It is up to
+the function to interpret the values.
+
+A `nominatim_api.search.Phrase` describes a part of the query that contains one or more independent
+search terms. Breaking a query into phrases helps reducing the number of
+possible tokens Nominatim has to take into account. However a phrase break
+is definitive: a multi-term search word cannot go over a phrase break.
+A Phrase object has two fields:
+
+ * `ptype` further refines the type of phrase (see list below)
+ * `text` contains the query text for the phrase
+
+The order of phrases matters to Nominatim when doing further processing.
+Thus, while you may split or join phrases, you should not reorder them
+unless you really know what you are doing.
+
+Phrase types can further help narrowing down how the tokens in the phrase
+are interpreted. The following phrase types are known:
+
+| Name           | Description |
+|----------------|-------------|
+| PHRASE_ANY     | No specific designation (i.e. source is free-form query) |
+| PHRASE_AMENITY | Contains name or type of a POI |
+| PHRASE_STREET  | Contains a street name optionally with a housenumber |
+| PHRASE_CITY    | Contains the postal city |
+| PHRASE_COUNTY  | Contains the equivalent of a county |
+| PHRASE_STATE   | Contains a state or province |
+| PHRASE_POSTCODE| Contains a postal code |
+| PHRASE_COUNTRY | Contains the country name or code |
+
+

 ## Custom sanitizer modules
 
 A sanitizer module must export a single factory function `create` with the
 ## Custom sanitizer modules
 
 A sanitizer module must export a single factory function `create` with the


@@ -52,9 +100,8 @@ the function.
 
 ### Sanitizer configuration
 
 
 ### Sanitizer configuration
 

-::: nominatim.tokenizer.sanitizers.config.SanitizerConfig
-    rendering:
-        show_source: no
+::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig
+    options:

         heading_level: 6
 
 ### The main filter function of the sanitizer
         heading_level: 6
 
 ### The main filter function of the sanitizer


@@ -62,12 +109,10 @@ the function.
 The filter function receives a single object of type `ProcessInfo`
 which has with three members:
 
 The filter function receives a single object of type `ProcessInfo`
 which has with three members:
 

- * `place`: read-only information about the place being processed.
+ * `place: PlaceInfo`: read-only information about the place being processed.

    See PlaceInfo below.
    See PlaceInfo below.

- * `names`: The current list of names for the place. Each name is a
-   PlaceName object.
- * `address`: The current list of address names for the place. Each name
-   is a PlaceName object.
+ * `names: List[PlaceName]`: The current list of names for the place.
+ * `address: List[PlaceName]`: The current list of address names for the place.

 
 While the `place` member is provided for information only, the `names` and
 `address` lists are meant to be manipulated by the sanitizer. It may add and
 
 While the `place` member is provided for information only, the `names` and
 `address` lists are meant to be manipulated by the sanitizer. It may add and


@@ -76,17 +121,15 @@ adding extra attributes) or completely replace the list with a different one.
 
 #### PlaceInfo - information about the place
 
 
 #### PlaceInfo - information about the place
 

-::: nominatim.data.place_info.PlaceInfo
-    rendering:
-        show_source: no
+::: nominatim_db.data.place_info.PlaceInfo
+    options:

         heading_level: 6
 
 
 #### PlaceName - extended naming information
 
         heading_level: 6
 
 
 #### PlaceName - extended naming information
 

-::: nominatim.data.place_name.PlaceName
-    rendering:
-        show_source: no
+::: nominatim_db.data.place_name.PlaceName
+    options:

         heading_level: 6
 
 
         heading_level: 6
 
 


@@ -95,21 +138,22 @@ adding extra attributes) or completely replace the list with a different one.
 The following sanitizer removes the directional prefixes from street names
 in the US:
 
 The following sanitizer removes the directional prefixes from street names
 in the US:
 

-``` python
-import re
-
-def _filter_function(obj):
-    if obj.place.country_code == 'us' \
-       and obj.place.rank_address >= 26 and obj.place.rank_address <= 27:
-        for name in obj.names:
-            name.name = re.sub(r'^(north|south|west|east) ',
-                               '',
-                               name.name,
-                               flags=re.IGNORECASE)
-
-def create(config):
-    return _filter_function
-```
+!!! example
+    ``` python
+    import re
+
+    def _filter_function(obj):
+        if obj.place.country_code == 'us' \
+           and obj.place.rank_address >= 26 and obj.place.rank_address <= 27:
+            for name in obj.names:
+                name.name = re.sub(r'^(north|south|west|east) ',
+                                   '',
+                                   name.name,
+                                   flags=re.IGNORECASE)
+
+    def create(config):
+        return _filter_function
+    ```

 
 This is the most simple form of a sanitizer module. If defines a single
 filter function and implements the required `create()` function by returning
 
 This is the most simple form of a sanitizer module. If defines a single
 filter function and implements the required `create()` function by returning


@@ -133,26 +177,24 @@ sanitizers:
 
 !!! warning
     This example is just a simplified show case on how to create a sanitizer.
 
 !!! warning
     This example is just a simplified show case on how to create a sanitizer.

-    It is not really read for real-world use: while the sanitizer would
-    correcly transform `West 5th Street` into `5th Street`. it would also
+    It is not really meant for real-world use: while the sanitizer would
+    correctly transform `West 5th Street` into `5th Street`. it would also

     shorten a simple `North Street` to `Street`.
 
 For more sanitizer examples, have a look at the sanitizers provided by Nominatim.
 They can be found in the directory
     shorten a simple `North Street` to `Street`.
 
 For more sanitizer examples, have a look at the sanitizers provided by Nominatim.
 They can be found in the directory

-[`nominatim/tokenizer/sanitizers`](https://github.com/osm-search/Nominatim/tree/master/nominatim/tokenizer/sanitizers).
+[`src/nominatim_db/tokenizer/sanitizers`](https://github.com/osm-search/Nominatim/tree/master/src/nominatim_db/tokenizer/sanitizers).

 
 
 ## Custom token analysis module
 
 
 
 ## Custom token analysis module
 

-::: nominatim.tokenizer.token_analysis.base.AnalysisModule
-    rendering:
-        show_source: no
+::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule
+    options:

         heading_level: 6
 
 
         heading_level: 6
 
 

-::: nominatim.tokenizer.token_analysis.base.Analyzer
-    rendering:
-        show_source: no
+::: nominatim_db.tokenizer.token_analysis.base.Analyzer
+    options:

         heading_level: 6
 
 ### Example: Creating acronym variants for long names
         heading_level: 6
 
 ### Example: Creating acronym variants for long names