X-Git-Url: https://git.openstreetmap.org./nominatim.git/blobdiff_plain/c4b8a3b7680ae51da4c7f0ac0849ecd9fe3d5660..baee6f3de09226c3dc41cb2314a0ac348e865561:/docs/develop/Tokenizers.md diff --git a/docs/develop/Tokenizers.md b/docs/develop/Tokenizers.md index 743e637c..2b4da005 100644 --- a/docs/develop/Tokenizers.md +++ b/docs/develop/Tokenizers.md @@ -6,7 +6,7 @@ tokenizers that use different strategies for normalisation. This page describes how tokenizers are expected to work and the public API that needs to be implemented when creating a new tokenizer. For information on how to configure a specific tokenizer for a database see the -[tokenizer chapter in the administration guide](../admin/Tokenizers.md). +[tokenizer chapter in the Customization Guide](../customize/Tokenizers.md). ## Generic Architecture @@ -50,7 +50,7 @@ tokenizer's internal token lists and creating a list of all token IDs for the specific place. This list is later needed in the PL/pgSQL part where the indexer needs to add the token IDs to the appropriate search tables. To be able to communicate the list between the Python part and the pl/pgSQL trigger, -the placex table contains a special JSONB column `token_info` which is there +the `placex` table contains a special JSONB column `token_info` which is there for the exclusive use of the tokenizer. The Python part of the tokenizer returns a structured information about the @@ -67,12 +67,17 @@ consequently not create any special indexes on it. ### Querying -The tokenizer is responsible for the initial parsing of the query. It needs -to split the query into appropriate words and terms and match them against -the saved tokens in the database. It then returns the list of possibly matching -tokens and the list of possible splits to the query parser. The parser uses -this information to compute all possible interpretations of the query and -rank them accordingly. +At query time, Nominatim builds up multiple _interpretations_ of the search +query. Each of these interpretations is tried against the database in order +of the likelihood with which they match to the search query. The first +interpretation that yields results wins. + +The interpretations are encapsulated in the `SearchDescription` class. An +instance of this class is created by applying a sequence of +_search tokens_ to an initially empty SearchDescription. It is the +responsibility of the tokenizer to parse the search query and derive all +possible sequences of search tokens. To that end the tokenizer needs to parse +the search query and look up matching words in its own data structures. ## Tokenizer API @@ -88,7 +93,7 @@ for a custom tokenizer implementation. Nominatim expects two files for a tokenizer: -* `nominiatim/tokenizer/_tokenizer.py` containing the Python part of the +* `nominatim/tokenizer/_tokenizer.py` containing the Python part of the implementation * `lib-php/tokenizer/_tokenizer.php` with the PHP part of the implementation @@ -185,22 +190,21 @@ be listed with a semicolon as delimiter. Must be NULL when the place has no house numbers. ```sql -FUNCTION token_addr_street_match_tokens(info JSONB) RETURNS INTEGER[] +FUNCTION token_matches_street(info JSONB, street_tokens INTEGER[]) RETURNS BOOLEAN ``` -Return the match token IDs by which to search a matching street from the -`addr:street` tag. These IDs will be matched against the IDs supplied by -`token_get_name_match_tokens`. Must be NULL when the place has no `addr:street` -tag. +Check if the given tokens (previously saved from `token_get_name_match_tokens()`) +match against the `addr:street` tag name. Must return either NULL or FALSE +when the place has no `addr:street` tag. ```sql -FUNCTION token_addr_place_match_tokens(info JSONB) RETURNS INTEGER[] +FUNCTION token_matches_place(info JSONB, place_tokens INTEGER[]) RETURNS BOOLEAN ``` -Return the match token IDs by which to search a matching place from the -`addr:place` tag. These IDs will be matched against the IDs supplied by -`token_get_name_match_tokens`. Must be NULL when the place has no `addr:place` -tag. +Check if the given tokens (previously saved from `token_get_name_match_tokens()`) +match against the `addr:place` tag name. Must return either NULL or FALSE +when the place has no `addr:place` tag. + ```sql FUNCTION token_addr_place_search_tokens(info JSONB) RETURNS INTEGER[] @@ -211,26 +215,34 @@ are used for searches by address when no matching place can be found in the database. Must be NULL when the place has no `addr:place` tag. ```sql -CREATE TYPE token_addresstoken AS ( - key TEXT, - match_tokens INT[], - search_tokens INT[] -); +FUNCTION token_get_address_keys(info JSONB) RETURNS SETOF TEXT +``` + +Return the set of keys for which address information is provided. This +should correspond to the list of (relevant) `addr:*` tags with the `addr:` +prefix removed or the keys used in the `address` dictionary of the place info. + +```sql +FUNCTION token_get_address_search_tokens(info JSONB, key TEXT) RETURNS INTEGER[] +``` + +Return the array of search tokens for the given address part. `key` can be +expected to be one of those returned with `token_get_address_keys()`. The +search tokens are added to the address search vector of the place, when no +corresponding OSM object could be found for the given address part from which +to copy the name information. -FUNCTION token_get_address_tokens(info JSONB) RETURNS SETOF token_addresstoken +```sql +FUNCTION token_matches_address(info JSONB, key TEXT, tokens INTEGER[]) ``` -Return the match and search token IDs for explicit `addr:*` tags for the place -other than `addr:street` and `addr:place`. For each address item there are -three pieces of information returned: +Check if the given tokens match against the address part `key`. - * _key_ contains the type of address item (city, county, etc.). This is the - key handed in with the `address` dictionary. - * *match_tokens* is the list of token IDs used to find the corresponding - place object for the address part. The list is matched against the IDs - from `token_get_name_match_tokens`. - * *search_tokens* is the list of token IDs under which to search the address - item. It is used when no corresponding place object was found. +__Warning:__ the tokens that are handed in are the lists previously saved +from `token_get_name_search_tokens()`, _not_ from the match token list. This +is an historical oddity which will be fixed at some point in the future. +Currently, tokenizers are encouraged to make sure that matching works against +both the search token list and the match token list. ```sql FUNCTION token_normalized_postcode(postcode TEXT) RETURNS TEXT @@ -301,6 +313,14 @@ public function extractTokensFromPhrases(array &$aPhrases) : TokenList Parse the given phrases, splitting them into word lists and retrieve the matching tokens. +The phrase array may take on two forms. In unstructured searches (using `q=` +parameter) the search query is split at the commas and the elements are +put into a sorted list. For structured searches the phrase array is an +associative array where the key designates the type of the term (street, city, +county etc.) The tokenizer may ignore the phrase type at this stage in parsing. +Matching phrase type and appropriate search token type will be done later +when the SearchDescription is built. + For each phrase in the list of phrases, the function must analyse the phrase string and then call `setWordSets()` to communicate the result of the analysis. A word set is a list of strings, where each string refers to a search token.