From: Sarah Hoffmann Date: Sat, 31 Jul 2021 07:49:29 +0000 (+0200) Subject: docs: querying and tokenizers X-Git-Tag: v4.0.0~41^2~7 X-Git-Url: https://git.openstreetmap.org./nominatim.git/commitdiff_plain/e25e268e2e730a81e0bb9e4528947fdc86ca56dd?ds=sidebyside docs: querying and tokenizers --- diff --git a/docs/develop/Tokenizers.md b/docs/develop/Tokenizers.md index 7d54247f..e10587a6 100644 --- a/docs/develop/Tokenizers.md +++ b/docs/develop/Tokenizers.md @@ -67,4 +67,9 @@ consequently not create any special indexes on it. ### Querying - +The tokenizer is responsible for the initial parsing of the query. It needs +to split the query into appropriate words and terms and match them against +the saved tokens in the database. It then returns the list of possibly matching +tokens and the list of possible splits to the query parser. The parser uses +this information to compute all possible interpretations of the query and +rank them accordingly.