From e25e268e2e730a81e0bb9e4528947fdc86ca56dd Mon Sep 17 00:00:00 2001 From: Sarah Hoffmann Date: Sat, 31 Jul 2021 09:49:29 +0200 Subject: [PATCH] docs: querying and tokenizers --- docs/develop/Tokenizers.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/develop/Tokenizers.md b/docs/develop/Tokenizers.md index 7d54247f..e10587a6 100644 --- a/docs/develop/Tokenizers.md +++ b/docs/develop/Tokenizers.md @@ -67,4 +67,9 @@ consequently not create any special indexes on it. ### Querying - +The tokenizer is responsible for the initial parsing of the query. It needs +to split the query into appropriate words and terms and match them against +the saved tokens in the database. It then returns the list of possibly matching +tokens and the list of possible splits to the query parser. The parser uses +this information to compute all possible interpretations of the query and +rank them accordingly. -- 2.39.5