docs: querying and tokenizers

diff --git a/docs/develop/Tokenizers.md b/docs/develop/Tokenizers.md
index 7d54247f6c6f2e0de25f013365d459e51738e36b..e10587a627480afc71f95d5a1601e1a2841df5e2 100644 (file)
--- a/docs/develop/Tokenizers.md
+++ b/docs/develop/Tokenizers.md
@@ -67,4 +67,9 @@ consequently not create any special indexes on it.
 
 ### Querying
 
-
+The tokenizer is responsible for the initial parsing of the query. It needs
+to split the query into appropriate words and terms and match them against
+the saved tokens in the database. It then returns the list of possibly matching
+tokens and the list of possible splits to the query parser. The parser uses
+this information to compute all possible interpretations of the query and
+rank them accordingly.