normalize names and queries. It also offers configurable decomposition and
abbreviation handling.
+To enable the tokenizer add the following line to your project configuration:
+
+```
+NOMINATIM_TOKENIZER=icu
+```
+
### How it works
On import the tokenizer processes names in the following four stages:
name:
``` yaml
-- ^south => n # matches only at the beginning of the name
+- ^south => s # matches only at the beginning of the name
- road$ => rd # matches only at the end of the name
```
The simple arrow causes an additional variant to be added. Note that
decomposition has an effect here on the source as well. So a rule
-```yaml
-- ~strasse => str
+``` yaml
+- "~strasse -> str"
```
means that for a word like `hauptstrasse` four variants are created: