3 The tokenizer module in Nominatim is responsible for analysing the names given
4 to OSM objects and the terms of an incoming query in order to make sure, they
5 can be matched appropriately.
7 Nominatim offers different tokenizer modules, which behave differently and have
8 different configuration options. This sections describes the tokenizers and how
9 they can be configured.
12 The use of a tokenizer is tied to a database installation. You need to choose
13 and configure the tokenizer before starting the initial import. Once the import
14 is done, you cannot switch to another tokenizer anymore. Reconfiguring the
15 chosen tokenizer is very limited as well. See the comments in each tokenizer
21 The Legacy tokenizer is deprecated and will be removed in Nominatim 5.0.
22 If you still use a database with the legacy tokenizer, you must reimport
23 it using the ICU tokenizer below.
25 The legacy tokenizer implements the analysis algorithms of older Nominatim
26 versions. It uses a special Postgresql module to normalize names and queries.
27 This tokenizer is automatically installed and used when upgrading an older
28 database. It should not be used for new installations anymore.
30 ### Compiling the PostgreSQL module
32 The tokeinzer needs a special C module for PostgreSQL which is not compiled
33 by default. If you need the legacy tokenizer, compile Nominatim as follows:
38 cmake -DBUILD_MODULE=on
42 ### Enabling the tokenizer
44 To enable the tokenizer add the following line to your project configuration:
47 NOMINATIM_TOKENIZER=legacy
50 The Postgresql module for the tokenizer is available in the `module` directory
51 and also installed with the remainder of the software under
52 `lib/nominatim/module/nominatim.so`. You can specify a custom location for
56 NOMINATIM_DATABASE_MODULE_PATH=<path to directory where nominatim.so resides>
59 This is in particular useful when the database runs on a different server.
60 See [Advanced installations](../admin/Advanced-Installations.md#using-an-external-postgresql-database) for details.
62 There are no other configuration options for the legacy tokenizer. All
63 normalization functions are hard-coded.
67 The ICU tokenizer uses the [ICU library](http://site.icu-project.org/) to
68 normalize names and queries. It also offers configurable decomposition and
69 abbreviation handling.
70 This tokenizer is currently the default.
72 To enable the tokenizer add the following line to your project configuration:
75 NOMINATIM_TOKENIZER=icu
80 On import the tokenizer processes names in the following three stages:
82 1. During the **Sanitizer step** incoming names are cleaned up and converted to
83 **full names**. This step can be used to regularize spelling, split multi-name
84 tags into their parts and tag names with additional attributes. See the
85 [Sanitizers section](#sanitizers) below for available cleaning routines.
86 2. The **Normalization** part removes all information from the full names
87 that are not relevant for search.
88 3. The **Token analysis** step takes the normalized full names and creates
89 all transliterated variants under which the name should be searchable.
90 See the [Token analysis](#token-analysis) section below for more
93 During query time, only normalization and transliteration are relevant.
94 An incoming query is first split into name chunks (this usually means splitting
95 the string at the commas) and the each part is normalised and transliterated.
96 The result is used to look up places in the search index.
100 The ICU tokenizer is configured using a YAML file which can be configured using
101 `NOMINATIM_TOKENIZER_CONFIG`. The configuration is read on import and then
102 saved as part of the internal database status. Later changes to the variable
105 Here is an example configuration file:
110 - "ß > 'ss'" # German szet is unambiguously equal to double ss
112 - !include /etc/nominatim/icu-rules/extended-unicode-to-asccii.yaml
115 - step: split-name-list
119 - !include icu-rules/variants-ca.yaml
122 - bridge -> bdge,br,brdg,bri,brg
125 replacements: ['ä', 'ae']
128 The configuration file contains four sections:
129 `normalization`, `transliteration`, `sanitizers` and `token-analysis`.
131 #### Normalization and Transliteration
133 The normalization and transliteration sections each define a set of
134 ICU rules that are applied to the names.
136 The **normalization** rules are applied after sanitation. They should remove
137 any information that is not relevant for search at all. Usual rules to be
138 applied here are: lower-casing, removing of special characters, cleanup of
141 The **transliteration** rules are applied at the end of the tokenization
142 process to transfer the name into an ASCII representation. Transliteration can
143 be useful to allow for further fuzzy matching, especially between different
146 Each section must contain a list of
147 [ICU transformation rules](https://unicode-org.github.io/icu/userguide/transforms/general/rules.html).
148 The rules are applied in the order in which they appear in the file.
149 You can also include additional rules from external yaml file using the
150 `!include` tag. The included file must contain a valid YAML list of ICU rules
151 and may again include other files.
154 The ICU rule syntax contains special characters that conflict with the
155 YAML syntax. You should therefore always enclose the ICU rules in
160 The sanitizers section defines an ordered list of functions that are applied
161 to the name and address tags before they are further processed by the tokenizer.
162 They allows to clean up the tagging and bring it to a standardized form more
163 suitable for building the search index.
166 Sanitizers only have an effect on how the search index is built. They
167 do not change the information about each place that is saved in the
168 database. In particular, they have no influence on how the results are
169 displayed. The returned results always show the original information as
170 stored in the OpenStreetMap database.
172 Each entry contains information of a sanitizer to be applied. It has a
173 mandatory parameter `step` which gives the name of the sanitizer. Depending
174 on the type, it may have additional parameters to configure its operation.
176 The order of the list matters. The sanitizers are applied exactly in the order
177 that is configured. Each sanitizer works on the results of the previous one.
179 The following is a list of sanitizers that are shipped with Nominatim.
181 ##### split-name-list
183 ::: nominatim_db.tokenizer.sanitizers.split_name_list
187 docstring_section_style: spacy
189 ##### strip-brace-terms
191 ::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
195 docstring_section_style: spacy
197 ##### tag-analyzer-by-language
199 ::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
203 docstring_section_style: spacy
205 ##### clean-housenumbers
207 ::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
211 docstring_section_style: spacy
213 ##### clean-postcodes
215 ::: nominatim_db.tokenizer.sanitizers.clean_postcodes
219 docstring_section_style: spacy
221 ##### clean-tiger-tags
223 ::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
227 docstring_section_style: spacy
231 ::: nominatim_db.tokenizer.sanitizers.delete_tags
235 docstring_section_style: spacy
239 ::: nominatim_db.tokenizer.sanitizers.tag_japanese
243 docstring_section_style: spacy
247 Token analyzers take a full name and transform it into one or more normalized
248 form that are then saved in the search index. In its simplest form, the
249 analyzer only applies the transliteration rules. More complex analyzers
250 create additional spelling variants of a name. This is useful to handle
251 decomposition and abbreviation.
253 The ICU tokenizer may use different analyzers for different names. To select
254 the analyzer to be used, the name must be tagged with the `analyzer` attribute
255 by a sanitizer (see for example the
256 [tag-analyzer-by-language sanitizer](#tag-analyzer-by-language)).
258 The token-analysis section contains the list of configured analyzers. Each
259 analyzer must have an `id` parameter that uniquely identifies the analyzer.
260 The only exception is the default analyzer that is used when no special
261 analyzer was selected. There are analysers with special ids:
263 * '@housenumber'. If an analyzer with that name is present, it is used
264 for normalization of house numbers.
265 * '@potcode'. If an analyzer with that name is present, it is used
266 for normalization of postcodes.
268 Different analyzer implementations may exist. To select the implementation,
269 the `analyzer` parameter must be set. The different implementations are
270 described in the following.
272 ##### Generic token analyzer
274 The generic analyzer `generic` is able to create variants from a list of given
275 abbreviation and decomposition replacements and introduce spelling variations.
279 The optional 'variants' section defines lists of replacements which create alternative
280 spellings of a name. To create the variants, a name is scanned from left to
281 right and the longest matching replacement is applied until the end of the
284 The variants section must contain a list of replacement groups. Each group
285 defines a set of properties that describes where the replacements are
286 applicable. In addition, the word section defines the list of replacements
287 to be made. The basic replacement description is of the form:
290 <source>[,<source>[...]] => <target>[,<target>[...]]
293 The left side contains one or more `source` terms to be replaced. The right side
294 lists one or more replacements. Each source is replaced with each replacement
298 The source and target terms are internally normalized using the
299 normalization rules given in the configuration. This ensures that the
300 strings match as expected. In fact, it is better to use unnormalized
301 words in the configuration because then it is possible to change the
302 rules for normalization later without having to adapt the variant rules.
306 In its standard form, only full words match against the source. There
307 is a special notation to match the prefix and suffix of a word:
310 - ~strasse => str # matches "strasse" as full word and in suffix position
311 - hinter~ => hntr # matches "hinter" as full word and in prefix position
314 There is no facility to match a string in the middle of the word. The suffix
315 and prefix notation automatically trigger the decomposition mode: two variants
316 are created for each replacement, one with the replacement attached to the word
317 and one separate. So in above example, the tokenization of "hauptstrasse" will
318 create the variants "hauptstr" and "haupt str". Similarly, the name "rote strasse"
319 triggers the variants "rote str" and "rotestr". By having decomposition work
320 both ways, it is sufficient to create the variants at index time. The variant
321 rules are not applied at query time.
323 To avoid automatic decomposition, use the '|' notation:
329 simply changes "hauptstrasse" to "hauptstr" and "rote strasse" to "rote str".
331 ###### Initial and final terms
333 It is also possible to restrict replacements to the beginning and end of a
337 - ^south => s # matches only at the beginning of the name
338 - road$ => rd # matches only at the end of the name
341 So the first example would trigger a replacement for "south 45th street" but
342 not for "the south beach restaurant".
344 ###### Replacements vs. variants
346 The replacement syntax `source => target` works as a pure replacement. It changes
347 the name instead of creating a variant. To create an additional version, you'd
348 have to write `source => source,target`. As this is a frequent case, there is
349 a shortcut notation for it:
352 <source>[,<source>[...]] -> <target>[,<target>[...]]
355 The simple arrow causes an additional variant to be added. Note that
356 decomposition has an effect here on the source as well. So a rule
362 means that for a word like `hauptstrasse` four variants are created:
363 `hauptstrasse`, `haupt strasse`, `hauptstr` and `haupt str`.
367 The 'mutation' section in the configuration describes an additional set of
368 replacements to be applied after the variants have been computed.
370 Each mutation is described by two parameters: `pattern` and `replacements`.
371 The pattern must contain a single regular expression to search for in the
372 variant name. The regular expressions need to follow the syntax for
373 [Python regular expressions](file:///usr/share/doc/python3-doc/html/library/re.html#regular-expression-syntax).
374 Capturing groups are not permitted.
375 `replacements` must contain a list of strings that the pattern
376 should be replaced with. Each occurrence of the pattern is replaced with
377 all given replacements. Be mindful of combinatorial explosion of variants.
381 The generic analyser supports a special mode `variant-only`. When configured
382 then it consumes the input token and emits only variants (if any exist). Enable
389 to the analyser configuration.
391 ##### Housenumber token analyzer
393 The analyzer `housenumbers` is purpose-made to analyze house numbers. It
394 creates variants with optional spaces between numbers and letters. Thus,
395 house numbers of the form '3 a', '3A', '3-A' etc. are all considered equivalent.
397 The analyzer cannot be customized.
399 ##### Postcode token analyzer
401 The analyzer `postcodes` is pupose-made to analyze postcodes. It supports
402 a 'lookup' variant of the token, which produces variants with optional
403 spaces. Use together with the clean-postcodes sanitizer.
405 The analyzer cannot be customized.
409 Changing the configuration after the import is currently not possible, although
410 this feature may be added at a later time.