From: Sarah Hoffmann Date: Sun, 31 Jul 2022 15:10:35 +0000 (+0200) Subject: fix various typos X-Git-Tag: v4.1.0~4^2 X-Git-Url: https://git.openstreetmap.org./nominatim.git/commitdiff_plain/9864b191b134a4681b5e21e23ca9e00c69db62db?hp=--cc fix various typos --- 9864b191b134a4681b5e21e23ca9e00c69db62db diff --git a/docs/develop/ICU-Tokenizer-Modules.md b/docs/develop/ICU-Tokenizer-Modules.md index 2427ab11..2cf30a56 100644 --- a/docs/develop/ICU-Tokenizer-Modules.md +++ b/docs/develop/ICU-Tokenizer-Modules.md @@ -57,9 +57,9 @@ the function. show_source: no heading_level: 6 -### The sanitation function +### The main filter function of the sanitizer -The sanitation function receives a single object of type `ProcessInfo` +The filter function receives a single object of type `ProcessInfo` which has with three members: * `place`: read-only information about the place being processed. @@ -74,6 +74,22 @@ While the `place` member is provided for information only, the `names` and remove entries, change information within a single entry (for example by adding extra attributes) or completely replace the list with a different one. +#### PlaceInfo - information about the place + +::: nominatim.data.place_info.PlaceInfo + rendering: + show_source: no + heading_level: 6 + + +#### PlaceName - extended naming information + +::: nominatim.data.place_name.PlaceName + rendering: + show_source: no + heading_level: 6 + + ### Example: Filter for US street prefixes The following sanitizer removes the directional prefixes from street names @@ -102,49 +118,32 @@ the filter. The filter function first checks if the object is interesting for the sanitizer. Namely it checks if the place is in the US (through `country_code`) and it the place is a street (a `rank_address` of 26 or 27). If the -conditions are met, then it goes through all available names and replaces -any removes any leading direction prefix using a simple regular expression. +conditions are met, then it goes through all available names and +removes any leading directional prefix using a simple regular expression. Save the source code in a file in your project directory, for example as `us_streets.py`. Then you can use the sanitizer in your `icu_tokenizer.yaml`: -``` +``` yaml ... sanitizers: - step: us_streets.py ... ``` -For more sanitizer examples, have a look at the sanitizers provided by Nominatim. -They can be found in the directory `nominatim/tokenizer/sanitizers`. - !!! warning This example is just a simplified show case on how to create a sanitizer. It is not really read for real-world use: while the sanitizer would correcly transform `West 5th Street` into `5th Street`. it would also shorten a simple `North Street` to `Street`. -#### PlaceInfo - information about the place - -::: nominatim.data.place_info.PlaceInfo - rendering: - show_source: no - heading_level: 6 - - -#### PlaceName - extended naming information +For more sanitizer examples, have a look at the sanitizers provided by Nominatim. +They can be found in the directory +[`nominatim/tokenizer/sanitizers`](https://github.com/osm-search/Nominatim/tree/master/nominatim/tokenizer/sanitizers). -::: nominatim.data.place_name.PlaceName - rendering: - show_source: no - heading_level: 6 ## Custom token analysis module -Setup of a token analyser is split into two parts: configuration and -analyser factory. A token analysis module must therefore implement two -functions: - ::: nominatim.tokenizer.token_analysis.base.AnalysisModule rendering: show_source: no diff --git a/nominatim/data/place_name.py b/nominatim/data/place_name.py index 4dfaf819..f4c5e0fa 100644 --- a/nominatim/data/place_name.py +++ b/nominatim/data/place_name.py @@ -20,7 +20,7 @@ class PlaceName: is the part of the key after the first colon. In addition to that, a name may have arbitrary additional attributes. - How attributes are used, depends on the sanatizers and token analysers. + How attributes are used, depends on the sanitizers and token analysers. The exception is is the 'analyzer' attribute. This attribute determines which token analysis module will be used to finalize the treatment of names. diff --git a/nominatim/tokenizer/sanitizers/config.py b/nominatim/tokenizer/sanitizers/config.py index 52cb2c04..8b9164c6 100644 --- a/nominatim/tokenizer/sanitizers/config.py +++ b/nominatim/tokenizer/sanitizers/config.py @@ -23,8 +23,8 @@ else: class SanitizerConfig(_BaseUserDict): """ The `SanitizerConfig` class is a read-only dictionary with configuration options for the sanitizer. - In addition to the usual dictionary function, the class provides - accessors to standard sanatizer options that are used by many of the + In addition to the usual dictionary functions, the class provides + accessors to standard sanitizer options that are used by many of the sanitizers. """ @@ -81,15 +81,15 @@ class SanitizerConfig(_BaseUserDict): def get_delimiter(self, default: str = ',;') -> Pattern[str]: """ Return the 'delimiters' parameter in the configuration as a - compiled regular expression that can be used to split names on these - delimiters. + compiled regular expression that can be used to split strings on + these delimiters. Arguments: - default: Delimiters to be used, when 'delimiters' parameter + default: Delimiters to be used when 'delimiters' parameter is not explicitly configured. Returns: - A regular expression pattern, which can be used to + A regular expression pattern which can be used to split a string. The regular expression makes sure that the resulting names are stripped and that repeated delimiters are ignored. It may still create empty fields on occasion. The diff --git a/nominatim/tokenizer/token_analysis/base.py b/nominatim/tokenizer/token_analysis/base.py index cbd445c8..68046f96 100644 --- a/nominatim/tokenizer/token_analysis/base.py +++ b/nominatim/tokenizer/token_analysis/base.py @@ -44,15 +44,18 @@ class Analyzer(Protocol): A list of possible spelling variants. All strings must have been transformed with the global normalizer and transliterator ICU rules. Otherwise they cannot be matched - against the query later. + against the input by the query frontend. The list may be empty, when there are no useful - spelling variants. This may happen, when an analyzer only - produces extra variants to the canonical spelling. + spelling variants. This may happen when an analyzer only + usually outputs additional variants to the canonical spelling + and there are no such variants. """ class AnalysisModule(Protocol): - """ Protocol for analysis modules. + """ The setup of the token analysis is split into two parts: + configuration and analyser factory. A token analysis module must + therefore implement the two functions here described. """ def configure(self, rules: Mapping[str, Any], @@ -64,13 +67,14 @@ class AnalysisModule(Protocol): Arguments: rules: A dictionary with the additional configuration options as specified in the tokenizer configuration. - normalizer: an ICU Transliterator with the compiled normalization - rules. - transliterator: an ICU transliterator with the compiled - transliteration rules. + normalizer: an ICU Transliterator with the compiled + global normalization rules. + transliterator: an ICU Transliterator with the compiled + global transliteration rules. Returns: - A data object with the configuration that was set up. May be + A data object with configuration data. This will be handed + as is into the `create()` function and may be used freely by the analysis module as needed. """ @@ -82,7 +86,7 @@ class AnalysisModule(Protocol): Arguments: normalizer: an ICU Transliterator with the compiled normalization rules. - transliterator: an ICU tranliterator with the compiled + transliterator: an ICU Transliterator with the compiled transliteration rules. config: The object that was returned by the call to configure().