From: Sarah Hoffmann <lonvia@denofr.de>
Date: Mon, 10 Jan 2022 16:40:03 +0000 (+0100)
Subject: consider "modifier letter apostrophe" to be punctuation
X-Git-Tag: v4.1.0~99^2
X-Git-Url: https://git.openstreetmap.org./nominatim.git/commitdiff_plain/fb54bd3fcff5a8a44ae59f0d552ed316b08966b3

consider "modifier letter apostrophe" to be punctuation

While technically being a letter, the apostrophe is often replaced
with a normal apostrophe in writing which is a punctuation mark.
This makes sure that the modifier letter apostrophe yields the same
normalization results and thus is really interchangable.

Only has an effect after the next reimport.

Fixes #2569.
---

diff --git a/settings/icu_tokenizer.yaml b/settings/icu_tokenizer.yaml
index e5cbeb6f..a3c62e67 100644
--- a/settings/icu_tokenizer.yaml
+++ b/settings/icu_tokenizer.yaml
@@ -7,7 +7,7 @@ normalization:
     - "'nº' > 'no'"
     - "ª > a"
     - "º > o"
-    - "[[:Punctuation:][:Symbol:]]  > ' '"
+    - "[[:Punctuation:][:Symbol:]\u02bc]  > ' '"
     - "ß > 'ss'" # German szet is unimbigiously equal to double ss
     - "[^[:Letter:] [:Number:] [:Space:]] >"
     - "[:Lm:] >"