contract duplicate spaces in transliteration string

There are some pathological cases where an isolated letter may
be deleted because it is in itself meaningless. If this happens in
the middle of a sentence, then the transliteration contains two
consecutive spaces. Add a final rule to fix this.

See #2909.

diff --git a/settings/icu_tokenizer.yaml b/settings/icu_tokenizer.yaml
index 16339970c2ae5727cf606fe686a607814cfec2a2..f30578a2322859ce287915a4049092f50eb3057a 100644 (file)
--- a/settings/icu_tokenizer.yaml
+++ b/settings/icu_tokenizer.yaml
@@ -24,6 +24,7 @@ transliteration:
     - ":: lower ()"
     - "[^a-z0-9[:Space:]] >"
     - ":: NFC ()"
     - ":: lower ()"
     - "[^a-z0-9[:Space:]] >"
     - ":: NFC ()"

+    - "[:Space:]+ > ' '"

 sanitizers:
     - step: clean-housenumbers
       filter-kind:
 sanitizers:
     - step: clean-housenumbers
       filter-kind: