adapt BDD tests to changed simplification

[nominatim.git] / docs / admin / Migration.md

diff --git a/docs/admin/Migration.md b/docs/admin/Migration.md
index 333c24778eef2b577b770d822df01c61eac71166..13e6d7f5e5ce64629a37f58e5344cc774731bfea 100644 (file)
--- a/docs/admin/Migration.md
+++ b/docs/admin/Migration.md
@@ -1,23 +1,124 @@
 # Database Migrations
 
 # Database Migrations
 

-This page describes database migrations necessary to update existing databases
-to newer versions of Nominatim.
+Nominatim offers automatic migrations for versions 4.3+. Please follow
+the following steps:

 
 

-SQL statements should be executed from the PostgreSQL commandline. Execute
-`psql nominatim` to enter command line mode.
+* Stop any updates that are potentially running
+* Update the backend: `pip install -U nominatim-db`
+* Go to your project directory and run `nominatim admin --migrate`
+* Update the frontend: `pip install -U nominatim-api`
+* (optionally) Restart updates

 
 

-## 3.6.0 -> master
+If you are still using CMake for the installation of Nominatim, then you
+need to update the software in one step before migrating the database.
+It is not recommended to do this while the machine is serving requests.

 
 

-### Status table contains now time zone information
+Below you find additional migrations and hints about other structural and
+breaking changes. **Please read them before running the migration.**

 
 

-The `import_status` table has been changed to include timezone information
-with the time stamp. You need to alter an existing table before running
-any replication functions with:
+!!! note
+    If you are migrating from a version <4.3, you need to install 4.3
+    first and migrate to 4.3 first. Then you can migrate to the current
+    version. It is strongly recommended to do a reimport instead.
+
+## 4.5.0 -> master
+
+### PHP frontend removed
+
+The PHP frontend has been completely removed. Please switch to the Python
+frontend.
+
+Without the PHP code, the `nominatim refresh --website` command is no longer
+needed. It currently omits a warning and does otherwise nothing. It will be
+removed in later versions of Nominatim. So make sure you remove it from your
+scripts.
+
+## 4.4.0 -> 4.5.0
+
+### New structure for Python packages
+
+The nominatim Python package has been split into `nominatim-db` and `nominatim-api`.
+Any imports need to be adapted accordingly.
+
+If you are running the Python frontend, change the server module from
+`nominatim.server.falcon.server` to `nominatim_api.server.falcon.server`.
+
+If you are using the Nominatim library, all imports need to be changed
+from `nominatim.api.<module>` to `nominatim_api.<module>`.
+
+If you have written custom tokenizers or sanitizers, the appropriate modules
+are now found in `nominatim_db`.
+
+## 4.2.0 -> 4.3.0
+
+### New indexes for reverse lookup
+
+The reverse lookup algorithm has changed slightly to improve performance.
+This change needs a different index in the database. The required index
+will be automatically build during migration. Until the new index is available
+performance of the /reverse endpoint is significantly reduced. You should
+therefore either remove traffic from the machine before attempting a
+version update or create the index manually **before** starting the update
+using the following SQL:

 
 ```sql
 
 ```sql

-ALTER TABLE import_status ALTER COLUMN lastimportdate TYPE timestamp with time zone;
+CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode
+  ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search)))
+  WHERE rank_address between 4 and 25 AND type != 'postcode'
+    AND name is not null AND linked_place_id is null AND osm_type = 'N';

 ```
 
 ```
 

+## 4.0.0 -> 4.1.0
+
+### ICU tokenizer is the new default
+
+Nominatim now installs the [ICU tokenizer](../customize/Tokenizers.md#icu-tokenizer)
+by default. This only has an effect on newly installed databases. When
+updating older databases, it keeps its installed tokenizer. If you still
+run with the legacy tokenizer, make sure to compile Nominatim with the
+PostgreSQL module, see [Installation](Installation.md#building-nominatim).
+
+### geocodejson output changed
+
+The `type` field of the geocodejson output has changed. It now contains
+the address class of the object instead of the value of the OSM tag. If
+your client has used the `type` field, switch them to read `osm_value`
+instead.
+
+## 3.7.0 -> 4.0.0
+
+### NOMINATIM_PHRASE_CONFIG removed
+
+Custom blacklist configurations for special phrases now need to be handed
+with the `--config` parameter to `nominatim special-phrases`. Alternatively
+you can put your custom configuration in the project directory in a file
+named `phrase-settings.json`.
+
+Version 3.8 also removes the automatic converter for the php format of
+the configuration in older versions. If you are updating from Nominatim < 3.7
+and still work with a custom `phrase-settings.php`, you need to manually
+convert it into a json format.
+
+### PHP utils removed
+
+The old PHP utils have now been removed completely. You need to switch to
+the appropriate functions of the nominatim  command line tool. See
+[Introducing `nominatim` command line tool](#introducing-nominatim-command-line-tool)
+below.
+
+## 3.6.0 -> 3.7.0
+
+### New format and name of configuration file
+
+The configuration for an import is now saved in a `.env` file in the project
+directory. This file follows the dotenv format. For more information, see
+the [installation chapter](Import.md#configuration-setup-in-env).
+
+To migrate to the new system, create a new project directory, add the `.env`
+file and port your custom configuration from `settings/local.php`. Most
+settings are named similar and only have received a `NOMINATIM_` prefix.
+Use the default settings in `settings/env.defaults` as a reference.
+

 ### New location for data files
 
 External data files for Wikipedia importance, postcodes etc. are no longer
 ### New location for data files
 
 External data files for Wikipedia importance, postcodes etc. are no longer


@@ -37,8 +138,8 @@ functionality of each script:
 * ./utils/setup.php: `import`, `freeze`, `refresh`
 * ./utils/update.php: `replication`, `add-data`, `index`, `refresh`
 * ./utils/specialphrases.php: `special-phrases`
 * ./utils/setup.php: `import`, `freeze`, `refresh`
 * ./utils/update.php: `replication`, `add-data`, `index`, `refresh`
 * ./utils/specialphrases.php: `special-phrases`

-* ./utils/check_import_finished.php: `check-database`
-* ./utils/warm.php: `warm`
+* ./utils/check_import_finished.php: `admin`
+* ./utils/warm.php: `admin`

 * ./utils/export.php: `export`
 
 Try `nominatim <command> --help` for more information about each subcommand.
 * ./utils/export.php: `export`
 
 Try `nominatim <command> --help` for more information about each subcommand.


@@ -46,6 +147,12 @@ Try `nominatim <command> --help` for more information about each subcommand.
 `./utils/query.php` no longer exists in its old form. `nominatim search`
 provides a replacement but returns different output.
 
 `./utils/query.php` no longer exists in its old form. `nominatim search`
 provides a replacement but returns different output.
 

+### Switch to normalized house numbers
+
+The housenumber column in the placex table uses now normalized version.
+The automatic migration step will convert the column but this may take a
+very long time. It is advisable to take the machine offline while doing that.
+

 ## 3.5.0 -> 3.6.0
 
 ### Change of layout of search_name_* tables
 ## 3.5.0 -> 3.6.0
 
 ### Change of layout of search_name_* tables


@@ -64,7 +171,7 @@ done
 
 The debugging UI is no longer directly provided with Nominatim. Instead we
 now provide a simple Javascript application. Please refer to
 
 The debugging UI is no longer directly provided with Nominatim. Instead we
 now provide a simple Javascript application. Please refer to

-[Setting up the Nominatim UI](../Setup-Nominatim-UI) for details on how to
+[Setting up the Nominatim UI](Setup-Nominatim-UI.md) for details on how to

 set up the UI.
 
 The icons served together with the API responses have been moved to the
 set up the UI.
 
 The icons served together with the API responses have been moved to the


@@ -108,6 +215,14 @@ configuration file, run the following command after updating:
 ./utils/setup.php --setup-website
 ```
 
 ./utils/setup.php --setup-website
 ```
 

+### Update SQL code
+
+To update the SQL code to the leatest version run:
+
+```
+./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions
+```
+

 ## 3.4.0 -> 3.5.0
 
 ### New Wikipedia/Wikidata importance tables
 ## 3.4.0 -> 3.5.0
 
 ### New Wikipedia/Wikidata importance tables