add documentation for new extra field

[nominatim.git] / docs / admin / Migration.md

diff --git a/docs/admin/Migration.md b/docs/admin/Migration.md
index 9b6395e6a3e278ca1a31c6f582f8d4d7bb564c5d..becf2f7b4a94b8a82b74ff9d49c37c778b88163a 100644 (file)
--- a/docs/admin/Migration.md
+++ b/docs/admin/Migration.md
@@ -1,17 +1,196 @@
 # Database Migrations
 
 # Database Migrations
 

-This page describes database migrations necessary to update existing databases
-to newer versions of Nominatim.
+Nominatim offers automatic migrations for versions 4.3+. Please follow
+the following steps:

 
 

-SQL statements should be executed from the PostgreSQL commandline. Execute
-`psql nominatim` to enter command line mode.
+* Stop any updates that are potentially running
+* Update the backend: `pip install -U nominatim-db`
+* Go to your project directory and run `nominatim admin --migrate`
+* Update the frontend: `pip install -U nominatim-api`
+* (optionally) Restart updates

 
 

-## 3.5.0 -> master
+Below you find additional migrations and hints about other structural and
+breaking changes. **Please read them before running the migration.**
+
+!!! note
+    If you are migrating from a version <4.3, you need to install 4.3
+    and migrate to 4.3 first. Then you can migrate to the current
+    version. It is strongly recommended to do a reimport instead.
+
+## 4.5.0 -> 5.0.0
+
+### PHP frontend removed
+
+The PHP frontend has been completely removed. Please switch to the Python
+frontend.
+
+Without the PHP code, the `nominatim refresh --website` command is no longer
+needed. It currently omits a warning and does otherwise nothing. It will be
+removed in later versions of Nominatim. So make sure you remove it from your
+scripts.
+
+### CMake building removed
+
+Nominatim can now only be installed via pip. Please follow the installation
+instructions for the current version to change to pip.
+
+### osm2pgsql no longer vendored in
+
+Nominatim no longer ships its own version of osm2pgsql. Please install a
+stock version of osm2pgsql from your distribution. See the
+[installation instruction for osm2pgsql](https://osm2pgsql.org/doc/install.html)
+for details. A minimum version of 1.8 is required. The current stable versions
+of Ubuntu and Debian already ship with an appropriate versions. For older
+installation, you may have to compile a newer osm2pgsql yourself.
+
+### Legacy tokenizer removed
+
+The `legacy` tokenizer is no longer enabled. This tokenizer has been superseded
+by the `ICU` tokenizer a long time ago. In the unlikely case that your database
+still uses the `legacy` tokenizer, you must reimport your database.
+
+### osm2pgsql style overhauled
+
+There are some fundamental changes to how customized osm2pgsql styles should
+be written. The changes are mostly backwards compatible, i.e. custom styles
+should still work with the new implementation. The only exception is a
+customization of the `process_tags()` function. This function is no longer
+considered public and neither are the helper functions used in it.
+They currently still work but will be removed at some point. If you have
+been making changes to `process_tags`, please review your style and try
+to switch to the new convenience functions.
+
+For more information on the changes, see the
+[pull request](https://github.com/osm-search/Nominatim/pull/3615)
+and read the new
+[customization documentation](https://nominatim.org/release-docs/latest/customize/Import-Styles/).
+
+## 4.4.0 -> 4.5.0
+
+### New structure for Python packages
+
+The nominatim Python package has been split into `nominatim-db` and `nominatim-api`.
+Any imports need to be adapted accordingly.
+
+If you are running the Python frontend, change the server module from
+`nominatim.server.falcon.server` to `nominatim_api.server.falcon.server`.
+
+If you are using the Nominatim library, all imports need to be changed
+from `nominatim.api.<module>` to `nominatim_api.<module>`.
+
+If you have written custom tokenizers or sanitizers, the appropriate modules
+are now found in `nominatim_db`.
+
+## 4.2.0 -> 4.3.0
+
+### New indexes for reverse lookup
+
+The reverse lookup algorithm has changed slightly to improve performance.
+This change needs a different index in the database. The required index
+will be automatically build during migration. Until the new index is available
+performance of the /reverse endpoint is significantly reduced. You should
+therefore either remove traffic from the machine before attempting a
+version update or create the index manually **before** starting the update
+using the following SQL:
+
+```sql
+CREATE INDEX IF NOT EXISTS idx_placex_geometry_reverse_lookupPlaceNode
+  ON placex USING gist (ST_Buffer(geometry, reverse_place_diameter(rank_search)))
+  WHERE rank_address between 4 and 25 AND type != 'postcode'
+    AND name is not null AND linked_place_id is null AND osm_type = 'N';
+```
+
+## 4.0.0 -> 4.1.0
+
+### ICU tokenizer is the new default
+
+Nominatim now installs the [ICU tokenizer](../customize/Tokenizers.md#icu-tokenizer)
+by default. This only has an effect on newly installed databases. When
+updating older databases, it keeps its installed tokenizer. If you still
+run with the legacy tokenizer, make sure to compile Nominatim with the
+PostgreSQL module, see [Installation](Installation.md#building-nominatim).
+
+### geocodejson output changed
+
+The `type` field of the geocodejson output has changed. It now contains
+the address class of the object instead of the value of the OSM tag. If
+your client has used the `type` field, switch them to read `osm_value`
+instead.
+
+## 3.7.0 -> 4.0.0
+
+### NOMINATIM_PHRASE_CONFIG removed
+
+Custom blacklist configurations for special phrases now need to be handed
+with the `--config` parameter to `nominatim special-phrases`. Alternatively
+you can put your custom configuration in the project directory in a file
+named `phrase-settings.json`.
+
+Version 3.8 also removes the automatic converter for the php format of
+the configuration in older versions. If you are updating from Nominatim < 3.7
+and still work with a custom `phrase-settings.php`, you need to manually
+convert it into a json format.
+
+### PHP utils removed
+
+The old PHP utils have now been removed completely. You need to switch to
+the appropriate functions of the nominatim  command line tool. See
+[Introducing `nominatim` command line tool](#introducing-nominatim-command-line-tool)
+below.
+
+## 3.6.0 -> 3.7.0
+
+### New format and name of configuration file
+
+The configuration for an import is now saved in a `.env` file in the project
+directory. This file follows the dotenv format. For more information, see
+the [installation chapter](Import.md#configuration-setup-in-env).
+
+To migrate to the new system, create a new project directory, add the `.env`
+file and port your custom configuration from `settings/local.php`. Most
+settings are named similar and only have received a `NOMINATIM_` prefix.
+Use the default settings in `settings/env.defaults` as a reference.
+
+### New location for data files
+
+External data files for Wikipedia importance, postcodes etc. are no longer
+expected to reside in the source tree by default. Instead they will be searched
+in the project directory. If you have an automated setup script you must
+either adapt the download location or explicitly set the location of the
+files to the old place in your `.env`.
+
+### Introducing `nominatim` command line tool
+
+The various php utilities have been replaced with a single `nominatim`
+command line tool. Make sure to adapt any scripts. There is no direct 1:1
+matching between the old utilities and the commands of nominatim CLI. The
+following list gives you a list of nominatim sub-commands that contain
+functionality of each script:
+
+* ./utils/setup.php: `import`, `freeze`, `refresh`
+* ./utils/update.php: `replication`, `add-data`, `index`, `refresh`
+* ./utils/specialphrases.php: `special-phrases`
+* ./utils/check_import_finished.php: `admin`
+* ./utils/warm.php: `admin`
+* ./utils/export.php: `export`
+
+Try `nominatim <command> --help` for more information about each subcommand.
+
+`./utils/query.php` no longer exists in its old form. `nominatim search`
+provides a replacement but returns different output.
+
+### Switch to normalized house numbers
+
+The housenumber column in the placex table uses now normalized version.
+The automatic migration step will convert the column but this may take a
+very long time. It is advisable to take the machine offline while doing that.
+
+## 3.5.0 -> 3.6.0

 
 ### Change of layout of search_name_* tables
 
 The table need a different index for nearest place lookup. Recreate the
 
 ### Change of layout of search_name_* tables
 
 The table need a different index for nearest place lookup. Recreate the

-indexs suing the following shell script:
+indexes using the following shell script:

 
 ```bash
 for table in `psql -d nominatim -c "SELECT tablename FROM pg_tables WHERE tablename LIKE 'search_name_%'" -tA | grep -v search_name_blank`;
 
 ```bash
 for table in `psql -d nominatim -c "SELECT tablename FROM pg_tables WHERE tablename LIKE 'search_name_%'" -tA | grep -v search_name_blank`;


@@ -24,7 +203,7 @@ done
 
 The debugging UI is no longer directly provided with Nominatim. Instead we
 now provide a simple Javascript application. Please refer to
 
 The debugging UI is no longer directly provided with Nominatim. Instead we
 now provide a simple Javascript application. Please refer to

-[Setting up the Nominatim UI](../Setup-Nominatim-UI) for details on how to
+[Setting up the Nominatim UI](Setup-Nominatim-UI.md) for details on how to

 set up the UI.
 
 The icons served together with the API responses have been moved to the
 set up the UI.
 
 The icons served together with the API responses have been moved to the


@@ -39,13 +218,15 @@ which needs a different database index. Create it with the following SQL command
 
 ```sql
 CREATE INDEX idx_placex_pendingsector_rank_address
 
 ```sql
 CREATE INDEX idx_placex_pendingsector_rank_address

-  ON placex USING BTREE (rank_address, geometry_sector) where indexed_status > 0;
+  ON placex
+  USING BTREE (rank_address, geometry_sector)
+  WHERE indexed_status > 0;

 ```
 
 You can then drop the old index with:
 
 ```sql
 ```
 
 You can then drop the old index with:
 
 ```sql

-DROP INDEX idx_placex_pendingsector
+DROP INDEX idx_placex_pendingsector;

 ```
 
 ### Unused index
 ```
 
 ### Unused index


@@ -53,7 +234,7 @@ DROP INDEX idx_placex_pendingsector
 This index has been unused ever since the query using it was changed two years ago. Saves about 12GB on a planet installation.
 
 ```sql
 This index has been unused ever since the query using it was changed two years ago. Saves about 12GB on a planet installation.
 
 ```sql

-DROP INDEX idx_placex_geometry_reverse_lookupPoint
+DROP INDEX idx_placex_geometry_reverse_lookupPoint;

 ```
 
 ### Switching to dotenv
 ```
 
 ### Switching to dotenv


@@ -66,6 +247,14 @@ configuration file, run the following command after updating:
 ./utils/setup.php --setup-website
 ```
 
 ./utils/setup.php --setup-website
 ```
 

+### Update SQL code
+
+To update the SQL code to the leatest version run:
+
+```
+./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions
+```
+

 ## 3.4.0 -> 3.5.0
 
 ### New Wikipedia/Wikidata importance tables
 ## 3.4.0 -> 3.5.0
 
 ### New Wikipedia/Wikidata importance tables


@@ -78,10 +267,14 @@ follows:
   * reimport the tables: `./utils/setup.php --import-wikipedia-articles`
   * update the functions: `./utils/setup.php --create-functions --enable-diff-updates`
   * create a new lookup index:
   * reimport the tables: `./utils/setup.php --import-wikipedia-articles`
   * update the functions: `./utils/setup.php --create-functions --enable-diff-updates`
   * create a new lookup index:

-```
-CREATE INDEX idx_placex_wikidata on placex
-USING BTREE ((extratags -> 'wikidata'))
-WHERE extratags ? 'wikidata' and class = 'place' and osm_type = 'N' and rank_search < 26
+```sql
+CREATE INDEX idx_placex_wikidata
+  ON placex
+  USING BTREE ((extratags -> 'wikidata'))
+  WHERE extratags ? 'wikidata'
+    AND class = 'place'
+    AND osm_type = 'N'
+    AND rank_search < 26;

 ```
   * compute importance: `./utils/update.php --recompute-importance`
 
 ```
   * compute importance: `./utils/update.php --recompute-importance`
 


@@ -138,7 +331,7 @@ The new format is
 
 ### Natural Earth country boundaries no longer needed as fallback
 
 
 ### Natural Earth country boundaries no longer needed as fallback
 

-```
+```sql

 DROP TABLE country_naturalearthdata;
 ```
 
 DROP TABLE country_naturalearthdata;
 ```
 


@@ -164,27 +357,37 @@ following command:
 The reverse algorithm has changed and requires new indexes. Run the following
 SQL statements to create the indexes:
 
 The reverse algorithm has changed and requires new indexes. Run the following
 SQL statements to create the indexes:
 

-```
+```sql

 CREATE INDEX idx_placex_geometry_reverse_lookupPoint
 CREATE INDEX idx_placex_geometry_reverse_lookupPoint

-  ON placex USING gist (geometry)
-  WHERE (name is not null or housenumber is not null or rank_address between 26 and 27)
-    AND class not in ('railway','tunnel','bridge','man_made')
-    AND rank_address >= 26 AND indexed_status = 0 AND linked_place_id is null;
+  ON placex
+  USING gist (geometry)
+  WHERE (name IS NOT null or housenumber IS NOT null or rank_address BETWEEN 26 AND 27)
+    AND class NOT IN ('railway','tunnel','bridge','man_made')
+    AND rank_address >= 26
+    AND indexed_status = 0
+    AND linked_place_id IS null;

 CREATE INDEX idx_placex_geometry_reverse_lookupPolygon
   ON placex USING gist (geometry)
   WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')
 CREATE INDEX idx_placex_geometry_reverse_lookupPolygon
   ON placex USING gist (geometry)
   WHERE St_GeometryType(geometry) in ('ST_Polygon', 'ST_MultiPolygon')

-    AND rank_address between 4 and 25 AND type != 'postcode'
-    AND name is not null AND indexed_status = 0 AND linked_place_id is null;
+    AND rank_address between 4 and 25
+    AND type != 'postcode'
+    AND name is not null
+    AND indexed_status = 0
+    AND linked_place_id is null;

 CREATE INDEX idx_placex_geometry_reverse_placeNode
   ON placex USING gist (geometry)
 CREATE INDEX idx_placex_geometry_reverse_placeNode
   ON placex USING gist (geometry)

-  WHERE osm_type = 'N' AND rank_search between 5 and 25
-    AND class = 'place' AND type != 'postcode'
-    AND name is not null AND indexed_status = 0 AND linked_place_id is null;
+  WHERE osm_type = 'N'
+    AND rank_search between 5 and 25
+    AND class = 'place'
+    AND type != 'postcode'
+    AND name is not null
+    AND indexed_status = 0
+    AND linked_place_id is null;

 ```
 
 You also need to grant the website user access to the `country_osm_grid` table:
 
 ```
 
 You also need to grant the website user access to the `country_osm_grid` table:
 

-```
+```sql

 GRANT SELECT ON table country_osm_grid to "www-user";
 ```
 
 GRANT SELECT ON table country_osm_grid to "www-user";
 ```
 


@@ -192,7 +395,7 @@ Replace the `www-user` with the user name of your website server if necessary.
 
 You can now drop the unused indexes:
 
 
 You can now drop the unused indexes:
 

-```
+```sql

 DROP INDEX idx_placex_reverse_geometry;
 ```
 
 DROP INDEX idx_placex_reverse_geometry;
 ```
 


@@ -221,8 +424,8 @@ CREATE INDEX idx_postcode_geometry ON location_postcode USING GIST (geometry);
 CREATE UNIQUE INDEX idx_postcode_id ON location_postcode USING BTREE (place_id);
 CREATE INDEX idx_postcode_postcode ON location_postcode USING BTREE (postcode);
 GRANT SELECT ON location_postcode TO "www-data";
 CREATE UNIQUE INDEX idx_postcode_id ON location_postcode USING BTREE (place_id);
 CREATE INDEX idx_postcode_postcode ON location_postcode USING BTREE (postcode);
 GRANT SELECT ON location_postcode TO "www-data";

-drop type if exists nearfeaturecentr cascade;
-create type nearfeaturecentr as (
+DROP TYPE IF EXISTS nearfeaturecentr CASCADE;
+CREATE TYPE nearfeaturecentr AS (

   place_id BIGINT,
   keywords int[],
   rank_address smallint,
   place_id BIGINT,
   keywords int[],
   rank_address smallint,