Merge pull request #2641 from lonvia/reinit-tokenizer-dir

[nominatim.git] / docs / admin / Advanced-Installations.md

diff --git a/docs/admin/Advanced-Installations.md b/docs/admin/Advanced-Installations.md
index 5b0275b32728ea445e102ddf3eb799f09a672143..aeb2fa5b0e79ffbf526f709b69bb8d0a4cc2d9ac 100644 (file)
--- a/docs/admin/Advanced-Installations.md
+++ b/docs/admin/Advanced-Installations.md
@@ -5,9 +5,34 @@ your Nominatim database. It is assumed that you have already successfully
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 

-## Importing multiple regions
+## Importing multiple regions (without updates)

 
 

-To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
+To import multiple regions in your database you can simply give multiple
+OSM files to the import command:
+
+```
+nominatim import --osm-file file1.pbf --osm-file file2.pbf
+```
+
+If you already have imported a file and want to add another one, you can
+use the add-data function to import the additional data as follows:
+
+```
+nominatim add-data --file <FILE>
+nominatim refresh --postcodes
+nominatim index -j <NUMBER OF THREADS>
+```
+
+Please note that adding additional data is always significantly slower than
+the original import.
+
+## Importing multiple regions (with updates)
+
+If you want to import multiple regions _and_ be able to keep them up-to-date
+with updates, then you can use the scripts provided in the `utils` directory.
+
+These scripts will set up an `update` directory in your project directory,
+which has the following structure:

 
 ```bash
 update
 
 ```bash
 update


@@ -17,7 +42,6 @@ update
     │   └── monaco
     │       └── sequence.state
     └── tmp
     │   └── monaco
     │       └── sequence.state
     └── tmp

-        ├── combined.osm.pbf

         └── europe
                 ├── andorra-latest.osm.pbf
                 └── monaco-latest.osm.pbf
         └── europe
                 ├── andorra-latest.osm.pbf
                 └── monaco-latest.osm.pbf


@@ -25,86 +49,59 @@ update
 
 ```
 
 
 ```
 

-The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
-
-### Configuring multiple regions
+The `sequence.state` files contain the sequence ID for each region. They will
+be used by pyosmium to get updates. The `tmp` folder is used for import dump and
+can be deleted once the import is complete.

 
 

-The file `import_multiple_regions.sh` needs to be edited as per your requirement:
-
-1. List of countries. eg:
-
-        COUNTRIES="europe/monaco europe/andorra"
-
-2. Path to Build directory. eg:
-
-        NOMINATIMBUILD="/srv/nominatim/build"
-
-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
-
-4. Replication URL. eg:
-    
-        BASEURL="https://download.geofabrik.de"
-        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
- 
-!!! tip
-    If your database already exists and you want to add more countries, replace the setting up part
-    `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
-    with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
-    where N is the numbers of CPUs in your system.

 
 ### Setting up multiple regions
 
 
 ### Setting up multiple regions
 

-Run the following command from your Nominatim directory after configuring the file.
-
-    bash ./utils/import_multiple_regions.sh
-
-!!! danger "Important"
-        This file uses osmium-tool. It must be installed before executing the import script.
-        Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
-
-### Updating multiple regions
+Create a project directory as described for the
+[simple import](Import.md#creating-the-project-directory). If necessary,
+you can also add an `.env` configuration with customized options. In particular,
+you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and
+`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update
+interval of the extract server you use.

 
 

-To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
-This uses the update directory set up while setting up the DB.   
+Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh`
+into the project directory.

 
 

-### Configuring multiple regions
+Now customize both files as per your requirements

 
 

-The file `update_database.sh` needs to be edited as per your requirement:
-
-1. List of countries. eg:
+1. List of countries. e.g.

 
         COUNTRIES="europe/monaco europe/andorra"
 
 
         COUNTRIES="europe/monaco europe/andorra"
 

-2. Path to Build directory. eg:
-
-        NOMINATIMBUILD="/srv/nominatim/build"
-
-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
+2. URL to the service providing the extracts and updates. eg:

 
 

-4. Replication URL. eg:
-    

         BASEURL="https://download.geofabrik.de"
         BASEURL="https://download.geofabrik.de"

-        DOWNCOUNTRYPOSTFIX="-updates"
+        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"

 
 

-5. Followup can be set according to your installation. eg: For Photon,
+5. Followup in the update script can be set according to your installation.
+   E.g. for Photon,

 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 
 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 

+
+To start the initial import, change into the project directory and run
+
+```
+    bash import_multiple_regions.sh
+```
+

 ### Updating the database
 
 ### Updating the database
 

-Run the following command from your Nominatim directory after configuring the file.
+Change into the project directory and run the following command:

 
 

-    bash ./utils/update_database.sh
+    bash update_database.sh

 
 

-This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
+This will get diffs from the replication server, import diffs and index
+the database. The default replication server in the
+script([Geofabrik](https://download.geofabrik.de)) provides daily updates.

 
 

-## Importing Nominatim to an external PostgreSQL database
+## Using an external PostgreSQL database

 
 You can install Nominatim using a database that runs on a different server when
 you have physical access to the file system on the other server. Nominatim
 
 You can install Nominatim using a database that runs on a different server when
 you have physical access to the file system on the other server. Nominatim


@@ -112,6 +109,11 @@ uses a custom normalization library that needs to be made accessible to the
 PostgreSQL server. This section explains how to set up the normalization
 library.
 
 PostgreSQL server. This section explains how to set up the normalization
 library.
 

+!!! note
+    The external module is only needed when using the legacy tokenizer.
+    If you have choosen the ICU tokenizer, then you can ignore this section
+    and follow the standard import documentation.
+

 ### Option 1: Compiling the library on the database server
 
 The most sure way to get a working library is to compile it on the database
 ### Option 1: Compiling the library on the database server
 
 The most sure way to get a working library is to compile it on the database


@@ -154,7 +156,7 @@ Make sure that the PostgreSQL server package is installed on the machine
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find

-the nomrmalization library in `build/module/nominatim.so`. Copy the file to
+the normalization library in `build/module/nominatim.so`. Copy the file to

 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 
 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 


@@ -162,11 +164,53 @@ PostgreSQL server process.
 
 On the client side you now need to configure the import to point to the
 correct location of the library **on the database server**. Add the following
 
 On the client side you now need to configure the import to point to the
 correct location of the library **on the database server**. Add the following

-line to your your `settings/local.php` file:
+line to your your `.env` file:

 
 ```php
 
 ```php

-@define('CONST_Database_Module_Path', '<directory on the database server where nominatim.so resides>');
+NOMINATIM_DATABASE_MODULE_PATH="<directory on the database server where nominatim.so resides>"

 ```
 
 ```
 

-Now change the `CONST_Database_DSN` to point to your remote server and continue
-to follow the [standard instructions for importing](/admin/Import).
+Now change the `NOMINATIM_DATABASE_DSN` to point to your remote server and continue
+to follow the [standard instructions for importing](Import.md).
+
+
+## Moving the database to another machine
+
+For some configurations it may be useful to run the import on one machine, then
+move the database to another machine and run the Nominatim service from there.
+For example, you might want to use a large machine to be able to run the import
+quickly but only want a smaller machine for production because there is not so
+much load. Or you might want to do the import once and then replicate the
+database to many machines.
+
+The important thing to keep in mind when transferring the Nominatim installation
+is that you need to transfer the database _and the project directory_. Both
+parts are essential for your installation.
+
+The Nominatim database can be transferred using the `pg_dump`/`pg_restore` tool.
+Make sure to use the same version of PostgreSQL and PostGIS on source and
+target machine.
+
+!!! note
+    Before creating a dump of your Nominatim database, consider running
+    `nominatim freeze` first. Your database looses the ability to receive further
+    data updates but the resulting database is only about a third of the size
+    of a full database.
+
+Next install Nominatim on the target machine by following the standard installation
+instructions. Again, make sure to use the same version as the source machine.
+
+Create a project directory on your destination machine and set up the `.env`
+file to match the configuration on the source machine. Finally run
+
+    nominatim refresh --website
+
+to make sure that the local installation of Nominatim will be used.
+
+If you are using the legacy tokenizer you might also have to switch to the
+PostgreSQL module that was compiled on your target machine. If you get errors
+that PostgreSQL cannot find or access `nominatim.so` then rerun
+
+   nominatim refresh --functions
+
+on the target machine to update the the location of the module.