Merge remote-tracking branch 'upstream/master'

[nominatim.git] / docs / admin / Advanced-Installations.md

diff --git a/docs/admin/Advanced-Installations.md b/docs/admin/Advanced-Installations.md
index 5701992cfd2d255d590ae88b2579a3f3e7b5443d..3b98fec39579a5b286542349525bcd1bd63bcc5f 100644 (file)
--- a/docs/admin/Advanced-Installations.md
+++ b/docs/admin/Advanced-Installations.md
@@ -5,106 +5,102 @@ your Nominatim database. It is assumed that you have already successfully
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 

-## Importing multiple regions
+## Importing multiple regions (without updates)

 
 

-To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
+To import multiple regions in your database you can simply give multiple
+OSM files to the import command:

 
 

-```bash
-update
-    ├── europe
-    │   ├── andorra
-    │   │   └── sequence.state
-    │   └── monaco
-    │       └── sequence.state
-    └── tmp
-        ├── combined.osm.pbf
-        └── europe
-                ├── andorra-latest.osm.pbf
-                └── monaco-latest.osm.pbf
+```
+nominatim import --osm-file file1.pbf --osm-file file2.pbf
+```

 
 

+If you already have imported a file and want to add another one, you can
+use the add-data function to import the additional data as follows:

 
 

+```
+nominatim add-data --file <FILE>
+nominatim refresh --postcodes
+nominatim index -j <NUMBER OF THREADS>

 ```
 
 ```
 

-The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
+Please note that adding additional data is always significantly slower than
+the original import.

 
 

-### Configuring multiple regions
+## Importing multiple regions (with updates)

 
 

-The file `import_multiple_regions.sh` needs to be edited as per your requirement:
+If you want to import multiple regions _and_ be able to keep them up-to-date
+with updates, then you can use the scripts provided in the `utils` directory.

 
 

-1. List of countries. eg:
+These scripts will set up an `update` directory in your project directory,
+which has the following structure:

 
 

-        COUNTRIES="europe/monaco europe/andorra"
+```bash
+update
+ ├── europe
+ │    ├── andorra
+ │    │    └── sequence.state
+ │    └── monaco
+ │         └── sequence.state
+ └── tmp
+      └── europe
+           ├── andorra-latest.osm.pbf
+           └── monaco-latest.osm.pbf

 
 

-2. Path to Build directory. eg:
+```

 
 

-        NOMINATIMBUILD="/srv/nominatim/build"
+The `sequence.state` files contain the sequence ID for each region. They will
+be used by pyosmium to get updates. The `tmp` folder is used for import dump and
+can be deleted once the import is complete.

 
 

-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
-
-4. Replication URL. eg:
-    
-        BASEURL="https://download.geofabrik.de"
-        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
- 
-!!! tip
-    If your database already exists and you want to add more countries, replace the setting up part
-    `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
-    with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
-    where N is the numbers of CPUs in your system.

 
 ### Setting up multiple regions
 
 
 ### Setting up multiple regions
 

-Run the following command from your Nominatim directory after configuring the file.
-
-    bash ./utils/import_multiple_regions.sh
-
-!!! danger "Important"
-        This file uses osmium-tool. It must be installed before executing the import script.
-        Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
-
-### Updating multiple regions
+Create a project directory as described for the
+[simple import](Import.md#creating-the-project-directory). If necessary,
+you can also add an `.env` configuration with customized options. In particular,
+you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and
+`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update
+interval of the extract server you use.

 
 

-To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
-This uses the update directory set up while setting up the DB.   
+Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh`
+into the project directory.

 
 

-### Configuring multiple regions
+Now customize both files as per your requirements

 
 

-The file `update_database.sh` needs to be edited as per your requirement:
-
-1. List of countries. eg:
+1. List of countries. e.g.

 
         COUNTRIES="europe/monaco europe/andorra"
 
 
         COUNTRIES="europe/monaco europe/andorra"
 

-2. Path to Build directory. eg:
-
-        NOMINATIMBUILD="/srv/nominatim/build"
+2. URL to the service providing the extracts and updates. eg:

 
 

-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
-
-4. Replication URL. eg:
-    

         BASEURL="https://download.geofabrik.de"
         BASEURL="https://download.geofabrik.de"

-        DOWNCOUNTRYPOSTFIX="-updates"
+        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"

 
 

-5. Followup can be set according to your installation. eg: For Photon,
+5. Followup in the update script can be set according to your installation.
+   E.g. for Photon,

 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 
 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 

+
+To start the initial import, change into the project directory and run
+
+```
+    bash import_multiple_regions.sh
+```
+

 ### Updating the database
 
 ### Updating the database
 

-Run the following command from your Nominatim directory after configuring the file.
+Change into the project directory and run the following command:

 
 

-    bash ./utils/update_database.sh
+    bash update_database.sh

 
 

-This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
+This will get diffs from the replication server, import diffs and index
+the database. The default replication server in the
+script ([Geofabrik](https://download.geofabrik.de)) provides daily updates.

 
 

-## Importing Nominatim to an external PostgreSQL database
+## Using an external PostgreSQL database

 
 You can install Nominatim using a database that runs on a different server when
 you have physical access to the file system on the other server. Nominatim
 
 You can install Nominatim using a database that runs on a different server when
 you have physical access to the file system on the other server. Nominatim


@@ -112,6 +108,11 @@ uses a custom normalization library that needs to be made accessible to the
 PostgreSQL server. This section explains how to set up the normalization
 library.
 
 PostgreSQL server. This section explains how to set up the normalization
 library.
 

+!!! note
+    The external module is only needed when using the legacy tokenizer.
+    If you have chosen the ICU tokenizer, then you can ignore this section
+    and follow the standard import documentation.
+

 ### Option 1: Compiling the library on the database server
 
 The most sure way to get a working library is to compile it on the database
 ### Option 1: Compiling the library on the database server
 
 The most sure way to get a working library is to compile it on the database


@@ -154,7 +155,7 @@ Make sure that the PostgreSQL server package is installed on the machine
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find

-the nomrmalization library in `build/module/nominatim.so`. Copy the file to
+the normalization library in `build/module/nominatim.so`. Copy the file to

 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 
 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 


@@ -169,4 +170,46 @@ NOMINATIM_DATABASE_MODULE_PATH="<directory on the database server where nominati
 ```
 
 Now change the `NOMINATIM_DATABASE_DSN` to point to your remote server and continue
 ```
 
 Now change the `NOMINATIM_DATABASE_DSN` to point to your remote server and continue

-to follow the [standard instructions for importing](/admin/Import).
+to follow the [standard instructions for importing](Import.md).
+
+
+## Moving the database to another machine
+
+For some configurations it may be useful to run the import on one machine, then
+move the database to another machine and run the Nominatim service from there.
+For example, you might want to use a large machine to be able to run the import
+quickly but only want a smaller machine for production because there is not so
+much load. Or you might want to do the import once and then replicate the
+database to many machines.
+
+The important thing to keep in mind when transferring the Nominatim installation
+is that you need to transfer the database _and the project directory_. Both
+parts are essential for your installation.
+
+The Nominatim database can be transferred using the `pg_dump`/`pg_restore` tool.
+Make sure to use the same version of PostgreSQL and PostGIS on source and
+target machine.
+
+!!! note
+    Before creating a dump of your Nominatim database, consider running
+    `nominatim freeze` first. Your database looses the ability to receive further
+    data updates but the resulting database is only about a third of the size
+    of a full database.
+
+Next install Nominatim on the target machine by following the standard installation
+instructions. Again, make sure to use the same version as the source machine.
+
+Create a project directory on your destination machine and set up the `.env`
+file to match the configuration on the source machine. Finally run
+
+    nominatim refresh --website
+
+to make sure that the local installation of Nominatim will be used.
+
+If you are using the legacy tokenizer you might also have to switch to the
+PostgreSQL module that was compiled on your target machine. If you get errors
+that PostgreSQL cannot find or access `nominatim.so` then rerun
+
+   nominatim refresh --functions
+
+on the target machine to update the the location of the module.