Merge pull request #3521 from lonvia/make-requests-optional

[nominatim.git] / docs / admin / Advanced-Installations.md

diff --git a/docs/admin/Advanced-Installations.md b/docs/admin/Advanced-Installations.md
index 5b0275b32728ea445e102ddf3eb799f09a672143..f8232fb29cea6189b36a2ae7d47a8617531813fb 100644 (file)
--- a/docs/admin/Advanced-Installations.md
+++ b/docs/admin/Advanced-Installations.md
@@ -5,106 +5,131 @@ your Nominatim database. It is assumed that you have already successfully
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 

-## Importing multiple regions
+## Importing with a database user without superuser rights

 
 

-To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
+Nominatim usually creates its own PostgreSQL database at the beginning of the
+import process. This makes usage easier for the user but means that the
+database user doing the import needs the appropriate rights.

 
 

-```bash
-update
-    ├── europe
-    │   ├── andorra
-    │   │   └── sequence.state
-    │   └── monaco
-    │       └── sequence.state
-    └── tmp
-        ├── combined.osm.pbf
-        └── europe
-                ├── andorra-latest.osm.pbf
-                └── monaco-latest.osm.pbf
+If you prefer to run the import with a database user with limited rights,
+you can do so by changing the import process as follows:

 
 

+1. Run the command for database preparation with a database user with
+   superuser rights. For example, to use a db user 'dbadmin' for a
+   database 'nominatim', execute:

 
 

-```
+   ```
+   NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim;user=dbadmin" nominatim import --prepare-database
+   ```

 
 

-The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
+2. Grant the import user the right to create tables. For example, foe user 'import-user':

 
 

-### Configuring multiple regions
+   ```
+   psql -d nominatim -c 'GRANT CREATE ON SCHEMA public TO "import-user"'
+   ```

 
 

-The file `import_multiple_regions.sh` needs to be edited as per your requirement:
+3. Now run the reminder of the import with the import user:

 
 

-1. List of countries. eg:
+   ```
+   NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim;user=import-user" nominatim import --continue import-from-file --osm-file file.pbf
+   ```

 
 

-        COUNTRIES="europe/monaco europe/andorra"
+## Importing multiple regions (without updates)

 
 

-2. Path to Build directory. eg:
+To import multiple regions in your database you can simply give multiple
+OSM files to the import command:

 
 

-        NOMINATIMBUILD="/srv/nominatim/build"
+```
+nominatim import --osm-file file1.pbf --osm-file file2.pbf
+```

 
 

-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
+If you already have imported a file and want to add another one, you can
+use the add-data function to import the additional data as follows:

 
 

-4. Replication URL. eg:
-    
-        BASEURL="https://download.geofabrik.de"
-        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
- 
-!!! tip
-    If your database already exists and you want to add more countries, replace the setting up part
-    `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
-    with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf --index --index-instances N 2>&1`
-    where N is the numbers of CPUs in your system.
+```
+nominatim add-data --file <FILE>
+nominatim refresh --postcodes
+nominatim index -j <NUMBER OF THREADS>
+```

 
 

-### Setting up multiple regions
+Please note that adding additional data is always significantly slower than
+the original import.
+
+## Importing multiple regions (with updates)
+
+If you want to import multiple regions _and_ be able to keep them up-to-date
+with updates, then you can use the scripts provided in the `utils` directory.

 
 

-Run the following command from your Nominatim directory after configuring the file.
+These scripts will set up an `update` directory in your project directory,
+which has the following structure:

 
 

-    bash ./utils/import_multiple_regions.sh
+```bash
+update
+ ├── europe
+ │    ├── andorra
+ │    │    └── sequence.state
+ │    └── monaco
+ │         └── sequence.state
+ └── tmp
+      └── europe
+           ├── andorra-latest.osm.pbf
+           └── monaco-latest.osm.pbf

 
 

-!!! danger "Important"
-        This file uses osmium-tool. It must be installed before executing the import script.
-        Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
+```

 
 

-### Updating multiple regions
+The `sequence.state` files contain the sequence ID for each region. They will
+be used by pyosmium to get updates. The `tmp` folder is used for import dump and
+can be deleted once the import is complete.

 
 

-To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
-This uses the update directory set up while setting up the DB.   

 
 

-### Configuring multiple regions
+### Setting up multiple regions

 
 

-The file `update_database.sh` needs to be edited as per your requirement:
+Create a project directory as described for the
+[simple import](Import.md#creating-the-project-directory). If necessary,
+you can also add an `.env` configuration with customized options. In particular,
+you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and
+`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update
+interval of the extract server you use.

 
 

-1. List of countries. eg:
+Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh`
+into the project directory.

 
 

-        COUNTRIES="europe/monaco europe/andorra"
+Now customize both files as per your requirements

 
 

-2. Path to Build directory. eg:
+1. List of countries. e.g.

 
 

-        NOMINATIMBUILD="/srv/nominatim/build"
+        COUNTRIES="europe/monaco europe/andorra"

 
 

-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
+2. URL to the service providing the extracts and updates. eg:

 
 

-4. Replication URL. eg:
-    

         BASEURL="https://download.geofabrik.de"
         BASEURL="https://download.geofabrik.de"

-        DOWNCOUNTRYPOSTFIX="-updates"
+        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"

 
 

-5. Followup can be set according to your installation. eg: For Photon,
+5. Followup in the update script can be set according to your installation.
+   E.g. for Photon,

 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 
 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 

+
+To start the initial import, change into the project directory and run
+
+```
+    bash import_multiple_regions.sh
+```
+

 ### Updating the database
 
 ### Updating the database
 

-Run the following command from your Nominatim directory after configuring the file.
+Change into the project directory and run the following command:

 
 

-    bash ./utils/update_database.sh
+    bash update_database.sh

 
 

-This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
+This will get diffs from the replication server, import diffs and index
+the database. The default replication server in the
+script ([Geofabrik](https://download.geofabrik.de)) provides daily updates.

 
 

-## Importing Nominatim to an external PostgreSQL database
+## Using an external PostgreSQL database

 
 You can install Nominatim using a database that runs on a different server when
 you have physical access to the file system on the other server. Nominatim
 
 You can install Nominatim using a database that runs on a different server when
 you have physical access to the file system on the other server. Nominatim


@@ -112,6 +137,11 @@ uses a custom normalization library that needs to be made accessible to the
 PostgreSQL server. This section explains how to set up the normalization
 library.
 
 PostgreSQL server. This section explains how to set up the normalization
 library.
 

+!!! note
+    The external module is only needed when using the legacy tokenizer.
+    If you have chosen the ICU tokenizer, then you can ignore this section
+    and follow the standard import documentation.
+

 ### Option 1: Compiling the library on the database server
 
 The most sure way to get a working library is to compile it on the database
 ### Option 1: Compiling the library on the database server
 
 The most sure way to get a working library is to compile it on the database


@@ -154,7 +184,7 @@ Make sure that the PostgreSQL server package is installed on the machine
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find
 the PostgreSQL server itself.
 
 Download and compile Nominatim as per standard instructions. Once done, you find

-the nomrmalization library in `build/module/nominatim.so`. Copy the file to
+the normalization library in `build/module/nominatim.so`. Copy the file to

 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 
 the database server at a location where it is readable and executable by the
 PostgreSQL server process.
 


@@ -162,11 +192,53 @@ PostgreSQL server process.
 
 On the client side you now need to configure the import to point to the
 correct location of the library **on the database server**. Add the following
 
 On the client side you now need to configure the import to point to the
 correct location of the library **on the database server**. Add the following

-line to your your `settings/local.php` file:
+line to your your `.env` file:

 
 ```php
 
 ```php

-@define('CONST_Database_Module_Path', '<directory on the database server where nominatim.so resides>');
+NOMINATIM_DATABASE_MODULE_PATH="<directory on the database server where nominatim.so resides>"

 ```
 
 ```
 

-Now change the `CONST_Database_DSN` to point to your remote server and continue
-to follow the [standard instructions for importing](/admin/Import).
+Now change the `NOMINATIM_DATABASE_DSN` to point to your remote server and continue
+to follow the [standard instructions for importing](Import.md).
+
+
+## Moving the database to another machine
+
+For some configurations it may be useful to run the import on one machine, then
+move the database to another machine and run the Nominatim service from there.
+For example, you might want to use a large machine to be able to run the import
+quickly but only want a smaller machine for production because there is not so
+much load. Or you might want to do the import once and then replicate the
+database to many machines.
+
+The important thing to keep in mind when transferring the Nominatim installation
+is that you need to transfer the database _and the project directory_. Both
+parts are essential for your installation.
+
+The Nominatim database can be transferred using the `pg_dump`/`pg_restore` tool.
+Make sure to use the same version of PostgreSQL and PostGIS on source and
+target machine.
+
+!!! note
+    Before creating a dump of your Nominatim database, consider running
+    `nominatim freeze` first. Your database looses the ability to receive further
+    data updates but the resulting database is only about a third of the size
+    of a full database.
+
+Next install Nominatim on the target machine by following the standard installation
+instructions. Again, make sure to use the same version as the source machine.
+
+Create a project directory on your destination machine and set up the `.env`
+file to match the configuration on the source machine. Finally run
+
+    nominatim refresh --website
+
+to make sure that the local installation of Nominatim will be used.
+
+If you are using the legacy tokenizer you might also have to switch to the
+PostgreSQL module that was compiled on your target machine. If you get errors
+that PostgreSQL cannot find or access `nominatim.so` then rerun
+
+    nominatim refresh --functions
+
+on the target machine to update the the location of the module.