replace pylint with flake8 everywhere

[nominatim.git] / docs / admin / Advanced-Installations.md

diff --git a/docs/admin/Advanced-Installations.md b/docs/admin/Advanced-Installations.md
index 7ff5619824674af87c74f9f5a036b286f61eb5ac..de3c5876984ce85e5ea48208b337cf4fa70898c4 100644 (file)
--- a/docs/admin/Advanced-Installations.md
+++ b/docs/admin/Advanced-Installations.md
@@ -5,167 +5,166 @@ your Nominatim database. It is assumed that you have already successfully
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 
 installed the Nominatim software itself, if not return to the 
 [installation page](Installation.md).
 

-## Importing multiple regions
+## Importing with a database user without superuser rights

 
 

-To import multiple regions in your database, you need to configure and run `utils/import_multiple_regions.sh` file. This script will set up the update directory which has the following structure:
+Nominatim usually creates its own PostgreSQL database at the beginning of the
+import process. This makes usage easier for the user but means that the
+database user doing the import needs the appropriate rights.

 
 

-```bash
-update
-    ├── europe
-    │   ├── andorra
-    │   │   └── sequence.state
-    │   └── monaco
-    │       └── sequence.state
-    └── tmp
-        ├── combined.osm.pbf
-        └── europe
-                ├── andorra-latest.osm.pbf
-                └── monaco-latest.osm.pbf
+If you prefer to run the import with a database user with limited rights,
+you can do so by changing the import process as follows:

 
 

+1. Run the command for database preparation with a database user with
+   superuser rights. For example, to use a db user 'dbadmin' for a
+   database 'nominatim', execute:

 
 

-```
+   ```
+   NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim;user=dbadmin" nominatim import --prepare-database
+   ```

 
 

-The `sequence.state` files will contain the sequence ID, which will be used by pyosmium to get updates. The tmp folder is used for import dump.
+2. Grant the import user the right to create tables. For example, foe user 'import-user':

 
 

-### Configuring multiple regions
+   ```
+   psql -d nominatim -c 'GRANT CREATE ON SCHEMA public TO "import-user"'
+   ```

 
 

-The file `import_multiple_regions.sh` needs to be edited as per your requirement:
+3. Now run the reminder of the import with the import user:

 
 

-1. List of countries. eg:
+   ```
+   NOMINATIM_DATABASE_DSN="pgsql:dbname=nominatim;user=import-user" nominatim import --continue import-from-file --osm-file file.pbf
+   ```

 
 

-        COUNTRIES="europe/monaco europe/andorra"
+## Importing multiple regions (without updates)

 
 

-2. Path to Build directory. eg:
+To import multiple regions in your database you can simply give multiple
+OSM files to the import command:

 
 

-        NOMINATIMBUILD="/srv/nominatim/build"
+```
+nominatim import --osm-file file1.pbf --osm-file file2.pbf
+```

 
 

-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
+If you already have imported a file and want to add another one, you can
+use the add-data function to import the additional data as follows:

 
 

-4. Replication URL. eg:
-    
-        BASEURL="https://download.geofabrik.de"
-        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"
- 
-!!! tip
-    If your database already exists and you want to add more countries, replace the setting up part
-    `${SETUPFILE} --osm-file ${UPDATEDIR}/tmp/combined.osm.pbf --all 2>&1`
-    with `${UPDATEFILE} --import-file ${UPDATEDIR}/tmp/combined.osm.pbf 2>&1`.
+```
+nominatim add-data --file <FILE>
+nominatim refresh --postcodes
+nominatim index -j <NUMBER OF THREADS>
+```

 
 

-### Setting up multiple regions
+Please note that adding additional data is always significantly slower than
+the original import.

 
 

-Run the following command from your Nominatim directory after configuring the file.
+## Importing multiple regions (with updates)

 
 

-    bash ./utils/import_multiple_regions.sh
+If you want to import multiple regions _and_ be able to keep them up-to-date
+with updates, then you can use the scripts provided in the `utils` directory.

 
 

-!!! danger "Important"
-        This file uses osmium-tool. It must be installed before executing the import script.
-        Installation instructions can be found [here](https://osmcode.org/osmium-tool/manual.html#installation).
+These scripts will set up an `update` directory in your project directory,
+which has the following structure:

 
 

-### Updating multiple regions
+```bash
+update
+ ├── europe
+ │    ├── andorra
+ │    │    └── sequence.state
+ │    └── monaco
+ │         └── sequence.state
+ └── tmp
+      └── europe
+           ├── andorra-latest.osm.pbf
+           └── monaco-latest.osm.pbf

 
 

-To import multiple regions in your database, you need to configure and run ```utils/update_database.sh```.
-This uses the update directory set up while setting up the DB.   
+```

 
 

-### Configuring multiple regions
+The `sequence.state` files contain the sequence ID for each region. They will
+be used by pyosmium to get updates. The `tmp` folder is used for import dump and
+can be deleted once the import is complete.

 
 

-The file `update_database.sh` needs to be edited as per your requirement:

 
 

-1. List of countries. eg:
+### Setting up multiple regions

 
 

-        COUNTRIES="europe/monaco europe/andorra"
+Create a project directory as described for the
+[simple import](Import.md#creating-the-project-directory). If necessary,
+you can also add an `.env` configuration with customized options. In particular,
+you need to make sure that `NOMINATIM_REPLICATION_UPDATE_INTERVAL` and
+`NOMINATIM_REPLICATION_RECHECK_INTERVAL` are set according to the update
+interval of the extract server you use.

 
 

-2. Path to Build directory. eg:
+Copy the scripts `utils/import_multiple_regions.sh` and `utils/update_database.sh`
+into the project directory.

 
 

-        NOMINATIMBUILD="/srv/nominatim/build"
+Now customize both files as per your requirements

 
 

-3. Path to Update directory. eg:
-        
-        UPDATEDIR="/srv/nominatim/update"
+1. List of countries. e.g.
+
+        COUNTRIES="europe/monaco europe/andorra"
+
+2. URL to the service providing the extracts and updates. eg:

 
 

-4. Replication URL. eg:
-    

         BASEURL="https://download.geofabrik.de"
         BASEURL="https://download.geofabrik.de"

-        DOWNCOUNTRYPOSTFIX="-updates"
+        DOWNCOUNTRYPOSTFIX="-latest.osm.pbf"

 
 

-5. Followup can be set according to your installation. eg: For Photon,
+5. Followup in the update script can be set according to your installation.
+   E.g. for Photon,

 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 
 
         FOLLOWUP="curl http://localhost:2322/nominatim-update"
 
     will handle the indexing.
 

-### Updating the database
-
-Run the following command from your Nominatim directory after configuring the file.
-
-    bash ./utils/update_database.sh
-
-This will get diffs from the replication server, import diffs and index the database. The default replication server in the script([Geofabrik](https://download.geofabrik.de)) provides daily updates.
-
-## Importing Nominatim to an external PostgreSQL database

 
 

-You can install Nominatim using a database that runs on a different server when
-you have physical access to the file system on the other server. Nominatim
-uses a custom normalization library that needs to be made accessible to the
-PostgreSQL server. This section explains how to set up the normalization
-library.
+To start the initial import, change into the project directory and run

 
 

-### Option 1: Compiling the library on the database server
-
-The most sure way to get a working library is to compile it on the database
-server. From the prerequisites you need at least cmake, gcc and the
-PostgreSQL server package.
+```
+    bash import_multiple_regions.sh
+```

 
 

-Clone or unpack the Nominatim source code, enter the source directory and
-create and enter a build directory.
+### Updating the database

 
 

-```sh
-cd Nominatim
-mkdir build
-cd build
-```
+Change into the project directory and run the following command:

 
 

-Now configure cmake to only build the PostgreSQL module and build it:
+    bash update_database.sh

 
 

-```
-cmake -DBUILD_IMPORTER=off -DBUILD_API=off -DBUILD_TESTS=off -DBUILD_DOCS=off -DBUILD_OSM2PGSQL=off ..
-make
-```
+This will get diffs from the replication server, import diffs and index
+the database. The default replication server in the
+script ([Geofabrik](https://download.geofabrik.de)) provides daily updates.

 
 

-When done, you find the normalization library in `build/module/nominatim.so`.
-Copy it to a place where it is readable and executable by the PostgreSQL server
-process.
+## Using an external PostgreSQL database

 
 

-### Option 2: Compiling the library on the import machine
+You can install Nominatim using a database that runs on a different server.
+Simply point the configuration variable `NOMINATIM_DATABASE_DSN` to the
+server and follow the standard import documentation.

 
 

-You can also compile the normalization library on the machine from where you
-run the import.
+The import will be faster, if the import is run directly from the database
+machine. You can easily switch to a different machine for the query frontend
+after the import.

 
 

-!!! important
-    You can only do this when the database server and the import machine have
-    the same architecture and run the same version of Linux. Otherwise there is
-    no guarantee that the compiled library is compatible with the PostgreSQL
-    server running on the database server.
+## Moving the database to another machine

 
 

-Make sure that the PostgreSQL server package is installed on the machine
-**with the same version as on the database server**. You do not need to install
-the PostgreSQL server itself.
+For some configurations it may be useful to run the import on one machine, then
+move the database to another machine and run the Nominatim service from there.
+For example, you might want to use a large machine to be able to run the import
+quickly but only want a smaller machine for production because there is not so
+much load. Or you might want to do the import once and then replicate the
+database to many machines.

 
 

-Download and compile Nominatim as per standard instructions. Once done, you find
-the nomrmalization library in `build/module/nominatim.so`. Copy the file to
-the database server at a location where it is readable and executable by the
-PostgreSQL server process.
+The important thing to keep in mind when transferring the Nominatim installation
+is that you need to transfer the database _and the project directory_. Both
+parts are essential for your installation.

 
 

-### Running the import
+The Nominatim database can be transferred using the `pg_dump`/`pg_restore` tool.
+Make sure to use the same version of PostgreSQL and PostGIS on source and
+target machine.

 
 

-On the client side you now need to configure the import to point to the
-correct location of the library **on the database server**. Add the following
-line to your your `settings/local.php` file:
+!!! note
+    Before creating a dump of your Nominatim database, consider running
+    `nominatim freeze` first. Your database looses the ability to receive further
+    data updates but the resulting database is only about a third of the size
+    of a full database.

 
 

-```php
-@define('CONST_Database_Module_Path', '<directory on the database server where nominatim.so resides>');
-```
+Next install nominatim-api on the target machine by following the standard
+installation instructions. Again, make sure to use the same version as the
+source machine.

 
 

-Now change the `CONST_Database_DSN` to point to your remote server and continue
-to follow the [standard instructions for importing](/admin/Import).
+Create a project directory on your destination machine and set up the `.env`
+file to match the configuration on the source machine. That's all.