X-Git-Url: https://git.openstreetmap.org./nominatim.git/blobdiff_plain/d1b1acaf1dce32d0e7dfd6879459e51fd6d55535..8498593a634e3d32b3b9417907c31bc95b6122f4:/docs/Import_and_update.md diff --git a/docs/Import_and_update.md b/docs/Import_and_update.md index c2dcf937..266a5d8b 100644 --- a/docs/Import_and_update.md +++ b/docs/Import_and_update.md @@ -3,8 +3,8 @@ Importing a new database The following instructions explain how to create a Nominatim database from an OSM planet file and how to keep the database up to date. It -is assumed that you have already sucessfully installed the Nominatim -software itself, if not return to the [prerequisites page](Prerequisites.md). +is assumed that you have already successfully installed the Nominatim +software itself, if not return to the [installation page](Installation.md). Configuration setup in settings/local.php ----------------------------------------- @@ -15,10 +15,12 @@ If you plan to import a large dataset (e.g. Europe, North America, planet), you should also enable flatnode storage of node locations. With this setting enabled, node coordinates are stored in a simple file instead of the database. This will save you import time and disk storage. -Add to your settings/local.php: +Add to your `settings/local.php`: @define('CONST_Osm2pgsql_Flatnode_File', '/path/to/flatnode.file'); +Replace the second part with a suitable path on your system and make sure +the directory exists. There should be at least 35GB of free space. Downloading additional data --------------------------- @@ -45,33 +47,131 @@ Nominatim can use postcodes from an external source to improve searches that inv wget http://www.nominatim.org/data/gb_postcode_data.sql.gz -Initial Import of the Data +Initial import of the data -------------------------- -**Important:** first try the import with a small excerpt, for example from Geofabrik. +**Important:** first try the import with a small excerpt, for example from +[Geofabrik](http://download.geofabrik.de). Download the data to import and load the data with the following command: ./utils/setup.php --osm-file --all [--osm2pgsql-cache 28000] 2>&1 | tee setup.log -The --osm2pgsql-cache parameter is optional but strongly recommended for +The `--osm2pgsql-cache` parameter is optional but strongly recommended for planet imports. It sets the node cache size for the osm2pgsql import part -(see -C parameter in osm2pgsql help). 28GB are recommended for a full planet -imports, for excerpts you can use less. -Adapt to your available RAM to avoid swapping. +(see `-C` parameter in osm2pgsql help). 28GB are recommended for a full planet +import, for excerpts you can use less. Adapt to your available RAM to +avoid swapping, never give more than 2/3 of RAM to osm2pgsql. -The import will take as little as an hour for a small country extract -and as much as 10 days for a full-scale planet import on less powerful -hardware. - -Loading Additional Datasets +Loading additional datasets --------------------------- -The following commands will create additional entries for countries and POI searches: +The following commands will create additional entries for POI searches: + + ./utils/specialphrases.php --wiki-import > specialphrases.sql + psql -d nominatim -f specialphrases.sql + + +Installing Tiger housenumber data for the US +============================================ + +Nominatim is able to use the official TIGER address set to complement the +OSM housenumber data in the US. You can add TIGER data to your own Nominatim +instance by following these steps: + + 1. Install the GDAL library and python bindings and the unzip tool + + * Ubuntu: `sudo apt-get install python-gdal unzip` + * CentOS: `sudo yum install gdal-python unzip` + + 2. Get the TIGER 2015 data. You will need the EDGES files + (3,234 zip files, 11GB total). Choose one of the two sources: + + wget -r ftp://ftp2.census.gov/geo/tiger/TIGER2015/EDGES/ + wget -r ftp://mirror1.shellbot.com/census/geo/tiger/TIGER2015/EDGES/ + + The first one is the original source, the second a considerably faster + mirror. + + 3. Convert the data into SQL statements (stored in data/tiger): + + ./utils/imports.php --parse-tiger + + 4. Import the data into your Nominatim database: + + ./utils/setup.php --import-tiger-data + + 5. Enable use of the Tiger data in your `settings/local.php` by adding: + + @define('CONST_Use_US_Tiger_Data', true); + + 6. Apply the new settings: + + ./utils/setup.php --create-functions --enable-diff-updates --create-partition-functions + +Be warned that the import can take a very long time, especially if you +import all of the US. The entire US adds about 10GB to your database. + + +Updates +======= + +There are many different possibilities to update your Nominatim database. +The following section describes how to keep it up-to-date with Pyosmium. +For a list of other methods see the output of `./utils/update.php --help`. + +Installing the newest version of Pyosmium +----------------------------------------- + +It is recommended to install Pyosmium via pip: + + pip install --user osmium + +Nominatim needs a tool called `pyosmium-get-updates` that comes with +Pyosmium. You need to tell Nominatim where to find it. Add the +following line to your `settings/local.php`: + + @define('CONST_Pyosmium_Binary', '/home/user/.local/bin/pyosmium-get-changes'); + +The path above is fine if you used the `--user` parameter with pip. +Replace `user` with your user name. + +Setting up the update process +----------------------------- + +Next the update needs to be initialised. By default Nominatim is configured +to update using the global minutely diffs. + +If you want a different update source you will need to add some settings +to `settings/local.php`. For example, to use the daily country extracts +diffs for Ireland from geofabrik add the following: + + // base URL of the replication service + @define('CONST_Replication_Url', 'http://download.geofabrik.de/europe/ireland-and-northern-ireland-updates'); + // How often upstream publishes diffs + @define('CONST_Replication_Update_Interval', '86400'); + // How long to sleep if no update found yet + @define('CONST_Replication_Recheck_Interval', '900'); + +To set up the update process now run the following command: + + ./utils/update --init-updates + +It outputs the date where updates will start. Recheck that this date is +what you expect. + +The --init-updates command needs to be rerun whenever the replication service +is changed. + +Updating Nominatim +------------------ + +The following command will keep your database constantly up to date: + + ./utils/update.php --import-osmosis-all - ./utils/specialphrases.php --countries > data/specialphrases_countries.sql - psql -d nominatim -f data/specialphrases_countries.sql - ./utils/specialphrases.php --wiki-import > data/specialphrases.sql - psql -d nominatim -f data/specialphrases.sql +If you have imported multiple country extracts and want to keep them +up-to-date, have a look at the script in +[issue #60](https://github.com/openstreetmap/Nominatim/issues/60).