* the result you are getting
* the expected result, preferably a link to the OSM object you want to find,
otherwise an address that is as precise as possible
-
- To get the link to the OSM object, you can try the following:
-
+
+To get the link to the OSM object, you can try the following:
+
* go to https://openstreetmap.org
* zoom to the area of the map where you expect the result and
zoom in as much as possible
* find the object of interest in the list that appears on the left side
* click on the object and report the URL back that the browser shows
-### When Reporting Problems with your Installation...
+### When Reporting Bugs...
Please add the following information to your issue:
if you run from the git repo, the output of `git rev-parse HEAD`)
* (if applicable) exact command line of the command that was causing the issue
+Bug reports that do not include extensive information about your system,
+about the problem and about what you have been trying to debug the problem
+will be closed.
## Workflow for Pull Requests
Installation
============
+**Nominatim is a complex piece of software and runs in a complex environment.
+Installing and running Nominatim is something for experienced system
+administrators only who can do some trouble-shooting themselves. We are sorry,
+but we can not provide installation support. We are all doing this in our free
+time and there is just so much of that time to go around. Do not open issues in
+our bug tracker if you need help. You can ask questions on the mailing list
+(see below) or on [help.openstreetmap.org](https://help.openstreetmap.org/).**
+
The latest stable release can be downloaded from https://nominatim.org.
-There you can also find [installation instructions for the release](https://nominatim.org/release-docs/latest/admin/Installation).
+There you can also find [installation instructions for the release](https://nominatim.org/release-docs/latest/admin/Installation), as well as an extensive [Troubleshooting/FAQ section](https://nominatim.org/release-docs/latest/admin/Faq/).
Detailed installation instructions for the development version can be
found at [nominatim.org](https://nominatim.org/release-docs/develop/admin/Installation)
PHP Warning: file_get_contents(): open_basedir restriction in effect.
-You need to adjust the [open_basedir](https://www.php.net/manual/en/ini.core.php#ini.open-basedir) setting
-in your PHP configuration (`php.ini file`). By default this setting may look like this:
+You need to adjust the
+[open_basedir](https://www.php.net/manual/en/ini.core.php#ini.open-basedir)
+setting in your PHP configuration (`php.ini` file). By default this setting may
+look like this:
open_basedir = /srv/http/:/home/:/tmp/:/usr/share/pear/
-Either add reported directories to the list or disable this setting temporarily by
-dding ";" at the beginning of the line. Don't forget to enable this setting again
-once you are done with the PHP command line operations.
+Either add reported directories to the list or disable this setting temporarily
+by adding ";" at the beginning of the line. Don't forget to enable this setting
+again once you are done with the PHP command line operations.
### PHP timzeone warnings
`could not connect to server: No such file or directory`
-On CentOS v7 the PostgreSQL server is started with `systemd`.
-Check if `/usr/lib/systemd/system/httpd.service` contains a line `PrivateTmp=true`.
-If so then Apache cannot see the `/tmp/.s.PGSQL.5432` file. It's a good security feature,
-so use the [preferred solution](../appendix/Install-on-Centos-7/#adding-selinux-security-settings).
+On CentOS v7 the PostgreSQL server is started with `systemd`. Check if
+`/usr/lib/systemd/system/httpd.service` contains a line `PrivateTmp=true`. If
+so then Apache cannot see the `/tmp/.s.PGSQL.5432` file. It's a good security
+feature, so use the
+[preferred solution](../appendix/Install-on-Centos-7/#adding-selinux-security-settings).
However, you can solve this the quick and dirty way by commenting out that line and then run
sudo systemctl restart httpd
-### "must be an array or an object that implements Countable" warning in /usr/share/pear/DB.php
-
-The warning started with PHP 7.2. Make sure you have at least [version 1.9.3 of PEAR DB](https://github.com/pear/DB/releases)
-installed.
-
### Website reports "DB Error: insufficient permissions"
-The user the webserver, e.g. Apache, runs under needs to have access to the Nominatim database. You can find the user like [this](https://serverfault.com/questions/125865/finding-out-what-user-apache-is-running-as), for default Ubuntu operating system for example it's `www-data`.
+The user the webserver, e.g. Apache, runs under needs to have access to the
+Nominatim database. You can find the user like
+[this](https://serverfault.com/questions/125865/finding-out-what-user-apache-is-running-as),
+for default Ubuntu operating system for example it's `www-data`.
1. Repeat the `createuser` step of the installation instructions.
### Setup.php fails with "DB Error: extension not found"
Make sure you have the PostgreSQL extensions "hstore" and "postgis" installed.
-See the installation instruction for a full list of required packages.
-
-
-### Setup.php reports "Cannot redeclare getDB()"
-
-`Cannot redeclare getDB() (previously declared in /your/path/Nominatim/lib/db.php:4)`
-
-The message is a bit misleading as PHP needs to load the file `DB.php` and
-instead re-loads Nominatim's `db.php`. To solve this make sure you
-have the [Pear module 'DB'](https://pear.php.net/package/DB/) installed.
+See the installation instructions for a full list of required packages.
- sudo pear install DB
### I forgot to delete the flatnodes file before starting an import.
@define('CONST_Osm2pgsql_Flatnode_File', '/path/to/flatnode.file');
Replace the second part with a suitable path on your system and make sure
-the directory exists. There should be at least 40GB of free space.
+the directory exists. There should be at least 64GB of free space.
## Downloading additional data
-### Wikipedia rankings
+### Wikipedia/Wikidata rankings
Wikipedia can be used as an optional auxiliary data source to help indicate
the importance of OSM features. Nominatim will work without this information
This data is available as a binary download:
cd $NOMINATIM_SOURCE_DIR/data
- wget https://www.nominatim.org/data/wikipedia_article.sql.bin
- wget https://www.nominatim.org/data/wikipedia_redirect.sql.bin
+ wget https://www.nominatim.org/data/wikimedia-importance.sql.gz
-Combined the 2 files are around 1.5GB and add around 30GB to the install
-size of Nominatim. They also increase the install time by an hour or so.
+The file is about 400MB and adds around 4GB to Nominatim database.
-*NOTE:* you'll need to download the Wikipedia rankings before performing
-the initial import of the data if you want the rankings applied to the
-loaded data.
+*NOTE:* if you forgot to download the wikipedia rankings, you can also add
+them after the import by running `./utils/setup.php --import-wikipedia-articles`
+and then `./utils/update.php --recompute-importance`.
### Great Britain, USA postcodes
In its default setup Nominatim is configured to import the full OSM data
set for the entire planet. Such a setup requires a powerful machine with
-at least 32GB of RAM and around 800GB of SSD hard disks. Depending on your
+at least 64GB of RAM and around 800GB of SSD hard disks. Depending on your
use case there are various ways to reduce the amount of data imported. This
section discusses these methods. They can also be combined.
To give you an idea of the impact of using the different styles, the table
below gives rough estimates of the final database size after import of a
2018 planet and after using the `--drop` option. It also shows the time
-needed for the import on a machine with 32GB RAM, 4 CPUS and SSDs. Note that
+needed for the import on a machine with 64GB RAM, 4 CPUS and SSDs. Note that
the given sizes are just an estimate meant for comparison of style requirements.
Your planet import is likely to be larger as the OSM data grows with time.
address | 59h | 500 GB | 260 GB
full | 80h | 575 GB | 300 GB
-You can also customize the styles further. For an description of the
+You can also customize the styles further. For a description of the
style format see [the development section](../develop/Import.md).
## Initial import of the data
from the build directory:
```sh
-./utils/setup.php --osm-file <data file> --all [--osm2pgsql-cache 28000] 2>&1 | tee setup.log
+./utils/setup.php --osm-file <data file> --all 2>&1 | tee setup.log
```
-The `--osm2pgsql-cache` parameter is optional but strongly recommended for
-planet imports. It sets the node cache size for the osm2pgsql import part
-(see `-C` parameter in osm2pgsql help). As a rule of thumb, this should be
-about the same size as the file you are importing but never more than
-2/3 of RAM available. If your machine starts swapping reduce the size.
-
-Computing word frequency for search terms can improve the performance of
-forward geocoding in particular under high load as it helps PostgreSQL's query
-planner to make the right decisions. To recompute word counts run:
+***Note for full planet imports:*** Even on a perfectly configured machine
+the import of a full planet takes at least 2 days. Once you see messages
+with `Rank .. ETA` appear, the indexing process has started. This part takes
+the most time. There are 30 ranks to process. Rank 26 and 30 are the most complex.
+They take each about a third of the total import time. If you have not reached
+rank 26 after two days of import, it is worth revisiting your system
+configuration as it may not be optimal for the import.
+
+### Notes on memory usage
+
+In the first step of the import Nominatim uses osm2pgsql to load the OSM data
+into the PostgreSQL database. This step is very demanding in terms of RAM usage.
+osm2pgsql and PostgreSQL are running in parallel at this point. PostgreSQL
+blocks at least the part of RAM that has been configured with the
+`shared_buffers` parameter during [PostgreSQL tuning](Installation#PostgreSQL_tuning)
+and needs some memory on top of that. osm2pgsql needs at least 2GB of RAM for
+its internal data structures, potentially more when it has to process very large
+relations. In addition it needs to maintain a cache for node locations. The size
+of this cache can be configured with the parameter `--osm2pgsql-cache`.
+
+When importing with a flatnode file, it is best to disable the node cache
+completely and leave the memory for the flatnode file. Nominatim will do this
+by default, so you do not need to configure anything in this case.
+
+For imports without a flatnode file, set `--osm2pgsql-cache` approximately to
+the size of the OSM pbf file (in MB) you are importing. Make sure you leave
+enough RAM for PostgreSQL and osm2pgsql as mentioned above. If the system starts
+swapping or you are getting out-of-memory errors, reduce the cache size or
+even consider using a flatnode file.
+
+## Tuning the database
+
+Accurate word frequency information for search terms helps PostgreSQL's query
+planner to make the right decisions. Recomputing them can improve the performance
+of forward geocoding in particular under high load. To recompute word counts run:
```sh
./utils/update.php --recompute-word-counts
./utils/specialphrases.php --wiki-import > specialphrases.sql
psql -d nominatim -f specialphrases.sql
-Note that this command downloads the phrases from the wiki link above.
+Note that this command downloads the phrases from the wiki link above. You
+need internet access for the step.
## Installing Tiger housenumber data for the US
* a recent C++ compiler
Nominatim comes with its own version of osm2pgsql. See the
-osm2pgsql README for additional dependencies required for compiling osm2pgsql.
+[osm2pgsql README](https://github.com/openstreetmap/osm2pgsql/blob/master/README.md#building)
+for additional dependencies required for compiling osm2pgsql.
For running tests:
* [PHP](https://php.net) (7.0 or later)
* PHP-pgsql
* PHP-intl (bundled with PHP)
- * [PEAR::DB](https://pear.php.net/package/DB)
* a webserver (apache or nginx are recommended)
For running continuous updates:
### Hardware
A minimum of 2GB of RAM is required or installation will fail. For a full
-planet import 32GB of RAM or more are strongly recommended.
+planet import 64GB of RAM or more are strongly recommended. Do not report
+out of memory problems if you have less than 64GB RAM.
-For a full planet install you will need at least 700GB of hard disk space
+For a full planet install you will need at least 800GB of hard disk space
(take into account that the OSM database is growing fast). SSD disks
will help considerably to speed up import and queries.
-On a 6-core machine with 32GB RAM and SSDs the import of a full planet takes
-a bit more than 2 days. Without SSDs 7-8 days are more realistic.
-
+Even on a well configured machine the import of a full planet takes
+at least 2 days. Without SSDs 7-8 days are more realistic.
## Setup of the server
make best use of your hardware. You should tune the following parameters in
your `postgresql.conf` file.
- shared_buffers (2GB)
- maintenance_work_mem (10GB)
- work_mem (50MB)
- effective_cache_size (24GB)
+ shared_buffers = 2GB
+ maintenance_work_mem = (10GB)
+ autovacuum_work_mem = 2GB
+ work_mem = (50MB)
+ effective_cache_size = (24GB)
synchronous_commit = off
checkpoint_segments = 100 # only for postgresql <= 9.4
+ max_wal_size = 1GB # postgresql > 9.4
checkpoint_timeout = 10min
checkpoint_completion_target = 0.9
The numbers in brackets behind some parameters seem to work fine for
-32GB RAM machine. Adjust to your setup.
+64GB RAM machine. Adjust to your setup. A higher number for `max_wal_size`
+means that PostgreSQL needs to run checkpoints less often but it does require
+the additional space on your disk.
For the initial import, you should also set:
SQL statements should be executed from the PostgreSQL commandline. Execute
`psql nominatim` to enter command line mode.
+## 3.4.0 -> master
+
+### New Wikipedia/Wikidata importance tables
+
+The `wikipedia_*` tables have a new format that also includes references to
+Wikidata. You need to update the computation functions and the tables as
+follows:
+
+ * download the new Wikipedia tables as described in the import section
+ * reimport the tables: `./utils/setup.php --import-wikipedia-articles`
+ * update the functions: `./utils/setup.php --create-functions --enable-diff-updates`
+ * compute importance: `./utils/update.php --recompute-importance`
+
+The last step takes about 10 hours on the full planet.
+
## 3.3.0 -> 3.4.0
### Reorganisation of location_area_country table
The possible fields are:
- * `place_id` - reference to the Nominatim internal database ID (see notes below)
+ * `place_id` - reference to the Nominatim internal database ID ([see notes](#place_id-is-not-a-persistent-id))
* `osm_type`, `osm_id` - reference to the OSM object
- * `boundingbox` - area of corner coordinates
+ * `boundingbox` - area of corner coordinates ([see notes](#boundingbox))
* `lat`, `lon` - latitude and longitude of the centroid of the object
* `display_name` - full comma-separated address
* `class`, `type` - key and value of the main OSM tag
The feature list has the following fields:
- * `place_id` - reference to the Nominatim internal database ID (see notes below)
+ * `place_id` - reference to the Nominatim internal database ID ([see notes](#place_id-is-not-a-persistent-id))
* `osm_type`, `osm_id` - reference to the OSM object
* `category`, `type` - key and value of the main OSM tag
* `display_name` - full comma-separated address
The place information can be found in the `result` element. The attributes of that element contain:
- * `place_id` - reference to the Nominatim internal database ID (see notes below)
+ * `place_id` - reference to the Nominatim internal database ID ([see notes](#place_id-is-not-a-persistent-id))
* `osm_type`, `osm_id` - reference to the OSM object
* `ref` - content of `ref` tag if it exists
* `lat`, `lon` - latitude and longitude of the centroid of the object
- * `boundingbox` - comma-separated list of corner coordinates
+ * `boundingbox` - comma-separated list of corner coordinates ([see notes](#boundingbox))
The full address of the result can be found in the content of the
`result` element as a comma-separated list.
The place information can be found in the `place` elements, of which there may
be more than one. The attributes of that element contain:
- * `place_id` - reference to the Nominatim internal database ID (see notes below)
+ * `place_id` - reference to the Nominatim internal database ID ([see notes](#place_id-is-not-a-persistent-id))
* `osm_type`, `osm_id` - reference to the OSM object
* `ref` - content of `ref` tag if it exists
* `lat`, `lon` - latitude and longitude of the centroid of the object
- * `boundingbox` - comma-separated list of corner coordinates
+ * `boundingbox` - comma-separated list of corner coordinates ([see notes](#boundingbox))
* `place_rank` - class search rank
* `display_name` - full comma-separated address
* `class`, `type` - key and value of the main OSM tag
Comma separated list of min latitude, max latitude, min longitude, max longitude.
The whole planet would be `-90,90,-180,180`.
+
+Can we used to pan and center the map on the result, for example with leafletjs
+mapping library
+`map.fitBounds([[bbox[0],bbox[2]],[bbox[1],bbox[3]]], {padding: [20, 20], maxzoom: 16});`
+
+Bounds crossing the antimeridian have a min latitude -180 and max latitude 180,
+essentially covering the planet (See [issue 184](https://github.com/openstreetmap/Nominatim/issues/184)).
# Documentation Pages
-The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/]()
+The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
-To preview local changes:
+To preview local changes, first install MkDocs
-1. Install MkDocs
+```
+pip3 install --user mkdocs
+```
- ```
- pip3 install --user mkdocs
- ```
+Then go to the build directory and run
-2. In build directory run
+```
+make doc
+INFO - Cleaning site directory
+INFO - Building documentation to directory: /home/vagrant/build/site-html
+```
- ```
- make doc
- INFO - Cleaning site directory
- INFO - Building documentation to directory: /home/vagrant/build/site-html
- ```
+This runs `mkdocs build` plus extra transformation of some files and adds
+symlinks (see `CMakeLists.txt` for the exact steps).
- This runs `mkdocs build` plus extra transformion of some files and adds symlinks (see `CMakeLists.txt` for the exact steps).
+Now you can start webserver for local testing
+```
+build> mkdocs serve
+[server:296] Serving on http://127.0.0.1:8000
+[handlers:62] Start watching changes
+```
-3. Start webserver for local testing
+If you develop inside a Vagrant virtual machine:
- ```
- mkdocs serve
- [server:296] Serving on http://127.0.0.1:8000
- [handlers:62] Start watching changes
- ```
-
- If you develop inside a Vagrant virtual machine:
- * add port forwarding to your Vagrantfile, e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000`
- * use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost
- IP does not get forwarded.
+ * add port forwarding to your Vagrantfile,
+ e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000`
+ * use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost
+ IP does not get forwarded.
--- /dev/null
+# Postcodes in Nominatim
+
+The blog post
+[Nominatim and Postcodes](https://www.openstreetmap.org/user/lonvia/diary/43143)
+describes the handling implemented since Nominatim 3.1.
+
+Postcode centroids (aka 'calculated postcodes') are generated by looking at all
+postcodes of a country, grouping them and calculating the geometric centroid.
+There is currently no logic to deal with extreme outliers (typos or other
+mistakes in OSM data). There is also no check if a postcodes adheres to a
+country's format, e.g. if Swiss postcodes are 4 digits.
+
+
+## Regular updating calculated postcodes
+
+The script to rerun the calculation is
+`build/utils/update.php --calculate-postcodes`
+and runs once per night on nominatim.openstreetmap.org.
+
+
+## Finding places that share a specific postcode
+
+In the Nominatim database run
+
+```sql
+SELECT address->'postcode' as pc,
+ osm_type, osm_id, class, type,
+ st_x(centroid) as lon, st_y(centroid) as lat
+FROM placex
+WHERE country_code='fr'
+ AND upper(trim (both ' ' from address->'postcode')) = '33210';
+```
+
+Alternatively on [Overpass](https://overpass-turbo.eu/) run the following query
+
+```
+[out:json][timeout:250];
+area["name"="France"]->.boundaryarea;
+(
+nwr(area.boundaryarea)["addr:postcode"="33210"];
+);
+out body;
+>;
+out skel qt;
+```
- 'Overview' : 'develop/overview.md'
- 'OSM Data Import' : 'develop/Import.md'
- 'Place Ranking' : 'develop/Ranking.md'
+ - 'Postcodes' : 'develop/Postcodes.md'
- 'Documentation' : 'develop/Documentation.md'
- 'External Data Sources':
- 'Overview' : 'data-sources/overview.md'
{
// https://secure.php.net/manual/en/ref.pdo-pgsql.connection.php
$aInfo = array();
- if (preg_match('/^pgsql:(.+)/', $sDSN, $aMatches)) {
+ if (preg_match('/^pgsql:(.+)$/', $sDSN, $aMatches)) {
foreach (explode(';', $aMatches[1]) as $sKeyVal) {
list($sKey, $sVal) = explode('=', $sKeyVal, 2);
if ($sKey == 'host') $sKey = 'hostspec';
// Convert database DSN to psql parameters
$aDSNInfo = \Nominatim\DB::parseDSN(CONST_Database_DSN);
if (!isset($aDSNInfo['port']) || !$aDSNInfo['port']) $aDSNInfo['port'] = 5432;
- $sCMD = 'psql -p '.$aDSNInfo['port'].' -d '.$aDSNInfo['database'];
+ $sCMD = 'psql'
+ .' -p '.escapeshellarg($aDSNInfo['port'])
+ .' -d '.escapeshellarg($aDSNInfo['database']);
if (isset($aDSNInfo['hostspec']) && $aDSNInfo['hostspec']) {
- $sCMD .= ' -h ' . $aDSNInfo['hostspec'];
+ $sCMD .= ' -h ' . escapeshellarg($aDSNInfo['hostspec']);
}
if (isset($aDSNInfo['username']) && $aDSNInfo['username']) {
- $sCMD .= ' -U ' . $aDSNInfo['username'];
+ $sCMD .= ' -U ' . escapeshellarg($aDSNInfo['username']);
}
$aProcEnv = null;
if (isset($aDSNInfo['password']) && $aDSNInfo['password']) {
warn('resetting threads to '.$this->iInstances);
}
- // Assume we can steal all the cache memory in the box (unless told otherwise)
if (isset($aCMDResult['osm2pgsql-cache'])) {
$this->iCacheMemory = $aCMDResult['osm2pgsql-cache'];
+ } elseif (!is_null(CONST_Osm2pgsql_Flatnode_File)) {
+ // When flatnode files are enabled then disable cache per default.
+ $this->iCacheMemory = 0;
} else {
+ // Otherwise: Assume we can steal all the cache memory in the box.
$this->iCacheMemory = getCacheMemoryMB();
}
fail('database already exists ('.CONST_Database_DSN.')');
}
- $sCreateDBCmd = 'createdb -E UTF-8 -p '.$this->aDSNInfo['port'].' '.$this->aDSNInfo['database'];
+ $sCreateDBCmd = 'createdb -E UTF-8'
+ .' -p '.escapeshellarg($this->aDSNInfo['port'])
+ .' '.escapeshellarg($this->aDSNInfo['database']);
if (isset($this->aDSNInfo['username'])) {
- $sCreateDBCmd .= ' -U '.$this->aDSNInfo['username'];
+ $sCreateDBCmd .= ' -U '.escapeshellarg($this->aDSNInfo['username']);
}
if (isset($this->aDSNInfo['hostspec'])) {
- $sCreateDBCmd .= ' -h '.$this->aDSNInfo['hostspec'];
+ $sCreateDBCmd .= ' -h '.escapeshellarg($this->aDSNInfo['hostspec']);
}
$result = $this->runWithPgEnv($sCreateDBCmd);
if ($this->bNoPartitions) {
$this->pgsqlRunScript('update country_name set partition = 0');
}
-
- // the following will be needed by createFunctions later but
- // is only defined in the subsequently called createTables
- // Create dummies here that will be overwritten by the proper
- // versions in create-tables.
- $this->pgsqlRunScript('CREATE TABLE IF NOT EXISTS place_boundingbox ()');
- $this->pgsqlRunScript('CREATE TYPE wikipedia_article_match AS ()', false);
}
public function importData($sOSMFile)
fail("osm2pgsql not found in '$osm2pgsql'");
}
- $osm2pgsql .= ' -S '.CONST_Import_Style;
+ $osm2pgsql .= ' -S '.escapeshellarg(CONST_Import_Style);
if (!is_null(CONST_Osm2pgsql_Flatnode_File) && CONST_Osm2pgsql_Flatnode_File) {
- $osm2pgsql .= ' --flat-nodes '.CONST_Osm2pgsql_Flatnode_File;
+ $osm2pgsql .= ' --flat-nodes '.escapeshellarg(CONST_Osm2pgsql_Flatnode_File);
}
if (CONST_Tablespace_Osm2pgsql_Data)
- $osm2pgsql .= ' --tablespace-slim-data '.CONST_Tablespace_Osm2pgsql_Data;
+ $osm2pgsql .= ' --tablespace-slim-data '.escapeshellarg(CONST_Tablespace_Osm2pgsql_Data);
if (CONST_Tablespace_Osm2pgsql_Index)
- $osm2pgsql .= ' --tablespace-slim-index '.CONST_Tablespace_Osm2pgsql_Index;
+ $osm2pgsql .= ' --tablespace-slim-index '.escapeshellarg(CONST_Tablespace_Osm2pgsql_Index);
if (CONST_Tablespace_Place_Data)
- $osm2pgsql .= ' --tablespace-main-data '.CONST_Tablespace_Place_Data;
+ $osm2pgsql .= ' --tablespace-main-data '.escapeshellarg(CONST_Tablespace_Place_Data);
if (CONST_Tablespace_Place_Index)
- $osm2pgsql .= ' --tablespace-main-index '.CONST_Tablespace_Place_Index;
+ $osm2pgsql .= ' --tablespace-main-index '.escapeshellarg(CONST_Tablespace_Place_Index);
$osm2pgsql .= ' -lsc -O gazetteer --hstore --number-processes 1';
- $osm2pgsql .= ' -C '.$this->iCacheMemory;
- $osm2pgsql .= ' -P '.$this->aDSNInfo['port'];
+ $osm2pgsql .= ' -C '.escapeshellarg($this->iCacheMemory);
+ $osm2pgsql .= ' -P '.escapeshellarg($this->aDSNInfo['port']);
if (isset($this->aDSNInfo['username'])) {
- $osm2pgsql .= ' -U '.$this->aDSNInfo['username'];
+ $osm2pgsql .= ' -U '.escapeshellarg($this->aDSNInfo['username']);
}
if (isset($this->aDSNInfo['hostspec'])) {
- $osm2pgsql .= ' -H '.$this->aDSNInfo['hostspec'];
+ $osm2pgsql .= ' -H '.escapeshellarg($this->aDSNInfo['hostspec']);
}
- $osm2pgsql .= ' -d '.$this->aDSNInfo['database'].' '.$sOSMFile;
+ $osm2pgsql .= ' -d '.escapeshellarg($this->aDSNInfo['database']).' '.escapeshellarg($sOSMFile);
$this->runWithPgEnv($osm2pgsql);
public function importWikipediaArticles()
{
- $sWikiArticlesFile = CONST_Wikipedia_Data_Path.'/wikipedia_article.sql.bin';
- $sWikiRedirectsFile = CONST_Wikipedia_Data_Path.'/wikipedia_redirect.sql.bin';
+ $sWikiArticlesFile = CONST_Wikipedia_Data_Path.'/wikimedia-importance.sql.gz';
if (file_exists($sWikiArticlesFile)) {
- info('Importing wikipedia articles');
- $this->pgsqlRunDropAndRestore($sWikiArticlesFile);
- } else {
- warn('wikipedia article dump file not found - places will have default importance');
- }
- if (file_exists($sWikiRedirectsFile)) {
- info('Importing wikipedia redirects');
- $this->pgsqlRunDropAndRestore($sWikiRedirectsFile);
+ info('Importing wikipedia articles and redirects');
+ $this->pgExec('DROP TABLE IF EXISTS wikipedia_article');
+ $this->pgExec('DROP TABLE IF EXISTS wikipedia_redirect');
+ $this->pgsqlRunScriptFile($sWikiArticlesFile);
} else {
- warn('wikipedia redirect dump file not found - some place importance values may be missing');
+ warn('wikipedia importance dump file not found - places will have default importance');
}
}
echo '.';
$this->pgExec('TRUNCATE place_addressline');
echo '.';
- $this->pgExec('TRUNCATE place_boundingbox');
- echo '.';
$this->pgExec('TRUNCATE location_area');
echo '.';
if (!$this->dbReverseOnly()) {
{
info('Import Tiger data');
+ $aFilenames = glob(CONST_Tiger_Data_Path.'/*.sql');
+ info('Found '.count($aFilenames).' SQL files in path '.CONST_Tiger_Data_Path);
+ if (empty($aFilenames)) return;
+
$sTemplate = file_get_contents(CONST_BasePath.'/sql/tiger_import_start.sql');
$sTemplate = str_replace('{www-user}', CONST_Database_Web_User, $sTemplate);
$sTemplate = $this->replaceTablespace(
pg_ping($aDBInstances[$i]);
}
- foreach (glob(CONST_Tiger_Data_Path.'/*.sql') as $sFile) {
+ foreach ($aFilenames as $sFile) {
echo $sFile.': ';
$hFile = fopen($sFile, 'r');
$sSQL = fgets($hFile, 100000);
public function index($bIndexNoanalyse)
{
$sOutputFile = '';
- $sBaseCmd = CONST_InstallPath.'/nominatim/nominatim -i -d '.$this->aDSNInfo['database'].' -P '
- .$this->aDSNInfo['port'].' -t '.$this->iInstances.$sOutputFile;
+ $sBaseCmd = CONST_InstallPath.'/nominatim/nominatim -i'
+ .' -d '.escapeshellarg($this->aDSNInfo['database'])
+ .' -P '.escapeshellarg($this->aDSNInfo['port'])
+ .' -t '.escapeshellarg($this->iInstances.$sOutputFile);
if (isset($this->aDSNInfo['hostspec'])) {
- $sBaseCmd .= ' -H '.$this->aDSNInfo['hostspec'];
+ $sBaseCmd .= ' -H '.escapeshellarg($this->aDSNInfo['hostspec']);
}
if (isset($this->aDSNInfo['username'])) {
- $sBaseCmd .= ' -U '.$this->aDSNInfo['username'];
+ $sBaseCmd .= ' -U '.escapeshellarg($this->aDSNInfo['username']);
}
info('Index ranks 0 - 4');
}
}
- private function pgsqlRunDropAndRestore($sDumpFile)
- {
- $sCMD = 'pg_restore -p '.$this->aDSNInfo['port'].' -d '.$this->aDSNInfo['database'].' --no-owner -Fc --clean '.$sDumpFile;
- if ($this->oDB->getPostgresVersion() >= 9.04) {
- $sCMD .= ' --if-exists';
- }
- if (isset($this->aDSNInfo['hostspec'])) {
- $sCMD .= ' -h '.$this->aDSNInfo['hostspec'];
- }
- if (isset($this->aDSNInfo['username'])) {
- $sCMD .= ' -U '.$this->aDSNInfo['username'];
- }
-
- $this->runWithPgEnv($sCMD);
- }
-
private function pgsqlRunScript($sScript, $bfatal = true)
{
runSQLScript(
{
if (!file_exists($sFilename)) fail('unable to find '.$sFilename);
- $sCMD = 'psql -p '.$this->aDSNInfo['port'].' -d '.$this->aDSNInfo['database'];
+ $sCMD = 'psql'
+ .' -p '.escapeshellarg($this->aDSNInfo['port'])
+ .' -d '.escapeshellarg($this->aDSNInfo['database']);
if (!$this->bVerbose) {
$sCMD .= ' -q';
}
if (isset($this->aDSNInfo['hostspec'])) {
- $sCMD .= ' -h '.$this->aDSNInfo['hostspec'];
+ $sCMD .= ' -h '.escapeshellarg($this->aDSNInfo['hostspec']);
}
if (isset($this->aDSNInfo['username'])) {
- $sCMD .= ' -U '.$this->aDSNInfo['username'];
+ $sCMD .= ' -U '.escapeshellarg($this->aDSNInfo['username']);
}
$aProcEnv = null;
if (isset($this->aDSNInfo['password'])) {
1 => array('pipe', 'w'),
2 => array('file', '/dev/null', 'a')
);
- $hGzipProcess = proc_open('zcat '.$sFilename, $aDescriptors, $ahGzipPipes);
+ $hGzipProcess = proc_open('zcat '.escapeshellarg($sFilename), $aDescriptors, $ahGzipPipes);
if (!is_resource($hGzipProcess)) fail('unable to start zcat');
$aReadPipe = $ahGzipPipes[1];
fclose($ahGzipPipes[0]);
} else {
- $sCMD .= ' -f '.$sFilename;
+ $sCMD .= ' -f '.escapeshellarg($sFilename);
$aReadPipe = array('pipe', 'r');
}
$aDescriptors = array(
--- /dev/null
+<?php
+ header("content-type: text/html; charset=UTF-8");
+?>
+<?php include(CONST_BasePath.'/lib/template/includes/html-header.php'); ?>
+ <link href="css/common.css" rel="stylesheet" type="text/css" />
+ <link href="css/details.css" rel="stylesheet" type="text/css" />
+</head>
+
+
+<body id="details-index-page">
+ <div class="container">
+ <div class="row">
+ <div class="col-md-12">
+
+ <h1>Show details for place</h1>
+
+ <div class="search-form">
+ <h4>Search by place id</h4>
+
+ <form class="form-inline" action="details.php">
+ <input type="edit" class="form-control input-sm" pattern="^[0-9]+$" name="place_id" placeholder="12345" />
+ <input type="submit" class="btn btn-primary btn-sm" value="Show" />
+ </form>
+ </div>
+
+ <div class="search-form">
+ <h4>Search by OSM type and OSM id</h4>
+
+ <form id="form-by-type-and-id" class="form-inline" action="details.php">
+ <input type="edit" class="form-control input-sm" pattern="^[NWR][0-9]+$" placeholder="N123 or W123 or R123" />
+ <input type="hidden" name="osmtype" />
+ <input type="hidden" name="osmid" />
+ <input type="submit" class="btn btn-primary btn-sm" value="Show" />
+ </form>
+ </div>
+
+ <div class="search-form">
+ <h4>Search by openstreetmap.org URL</h4>
+
+ <form id="form-by-osm-url" class="form-inline" action="details.php">
+ <input type="edit" class="form-control input-sm" pattern=".*openstreetmap.*" placeholder="https://www.openstreetmap.org/relation/123" />
+ <input type="hidden" name="osmtype" />
+ <input type="hidden" name="osmid" />
+ <input type="submit" class="btn btn-primary btn-sm" value="Show" />
+ </form>
+ </div>
+
+ </div>
+ </div>
+ </div>
+
+
+ <?php include(CONST_BasePath.'/lib/template/includes/html-footer.php'); ?>
+</body>
+</html>
<meta name="viewport" content="width=device-width, initial-scale=1">
<base href="<?php echo CONST_Website_BaseURL;?>" />
- <link href="nominatim.xml" rel="search" title="Nominatim Search" type="application/opensearchdescription+xml" />
<link href="css/leaflet.css" rel="stylesheet" />
<link href="css/Control.Minimap.min.css" rel="stylesheet" />
<link href="css/bootstrap-theme.min.css" rel="stylesheet" />
END LOOP;
NEW.importance := null;
- select language||':'||title,importance from get_wikipedia_match(NEW.extratags, NEW.country_code) INTO NEW.wikipedia,NEW.importance;
- IF NEW.importance IS NULL THEN
- select language||':'||title,importance from wikipedia_article where osm_type = NEW.osm_type and osm_id = NEW.osm_id order by importance desc limit 1 INTO NEW.wikipedia,NEW.importance;
- END IF;
+ SELECT wikipedia, importance
+ FROM compute_importance(NEW.extratags, NEW.country_code, NEW.osm_type, NEW.osm_id)
+ INTO NEW.wikipedia,NEW.importance;
--DEBUG: RAISE WARNING 'Importance computed from wikipedia: %', NEW.importance;
-- mark the linked place (excludes from search results)
UPDATE placex set linked_place_id = NEW.place_id where place_id = linkedPlacex.place_id;
- -- keep a note of the node id in case we need it for wikipedia in a bit
- linked_node_id := linkedPlacex.osm_id;
- select language||':'||title,importance from get_wikipedia_match(linkedPlacex.extratags, NEW.country_code) INTO linked_wikipedia,linked_importance;
+ select wikipedia, importance
+ FROM compute_importance(linkedPlacex.extratags, NEW.country_code,
+ 'N', linkedPlacex.osm_id)
+ INTO linked_wikipedia,linked_importance;
--DEBUG: RAISE WARNING 'Linked label member';
END LOOP;
-- mark the linked place (excludes from search results)
UPDATE placex set linked_place_id = NEW.place_id where place_id = linkedPlacex.place_id;
- -- keep a note of the node id in case we need it for wikipedia in a bit
- linked_node_id := linkedPlacex.osm_id;
- select language||':'||title,importance from get_wikipedia_match(linkedPlacex.extratags, NEW.country_code) INTO linked_wikipedia,linked_importance;
+ select wikipedia, importance
+ FROM compute_importance(linkedPlacex.extratags, NEW.country_code,
+ 'N', linkedPlacex.osm_id)
+ INTO linked_wikipedia,linked_importance;
--DEBUG: RAISE WARNING 'Linked admin_center';
END IF;
-- mark the linked place (excludes from search results)
UPDATE placex set linked_place_id = NEW.place_id where place_id = linkedPlacex.place_id;
- -- keep a note of the node id in case we need it for wikipedia in a bit
- linked_node_id := linkedPlacex.osm_id;
- select language||':'||title,importance from get_wikipedia_match(linkedPlacex.extratags, NEW.country_code) INTO linked_wikipedia,linked_importance;
+ select wikipedia, importance
+ FROM compute_importance(linkedPlacex.extratags, NEW.country_code,
+ 'N', linkedPlacex.osm_id)
+ INTO linked_wikipedia,linked_importance;
--DEBUG: RAISE WARNING 'Linked named place';
END LOOP;
END IF;
(NEW.importance is null or NEW.importance < linked_importance) THEN
NEW.importance = linked_importance;
END IF;
-
- -- Still null? how about looking it up by the node id
- IF NEW.importance IS NULL THEN
- --DEBUG: RAISE WARNING 'Looking up importance by linked node id';
- select language||':'||title,importance from wikipedia_article where osm_type = 'N'::char(1) and osm_id = linked_node_id order by importance desc limit 1 INTO NEW.wikipedia,NEW.importance;
- END IF;
-
END IF;
-- make sure all names are in the word table
$$
LANGUAGE plpgsql IMMUTABLE;
-DROP TYPE wikipedia_article_match CASCADE;
+DROP TYPE IF EXISTS wikipedia_article_match CASCADE;
create type wikipedia_article_match as (
language TEXT,
title TEXT,
$$
LANGUAGE plpgsql;
+DROP TYPE IF EXISTS place_importance CASCADE;
+create type place_importance as (
+ importance FLOAT,
+ wikipedia TEXT
+);
+
+CREATE OR REPLACE FUNCTION compute_importance(extratags HSTORE, country_code varchar(2), osm_type varchar(1), osm_id BIGINT)
+ RETURNS place_importance
+ AS $$
+DECLARE
+ match RECORD;
+ result place_importance;
+BEGIN
+ FOR match IN SELECT * FROM get_wikipedia_match(extratags, country_code)
+ WHERE language is not NULL
+ LOOP
+ result.importance := match.importance;
+ result.wikipedia := match.language || ':' || match.title;
+ RETURN result;
+ END LOOP;
+
+ IF extratags ? 'wikidata' THEN
+ FOR match IN SELECT * FROM wikipedia_article
+ WHERE wd_page_title = extratags->'wikidata'
+ ORDER BY language = 'en' DESC, langcount DESC LIMIT 1 LOOP
+ result.importance := match.importance;
+ result.wikipedia := match.language || ':' || match.title;
+ RETURN result;
+ END LOOP;
+ END IF;
+
+ RETURN null;
+END;
+$$
+LANGUAGE plpgsql;
+
CREATE OR REPLACE FUNCTION quad_split_geometry(geometry GEOMETRY, maxarea FLOAT, maxdepth INTEGER)
RETURNS SETOF GEOMETRY
AS $$
lon double precision,
importance double precision,
osm_type character(1),
- osm_id bigint
+ osm_id bigint,
+ wd_page_title text,
+ instance_of text
);
ALTER TABLE ONLY wikipedia_article ADD CONSTRAINT wikipedia_article_pkey PRIMARY KEY (language, title);
CREATE INDEX idx_wikipedia_article_osm_id ON wikipedia_article USING btree (osm_type, osm_id);
| en |
Then result addresses contain
| ID | state |
- | 0 | Salto |
+ | 0 | Florida |
When sending json search query "25 de Mayo" with address
| accept-language | viewbox |
- | en | -56.35879,-34.18330,-56.31618,-34.20815 |
+ | en | -57.95468,-31.39261,-57.94741,-31.39490 |
Then result addresses contain
| ID | state |
- | 0 | Florida |
+ | 0 | Salto |
Scenario: viewboxes cannot be points
When sending json search query "foo"
(';password=' + self.db_pass) if self.db_pass else ''
))
f.write("@define('CONST_Osm2pgsql_Flatnode_File', null);\n")
+ f.write("@define('CONST_Import_Style', CONST_BasePath.'/settings/import-full.style');\n")
f.close()
def cleanup(self):
array('deduplicate', '', 0, 1, 0, 0, 'bool', 'Deduplicate tokens'),
array('recompute-word-counts', '', 0, 1, 0, 0, 'bool', 'Compute frequency of full-word search terms'),
array('update-address-levels', '', 0, 1, 0, 0, 'bool', 'Reimport address level configuration (EXPERT)'),
+ array('recompute-importance', '', 0, 1, 0, 0, 'bool', 'Recompute place importances'),
array('no-npi', '', 0, 1, 0, 0, 'bool', '(obsolete)'),
);
$oAlParser->createTable($oDB, 'address_levels');
}
+if ($aResult['recompute-importance']) {
+ echo "Updating importance values for database.\n";
+ $oDB = new Nominatim\DB();
+ $oDB->connect();
+
+ $sSQL = 'ALTER TABLE placex DISABLE TRIGGER ALL;';
+ $sSQL .= 'UPDATE placex SET (wikipedia, importance) =';
+ $sSQL .= ' (SELECT wikipedia, importance';
+ $sSQL .= ' FROM compute_importance(extratags, country_code, osm_type, osm_id));';
+ $sSQL .= 'UPDATE placex s SET wikipedia = d.wikipedia, importance = d.importance';
+ $sSQL .= ' FROM placex d';
+ $sSQL .= ' WHERE s.place_id = d.linked_place_id and d.wikipedia is not null';
+ $sSQL .= ' and (s.wikipedia is null or s.importance < d.importance);';
+ $sSQL .= 'ALTER TABLE placex ENABLE TRIGGER ALL;';
+ $oDB->exec($sSQL);
+}
+
if ($aResult['import-osmosis'] || $aResult['import-osmosis-all']) {
//
if (strpos(CONST_Replication_Url, 'download.geofabrik.de') !== false && CONST_Replication_Update_Interval < 86400) {
sudo yum install -y epel-release
+# More repositories for postgresql 11 (CentOS default 'postgresql' is 9.2), postgis
+# and llvm-toolset (https://github.com/theory/pg-semver/issues/35)
+
+ sudo yum install -y https://download.postgresql.org/pub/repos/yum/11/redhat/rhel-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
+ sudo yum install -y centos-release-scl-rh
+
# Now you can install all packages needed for Nominatim:
#DOCS: :::sh
- sudo yum install -y postgresql-server postgresql-contrib postgresql-devel \
- postgis postgis-utils \
+
+ sudo yum install -y postgresql11-server postgresql11-contrib postgresql11-devel \
+ postgis25_11 postgis25_11-utils \
wget git cmake make gcc gcc-c++ libtool policycoreutils-python \
+ devtoolset-7 llvm-toolset-7 \
php-pgsql php php-intl libpqxx-devel \
proj-epsg bzip2-devel proj-devel libxml2-devel boost-devel \
expat-devel zlib-devel
# CentOS does not automatically create a database cluster. Therefore, start
# with initializing the database, then enable the server to start at boot:
- sudo postgresql-setup initdb
- sudo systemctl enable postgresql
+ echo 'PATH=$PATH:/usr/pgsql-11/bin' > .bash_profile
+ source .bash_profile
+
+ sudo /usr/pgsql-11/bin/postgresql-11-setup initdb
+ sudo systemctl enable postgresql-11
#
# Next tune the postgresql configuration, which is located in
#
# Now start the postgresql service after updating this config file.
- sudo systemctl restart postgresql
+ sudo systemctl restart postgresql-11
#
# Finally, we need to add two postgres users: one for the user that does
# download the country grid:
if [ ! -f data/country_osm_grid.sql.gz ]; then #DOCS: :::sh
- wget -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
+ wget --no-verbose -O data/country_osm_grid.sql.gz https://www.nominatim.org/data/country_grid.sql.gz
fi #DOCS:
# The code must be built in a separate directory. Create this directory,
margin: 10px 0;
}
+#details-index-page .search-form {
+ padding: 20px 10px;
+ margin: 2em 0;
+}
+#details-index-page .search-form h4 {
+ margin-top: 0;
+}
+#details-index-page .search-form .form-control{
+ width: 30em;
+}
+
footer {
text-align: center;
padding: 2em 0;
$sLanguagePrefArraySQL = $oDB->getArraySQL($oDB->getDBQuotedList($aLangPrefOrder));
+if ($sOutputFormat == 'html' && !$sPlaceId && !$sOsmType) {
+ include(CONST_BasePath.'/lib/template/details-index-html.php');
+ exit;
+}
+
if ($sOsmType && $iOsmId > 0) {
$sSQL = 'SELECT place_id FROM placex WHERE osm_type = :type AND osm_id = :id';
// osm_type and osm_id are not unique enough
});
L.tileLayer(nominatim_map_init.tile_url, {
- noWrap: true, // otherwise we end up with click coordinates like latitude -728
// moved to footer
attribution: (nominatim_map_init.tile_attribution || null ) //'© <a href="http://osm.org/copyright">OpenStreetMap</a> contributors'
}).addTo(map);
function display_map_position(mouse_lat_lng){
+ if (mouse_lat_lng) {
+ mouse_lat_lng = map.wrapLatLng(mouse_lat_lng);
+ }
html_mouse = "mouse position " + (mouse_lat_lng ? [mouse_lat_lng.lat.toFixed(5), mouse_lat_lng.lng.toFixed(5)].join(',') : '-');
html_click = "last click: " + (last_click_latlng ? [last_click_latlng.lat.toFixed(5),last_click_latlng.lng.toFixed(5)].join(',') : '-');
$('#map-position-inner').html([html_center,html_zoom,html_viewbox,html_click,html_mouse].join('<br/>'));
+ var center_lat_lng = map.wrapLatLng(map.getCenter());
var reverse_params = {
- lat: map.getCenter().lat.toFixed(5),
- lon: map.getCenter().lng.toFixed(5),
+ lat: center_lat_lng.lat.toFixed(5),
+ lon: center_lat_lng.lng.toFixed(5),
zoom: map.getZoom(),
format: 'html'
}
function map_viewbox_as_string() {
- // since .toBBoxString() doesn't round numbers
+ var bounds = map.getBounds();
+ var west = bounds.getWest();
+ var east = bounds.getEast();
+
+ if ((east - west) >= 360) { // covers more than whole planet
+ west = map.getCenter().lng-179.999;
+ east = map.getCenter().lng+179.999;
+ }
+ east = L.latLng(77, east).wrap().lng;
+ west = L.latLng(77, west).wrap().lng;
+
return [
- map.getBounds().getSouthWest().lng.toFixed(5), // left
- map.getBounds().getNorthEast().lat.toFixed(5), // top
- map.getBounds().getNorthEast().lng.toFixed(5), // right
- map.getBounds().getSouthWest().lat.toFixed(5) // bottom
+ west.toFixed(5), // left
+ bounds.getNorth().toFixed(5), // top
+ east.toFixed(5), // right
+ bounds.getSouth().toFixed(5) // bottom
].join(',');
}
function map_link_to_osm(){
if ( is_reverse_search ){
map.on('click', function(e){
$('form input[name=lat]').val( e.latlng.lat);
- $('form input[name=lon]').val( e.latlng.lng);
+ $('form input[name=lon]').val( e.latlng.wrap().lng);
$('form').submit();
});
});
+jQuery(document).ready(function(){
+
+ if ( !$('#details-index-page').length ){ return; }
+
+ $('#form-by-type-and-id,#form-by-osm-url').on('submit', function(e){
+ e.preventDefault();
+
+ var val = $(this).find('input[type=edit]').val();
+ var matches = val.match(/^\s*([NWR])(\d+)\s*$/i);
+
+ if (!matches) {
+ matches = val.match(/\/(relation|way|node)\/(\d+)\s*$/);
+ }
+
+ if (matches) {
+ $(this).find('input[name=osmtype]').val(matches[1].charAt(0).toUpperCase());
+ $(this).find('input[name=osmid]').val(matches[2]);
+ $(this).get(0).submit();
+ } else {
+ alert('invalid input');
+ }
+ });
+});
+
jQuery(document).ready(function(){
if ( !$('#details-page').length ){ return; }
projects / nominatim.git / commitdiff