* [pyosmium](https://osmcode.org/pyosmium/) (with Python 3)
-For running tests:
-
- * [behave](http://pythonhosted.org/behave/)
- * [nose](https://nose.readthedocs.io)
- * [phpunit](https://phpunit.de) >= 7.3
+For dependencies for running tests and building documentation, see
+the [Development section](../develop/Development-Environment.md).
### Hardware
--- /dev/null
+# Setting up Nominatim for Development
+
+This chapter gives an overview how to set up Nominatim for developement
+and how to run tests.
+
+!!!Important
+This guide assumes that you develop under the latest version of Ubuntu. You
+can of course also use your favourite distribution. You just might have to
+adapt the commands below slightly, in particular the commands for installing
+additional software.
+
+## Installing Nominatim
+
+The first step is to install Nominatim itself. Please follow the installation
+instructions in the [Admin section](../admin/Installation.md). You don't need
+to set up a webserver for development, the webserver that is included with PHP
+is sufficient.
+
+If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
+Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
+install an NFS daemon to enable directory sharing between host and guest. The
+following packages should get you started:
+
+ sudo apt install vagrant vagrant-libvirt libvirt-daemon nfs-kernel-server
+
+## Prerequisites for testing and documentation
+
+The Nominatim tests suite consists of behavioural tests (using behave) and
+unit tests (using PHPUnit). It has the following additional requirements:
+
+* [behave test framework](https://github.com/behave/behave) >= 1.2.5
+* [nose](https://nose.readthedocs.org)
+* [pytidylib](http://countergram.com/open-source/pytidylib)
+* [phpunit](https://phpunit.de) >= 7.3
+* [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
+
+The documentation is built with mkdocs:
+
+* [mkdocs](https://www.mkdocs.org/) >= 1.1.2
+
+### Installing prerequisites on Ubuntu/Debian
+
+Some of the Python packages require the newest version which is not yet
+available with the current distributions. Therefore it is recommended to
+install pip to get the newest versions.
+
+To install all necessary packages run:
+
+```sh
+sudo apt install php-cgi phpunit php-codesniffer \
+ python3-pip python3-setuptools python3-dev
+
+pip3 install --user behave nose mkdocs
+```
+
+The `mkdocs` executable will be located in `.local/bin`. You may have to add
+this directory to your path, for example by running:
+
+```
+echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
+```
+
+If your distribution does not have PHPUnit 7.3+, you can install it (as well
+as CodeSniffer) via composer:
+
+```
+sudo apt-get install composer
+composer global require "squizlabs/php_codesniffer=*"
+composer global require "phpunit/phpunit=8.*"
+```
+
+The binaries are found in `.config/composer/vendor/bin`. You need to add this
+to your PATH as well:
+
+```
+echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
+```
+
+
+## Executing Tests
+
+All tests are located in the `\test` directory.
+
+### Preparing the test database
+
+Some of the behavioural test expect a test database to be present. You need at
+least 2GB RAM and 10GB disk space to create the database.
+
+First create a separate directory for the test DB and Fetch the test planet
+data and the Tiger data for South Dakota:
+
+```
+mkdir testdb
+cd testdb
+wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
+wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*'
+```
+
+Configure and build Nominatim in the usual way:
+
+```
+cmake $USERNAME/Nominatim
+make
+```
+
+Copy the test settings:
+
+```
+cp $USERNAME/Nominatim/test/testdb/local.php settings/
+```
+
+Inspect the file to check that all settings are correct for your local setup.
+
+Now you can import the test database:
+
+```
+dropdb --if-exists test_api_nominatim
+./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log
+./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log
+./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log
+```
+
+### Running the tests
+
+To run all tests just go to the test directory and run make:
+
+```sh
+cd test
+make
+```
+
+To skip tests that require the test database, run `make no-test-db` instead.
+
+For more information about the structure of the tests and how to change and
+extend the test suite, see the [Testing chapter](Testing.md).
+
+## Documentation Pages
+
+The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is
+built using the [MkDocs](https://www.mkdocs.org/) static site generation
+framework. The master branch is automatically deployed every night on
+[https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
+
+To build the documentation, go to the build directory and run
+
+```
+make doc
+INFO - Cleaning site directory
+INFO - Building documentation to directory: /home/vagrant/build/site-html
+```
+
+This runs `mkdocs build` plus extra transformation of some files and adds
+symlinks (see `CMakeLists.txt` for the exact steps).
+
+Now you can start webserver for local testing
+
+```
+build> mkdocs serve
+[server:296] Serving on http://127.0.0.1:8000
+[handlers:62] Start watching changes
+```
+
+If you develop inside a Vagrant virtual machine, use a port that is forwarded
+to your host:
+
+```
+build> mkdocs serve --dev-addr 0.0.0.0:8088
+[server:296] Serving on http://0.0.0.0:8088
+[handlers:62] Start watching changes
+```
+++ /dev/null
-# Documentation Pages
-
-The [Nominatim documentation](https://nominatim.org/release-docs/develop/) is built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on under [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
-
-To preview local changes, first install MkDocs
-
-```
-pip3 install --user mkdocs
-```
-
-If `mkdocs` can't be found after the installation, the $PATH might have not
-been set correctly yet. Try opening a new terminal session.
-
-
-Then go to the build directory and run
-
-```
-make doc
-INFO - Cleaning site directory
-INFO - Building documentation to directory: /home/vagrant/build/site-html
-```
-
-This runs `mkdocs build` plus extra transformation of some files and adds
-symlinks (see `CMakeLists.txt` for the exact steps).
-
-Now you can start webserver for local testing
-
-```
-build> mkdocs serve
-[server:296] Serving on http://127.0.0.1:8000
-[handlers:62] Start watching changes
-```
-
-If you develop inside a Vagrant virtual machine:
-
- * add port forwarding to your Vagrantfile,
- e.g. `config.vm.network "forwarded_port", guest: 8000, host: 8000`
- * use `mkdocs serve --dev-addr 0.0.0.0:8000` because the default localhost
- IP does not get forwarded.
+++ /dev/null
-# Setup Test Environment
-
-To test changes and contribute to Nominatim you should be able to run
-the test suite(s). For many usecases it's enough to create a Vagrant
-virtual machine (see `VAGRANT.md`), import one small country into the
-database.
-
-## Prerequisites
-
-Nominatim supports a range of PHP versions and PHPUnit versions also
-move fast. We try to test against the newest stable PHP release and
-PHPUnit version even though we expect many Nominatim users will install
-older version on their production servers.
-
- * Python 3 (https://www.python.org/)
- * behave test framework >= 1.2.5 (https://github.com/behave/behave)
- * nose (https://nose.readthedocs.org)
- * pytidylib (http://countergram.com/open-source/pytidylib)
- * psycopg2 (http://initd.org/psycopg/)
-
-#### Ubuntu 20
-
- sudo apt-get install -y phpunit php-codesniffer php-cgi
- pip3 install --user behave nose
-
-#### Ubuntu 18
-
- pip3 install --user behave nose
-
- sudo apt-get install -y composer php-cgi php-cli php-mbstring php-xml zip unzip
-
- composer global require "squizlabs/php_codesniffer=*"
- sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
-
- composer global require "phpunit/phpunit=8.*"
- sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
-
-
-#### CentOS 7 or 8
-
- sudo dnf install -y php-dom php-mbstring
- pip3 install --user behave nose
-
- composer global require "squizlabs/php_codesniffer=*"
- sudo ln -s ~/.config/composer/vendor/bin/phpcs /usr/bin/
-
- composer global require "phpunit/phpunit=^7"
- sudo ln -s ~/.config/composer/vendor/bin/phpunit /usr/bin/
-
-## Run tests, code linter, code coverage
-
-See `README.md` in `test` subdirectory.
-This directory contains functional and unit tests for the Nominatim API.
+# Nominatim Test Suite
-Prerequisites
-=============
+This chapter describes the tests in the `/test` directory, how they are
+structured and how to extend them. For a quick introduction on how to run
+the tests, see the [Development setup chapter](Development-Environment.md).
-See `docs/develop/Setup.md`
-
-Overall structure
-=================
+## Overall structure
There are two kind of tests in this test suite. There are functional tests
which test the API interface using a BDD test framework and there are unit
+- testdb Base data for generating API test database
```
-PHP Unit Tests
-==============
+## PHP Unit Tests (`test/php`)
Unit tests can be found in the php/ directory and tests selected php functions.
Very low coverage.
It will use (and destroy) a local database 'nominatim_unit_tests'. You can set
a different connection string with e.g. UNIT_TEST_DSN='pgsql:dbname=foo_unit_tests'.
-BDD Functional Tests
-====================
+## BDD Functional Tests (`test/bdd`)
Functional tests are written as BDD instructions. For more information on
-the philosophy of BDD testing, see http://pythonhosted.org/behave/philosophy.html
+the philosophy of BDD testing, see the
+[Behave manual](http://pythonhosted.org/behave/philosophy.html).
+
+The following explanation assume that the reader is familiar with the BDD
+notations of features, scenarios and steps.
-Usage
------
+All possible steps can be found in the `steps` directory and should ideally
+be documented.
+
+### General Usage
To run the functional tests, do
run, otherwise the result is undefined.
Logging can be defined through command line parameters of behave itself. Check
-out `behave --help` for details. Also keep an eye out for the 'work-in-progress'
+out `behave --help` for details. Also have a look at the 'work-in-progress'
feature of behave which comes in handy when writing new tests.
-Writing Tests
--------------
-
-The following explanation assume that the reader is familiar with the BDD
-notations of features, scenarios and steps.
-
-All possible steps can be found in the `steps` directory and should ideally
-be documented.
-
### API Tests (`test/bdd/api`)
These tests are meant to test the different API endpoints and their parameters.
-They require a to import several datasets into a test database. You need at
-least 2GB RAM and 10GB discspace.
-
-
-1. Fetch the OSM planet extract (200MB). See end of document how it got created.
+They require a to import several datasets into a test database.
+See the [Development Setup chapter](Development-Environment.md#preparing-the-test-database)
+for instructions on how to set up this database.
- ```
- cd Nominatim/data
- mkdir testdb
- cd testdb
- wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
- ```
-
-2. Fetch `46*` (South Dakota) Tiger data
-
- ```
- cd Nominatim/data/testdb
- wget https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz
- tar xvf tiger2018-nominatim-preprocessed.tar.gz --wildcards --no-anchored '46*'
- rm tiger2018-nominatim-preprocessed.tar.gz
- ```
-
-3. Adapt build/settings/local.php settings:
-
- ```
- @define('CONST_Database_DSN', 'pgsql:dbname=test_api_nominatim');
- @define('CONST_Use_US_Tiger_Data', true);
- @define('CONST_Tiger_Data_Path', CONST_ExtraDataPath.'/testdb');
- @define('CONST_Wikipedia_Data_Path', CONST_BasePath.'/test/testdb');
- ```
+The official test dataset was derived from the 180924 planet (note: such
+file no longer exists at https://planet.openstreetmap.org/planet/2018/).
+Newer planets are likely to work as well but you may see isolated test
+failures where the data has changed.
-4. Import
+The official test dataset can always be downloaded from
+[nominatim.org](https://www.nominatim.org/data/test/nominatim-api-testdata.pbf)
+To recreate the input data for the test database run:
- ```
- LOGFILE=/tmp/nominatim-testdb.$$.log
- dropdb --if-exists test_api_nominatim
- ./utils/setup.php --all --osm-file ../Nominatim/data/testdb/nominatim-api-testdb.pbf 2>&1 | tee $LOGFILE
- ./utils/specialphrases.php --wiki-import > specialphrases.sql
- psql -d test_api_nominatim -f specialphrases.sql 2>&1 | tee -a $LOGFILE
- ./utils/setup.php --import-tiger-data 2>&1 | tee -a $LOGFILE
- ```
+```
+wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
+osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
+```
#### Code Coverage
apt-get install php-codecoverage php-xdebug
-The run the API tests as follows:
+Then run the API tests as follows:
behave api -DPHPCOV=<coverage output dir>
phpcov merge --html=<report output dir> <coverage output dir>
-### Indexing Tests (`test/bdd/db`)
+### DB Creation Tests (`test/bdd/db`)
These tests check the import and update of the Nominatim database. They do not
test the correctness of osm2pgsql. Each test will write some data into the `place`
These tests check that data is imported correctly into the place table. They
use the same template database as the Indexing tests, so the same remarks apply.
-
-
-
-How the test database extract was generated
--------------------------------------------
-The official test dataset was derived from the 180924 planet (note: such
-file no longer exists at https://planet.openstreetmap.org/planet/2018/).
-Newer planets are likely to work as well but you may see isolated test
-failures where the data has changed. To recreate the input data
-for the test database run:
-
- wget https://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/pbf/planet-180924.osm.pbf
- osmconvert planet-180924.osm.pbf -B=test/testdb/testdb.polys -o=testdb.pbf
- 'FAQ': 'api/Faq.md'
- 'Administration Guide':
- 'Basic Installation': 'admin/Installation.md'
- - 'Importing' : 'admin/Import.md'
- - 'Updating' : 'admin/Update.md'
- - 'Deploying' : 'admin/Deployment.md'
+ - 'Import' : 'admin/Import.md'
+ - 'Update' : 'admin/Update.md'
+ - 'Deploy' : 'admin/Deployment.md'
- 'Nominatim UI' : 'admin/Setup-Nominatim-UI.md'
- 'Advanced Installations' : 'admin/Advanced-Installations.md'
- 'Migration from older Versions' : 'admin/Migration.md'
- 'Troubleshooting' : 'admin/Faq.md'
- 'Developers Guide':
- - 'Overview' : 'develop/overview.md'
+ - 'Setup for Development' : 'develop/Development-Environment.md'
+ - 'Architecture Overview' : 'develop/overview.md'
- 'OSM Data Import' : 'develop/Import.md'
- 'Place Ranking' : 'develop/Ranking.md'
- 'Postcodes' : 'develop/Postcodes.md'
- - 'Setup Test Environment' : 'develop/Setup.md'
- - 'Building Documentation' : 'develop/Documentation.md'
+ - 'Testing' : 'develop/Testing.md'
- 'External Data Sources': 'develop/data-sources.md'
- 'Appendix':
- 'Installation on CentOS 7' : 'appendix/Install-on-Centos-7.md'
all: bdd php
+no-test-db: bdd-no-test-db php
bdd:
cd bdd && behave -DREMOVE_TEMPLATE=1
+bdd-no-test-db:
+ cd bdd && behave -DREMOVE_TEMPLATE=1 db osm2pgsql
+
php:
cd php && phpunit ./
-.PHONY: bdd php
+
+.PHONY: bdd php no-test-db
projects / nominatim.git / commitdiff