# Setting up Nominatim for Development
-This chapter gives an overview how to set up Nominatim for developement
+This chapter gives an overview how to set up Nominatim for development
and how to run tests.
!!! Important
- This guide assumes that you develop under the latest version of Ubuntu. You
- can of course also use your favourite distribution. You just might have to
- adapt the commands below slightly, in particular the commands for installing
- additional software.
+ This guide assumes you develop under the latest version of Debian/Ubuntu.
+ You can of course also use your favourite distribution. You just might have
+ to adapt the commands below slightly, in particular the commands for
+ installing additional software.
## Installing Nominatim
The first step is to install Nominatim itself. Please follow the installation
instructions in the [Admin section](../admin/Installation.md). You don't need
-to set up a webserver for development, the webserver that is included with PHP
-is sufficient.
+to set up a webserver for development, the webserver that can be started
+via `nominatim serve` is sufficient.
-If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
+If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup.
Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
install an NFS daemon to enable directory sharing between host and guest. The
following packages should get you started:
## Prerequisites for testing and documentation
The Nominatim test suite consists of behavioural tests (using behave) and
-unit tests (using PHPUnit). It has the following additional requirements:
+unit tests (using PHPUnit for PHP code and pytest for Python code).
+It has the following additional requirements:
-* [behave test framework](https://behave.readthedocs.io) >= 1.2.5
-* [phpunit](https://phpunit.de) >= 7.3
+* [behave test framework](https://behave.readthedocs.io) >= 1.2.6
+* [phpunit](https://phpunit.de) (9.5 is known to work)
* [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer)
+* [Pylint](https://pylint.org/) (CI always runs the latest version from pip)
+* [mypy](http://mypy-lang.org/) (plus typing information for external libs)
+* [Python Typing Extensions](https://github.com/python/typing_extensions) (for Python < 3.9)
+* [pytest](https://pytest.org)
+* [pytest-asyncio](https://pytest-asyncio.readthedocs.io)
+
+For testing the Python search frontend, you need to install extra dependencies
+depending on your choice of webserver framework:
+
+* [httpx](https://www.python-httpx.org/) (Starlette only)
+* [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) (Starlette only)
The documentation is built with mkdocs:
* [mkdocs](https://www.mkdocs.org/) >= 1.1.2
+* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25
+* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/)
+* [mkdocs-gen-files](https://oprypin.github.io/mkdocs-gen-files/)
-### Installing prerequisites on Ubuntu/Debian
+Please be aware that tests always run against the globally installed
+osm2pgsql, so you need to have this set up. If you want to test against
+the vendored version of osm2pgsql, you need to set the PATH accordingly.
-Some of the Python packages require the newest version which is not yet
-available with the current distributions. Therefore it is recommended to
-install pip to get the newest versions.
+### Installing prerequisites on Ubuntu/Debian
-To install all necessary packages run:
+The Python tools should always be run with the most recent version.
+In particular, pylint tends to have a lot of breaking changes between versions.
+The easiest way, to handle these Python dependencies is to run your
+development from within a virtual environment.
```sh
-sudo apt install php-cgi phpunit php-codesniffer \
- python3-pip python3-setuptools python3-dev
+sudo apt install libsqlite3-mod-spatialite php-cli
+```
+
+To set up the virtual environment with all necessary packages run:
-pip3 install --user behave mkdocs
+```sh
+virtualenv ~/nominatim-dev-venv
+~/nominatim-dev-venv/bin/pip install\
+ psutil psycopg[binary] PyICU SQLAlchemy \
+ python-dotenv jinja2 pyYAML datrie behave \
+ mkdocs mkdocstrings mkdocs-gen-files pytest pytest-asyncio pylint \
+ types-jinja2 types-markupsafe types-psutil types-psycopg2 \
+ types-pygments types-pyyaml types-requests types-ujson \
+ types-urllib3 typing-extensions unicorn falcon starlette \
+ uvicorn mypy osmium aiosqlite
```
-The `mkdocs` executable will be located in `.local/bin`. You may have to add
-this directory to your path, for example by running:
+Now enter the virtual environment whenever you want to develop:
+```sh
+. ~/nominatim-dev-venv/bin/activate
```
-echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
+
+For installing the PHP development tools, run:
+
+```sh
+sudo apt install php-cgi phpunit php-codesniffer
```
If your distribution does not have PHPUnit 7.3+, you can install it (as well
```
The binaries are found in `.config/composer/vendor/bin`. You need to add this
-to your PATH as well:
+to your PATH:
```
echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
```
+### Running Nominatim during development
-## Executing Tests
-
-All tests are located in the `\test` directory.
-
-### Preparing the test database
-
-Some of the behavioural test expect a test database to be present. You need at
-least 2GB RAM and 10GB disk space to create the database.
-
-First create a separate directory for the test DB and fetch the test planet
-data and the Tiger data for South Dakota:
-
-```
-mkdir testdb
-cd testdb
-wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf
-wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*'
-```
-
-Configure and build Nominatim in the usual way:
+The source code for Nominatim can be found in the `src` directory and can
+be run in-place. The source directory features a special script
+`nominatim-cli.py` which does the same as the installed 'nominatim' binary
+but executes against the code in the source tree. For example:
```
-cmake $USERNAME/Nominatim
-make
+me@machine:~$ cd Nominatim
+me@machine:~Nominatim$ ./nominatim-cli.py --version
+Nominatim version 4.4.99-1
```
-Create a minimal test settings file:
+Make sure you have activated the virtual environment holding all
+necessary dependencies.
-```
-tee .env << EOF
-NOMINATIM_DATABASE_DSN="pgsql:dbname=test_api_nominatim"
-NOMINATIM_USE_US_TIGER_DATA=yes
-NOMINATIM_TIGER_DATA_PATH=tiger
-NOMINATIM_WIKIPEDIA_DATA_PATH=$USERNAME/Nominatim/test/testdb
-EOF
-```
+## Executing Tests
-Inspect the file to check that all settings are correct for your local setup.
-In particular, the wikipedia path should point to the test directory in your
-Nominatim source directory.
+All tests are located in the `/test` directory.
-Now you can import the test database:
+To run all tests, run make from the source root:
+```sh
+make tests
```
-dropdb --if-exists test_api_nominatim
-./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log
-./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log
-./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log
-```
-
-### Running the tests
-To run all tests just go to the test directory and run make:
+There are also make targets for executing only parts of the test suite.
+For example to run linting only use:
```sh
-cd test
-make
+make lint
```
-To skip tests that require the test database, run `make no-test-db` instead.
+The possible testing targets are: mypy, lint, pytest, bdd.
For more information about the structure of the tests and how to change and
extend the test suite, see the [Testing chapter](Testing.md).
framework. The master branch is automatically deployed every night on
[https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
-To build the documentation, go to the build directory and run
+To build the documentation run
```
make doc
-INFO - Cleaning site directory
-INFO - Building documentation to directory: /home/vagrant/build/site-html
```
-This runs `mkdocs build` plus extra transformation of some files and adds
-symlinks (see `CMakeLists.txt` for the exact steps).
-Now you can start webserver for local testing
+For local testing, you can start webserver:
```
-build> mkdocs serve
+build> make serve-doc
[server:296] Serving on http://127.0.0.1:8000
[handlers:62] Start watching changes
```
projects / nominatim.git / blobdiff