X-Git-Url: https://git.openstreetmap.org./nominatim.git/blobdiff_plain/b1b25d9a10ea7b97e12effc4313f140c17730c78..330b8d2fdfdd935da123abf9ecf96b3bc7a12be5:/docs/develop/Development-Environment.md?ds=sidebyside diff --git a/docs/develop/Development-Environment.md b/docs/develop/Development-Environment.md index 4f790641..fd7820c6 100644 --- a/docs/develop/Development-Environment.md +++ b/docs/develop/Development-Environment.md @@ -1,22 +1,22 @@ # Setting up Nominatim for Development -This chapter gives an overview how to set up Nominatim for developement +This chapter gives an overview how to set up Nominatim for development and how to run tests. !!! Important - This guide assumes that you develop under the latest version of Ubuntu. You - can of course also use your favourite distribution. You just might have to - adapt the commands below slightly, in particular the commands for installing - additional software. + This guide assumes you develop under the latest version of Debian/Ubuntu. + You can of course also use your favourite distribution. You just might have + to adapt the commands below slightly, in particular the commands for + installing additional software. ## Installing Nominatim The first step is to install Nominatim itself. Please follow the installation instructions in the [Admin section](../admin/Installation.md). You don't need -to set up a webserver for development, the webserver that is included with PHP -is sufficient. +to set up a webserver for development, the webserver that can be started +via `nominatim serve` is sufficient. -If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup. +If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup. Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to install an NFS daemon to enable directory sharing between host and guest. The following packages should get you started: @@ -25,39 +25,71 @@ following packages should get you started: ## Prerequisites for testing and documentation -The Nominatim tests suite consists of behavioural tests (using behave) and -unit tests (using PHPUnit). It has the following additional requirements: +The Nominatim test suite consists of behavioural tests (using behave) and +unit tests (using PHPUnit for PHP code and pytest for Python code). +It has the following additional requirements: -* [behave test framework](https://behave.readthedocs.io) >= 1.2.5 -* [nose](https://nose.readthedocs.io) -* [pytidylib](http://countergram.com/open-source/pytidylib) -* [phpunit](https://phpunit.de) >= 7.3 +* [behave test framework](https://behave.readthedocs.io) >= 1.2.6 +* [phpunit](https://phpunit.de) (9.5 is known to work) * [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer) +* [Pylint](https://pylint.org/) (CI always runs the latest version from pip) +* [mypy](http://mypy-lang.org/) (plus typing information for external libs) +* [Python Typing Extensions](https://github.com/python/typing_extensions) (for Python < 3.9) +* [pytest](https://pytest.org) +* [pytest-asyncio](https://pytest-asyncio.readthedocs.io) + +For testing the Python search frontend, you need to install extra dependencies +depending on your choice of webserver framework: + +* [httpx](https://www.python-httpx.org/) (Starlette only) +* [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) (Starlette only) The documentation is built with mkdocs: * [mkdocs](https://www.mkdocs.org/) >= 1.1.2 +* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25 +* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/) +* [mkdocs-gen-files](https://oprypin.github.io/mkdocs-gen-files/) -### Installing prerequisites on Ubuntu/Debian +Please be aware that tests always run against the globally installed +osm2pgsql, so you need to have this set up. If you want to test against +the vendored version of osm2pgsql, you need to set the PATH accordingly. -Some of the Python packages require the newest version which is not yet -available with the current distributions. Therefore it is recommended to -install pip to get the newest versions. +### Installing prerequisites on Ubuntu/Debian -To install all necessary packages run: +The Python tools should always be run with the most recent version. +In particular, pylint tends to have a lot of breaking changes between versions. +The easiest way, to handle these Python dependencies is to run your +development from within a virtual environment. ```sh -sudo apt install php-cgi phpunit php-codesniffer \ - python3-pip python3-setuptools python3-dev +sudo apt install libsqlite3-mod-spatialite php-cli +``` + +To set up the virtual environment with all necessary packages run: -pip3 install --user behave nose mkdocs +```sh +virtualenv ~/nominatim-dev-venv +~/nominatim-dev-venv/bin/pip install\ + psutil psycopg[binary] PyICU SQLAlchemy \ + python-dotenv jinja2 pyYAML datrie behave \ + mkdocs mkdocstrings mkdocs-gen-files pytest pytest-asyncio pylint \ + types-jinja2 types-markupsafe types-psutil types-psycopg2 \ + types-pygments types-pyyaml types-requests types-ujson \ + types-urllib3 typing-extensions unicorn falcon starlette \ + uvicorn mypy osmium aiosqlite ``` -The `mkdocs` executable will be located in `.local/bin`. You may have to add -this directory to your path, for example by running: +Now enter the virtual environment whenever you want to develop: +```sh +. ~/nominatim-dev-venv/bin/activate ``` -echo 'export PATH=~/.local/bin:$PATH' > ~/.profile + +For installing the PHP development tools, run: + +```sh +sudo apt install php-cgi phpunit php-codesniffer ``` If your distribution does not have PHPUnit 7.3+, you can install it (as well @@ -70,66 +102,46 @@ composer global require "phpunit/phpunit=8.*" ``` The binaries are found in `.config/composer/vendor/bin`. You need to add this -to your PATH as well: +to your PATH: ``` echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile ``` +### Running Nominatim during development -## Executing Tests - -All tests are located in the `\test` directory. - -### Preparing the test database - -Some of the behavioural test expect a test database to be present. You need at -least 2GB RAM and 10GB disk space to create the database. - -First create a separate directory for the test DB and Fetch the test planet -data and the Tiger data for South Dakota: - -``` -mkdir testdb -cd testdb -wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf -wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*' -``` - -Configure and build Nominatim in the usual way: +The source code for Nominatim can be found in the `src` directory and can +be run in-place. The source directory features a special script +`nominatim-cli.py` which does the same as the installed 'nominatim' binary +but executes against the code in the source tree. For example: ``` -cmake $USERNAME/Nominatim -make +me@machine:~$ cd Nominatim +me@machine:~Nominatim$ ./nominatim-cli.py --version +Nominatim version 4.4.99-1 ``` -Copy the test settings: +Make sure you have activated the virtual environment holding all +necessary dependencies. -``` -cp $USERNAME/Nominatim/test/testdb/local.php settings/ -``` +## Executing Tests -Inspect the file to check that all settings are correct for your local setup. +All tests are located in the `/test` directory. -Now you can import the test database: +To run all tests, run make from the source root: +```sh +make tests ``` -dropdb --if-exists test_api_nominatim -./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log -./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log -./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log -``` - -### Running the tests -To run all tests just go to the test directory and run make: +There are also make targets for executing only parts of the test suite. +For example to run linting only use: ```sh -cd test -make +make lint ``` -To skip tests that require the test database, run `make no-test-db` instead. +The possible testing targets are: mypy, lint, pytest, bdd. For more information about the structure of the tests and how to change and extend the test suite, see the [Testing chapter](Testing.md). @@ -141,21 +153,17 @@ built using the [MkDocs](https://www.mkdocs.org/) static site generation framework. The master branch is automatically deployed every night on [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/) -To build the documentation, go to the build directory and run +To build the documentation run ``` make doc -INFO - Cleaning site directory -INFO - Building documentation to directory: /home/vagrant/build/site-html ``` -This runs `mkdocs build` plus extra transformation of some files and adds -symlinks (see `CMakeLists.txt` for the exact steps). -Now you can start webserver for local testing +For local testing, you can start webserver: ``` -build> mkdocs serve +build> make serve-doc [server:296] Serving on http://127.0.0.1:8000 [handlers:62] Start watching changes ```