X-Git-Url: https://git.openstreetmap.org./nominatim.git/blobdiff_plain/2b11a47a2f03efd93c990c8cc263fb9ea3491075..fa1c402b47bdd1f76b8528885c0e3a5e90731cdc:/docs/develop/Development-Environment.md?ds=sidebyside diff --git a/docs/develop/Development-Environment.md b/docs/develop/Development-Environment.md index 0334d0f1..20d02411 100644 --- a/docs/develop/Development-Environment.md +++ b/docs/develop/Development-Environment.md @@ -1,22 +1,22 @@ # Setting up Nominatim for Development -This chapter gives an overview how to set up Nominatim for developement +This chapter gives an overview how to set up Nominatim for development and how to run tests. -!!!Important -This guide assumes that you develop under the latest version of Ubuntu. You -can of course also use your favourite distribution. You just might have to -adapt the commands below slightly, in particular the commands for installing -additional software. +!!! Important + This guide assumes that you develop under the latest version of Debain/Ubuntu. + You can of course also use your favourite distribution. You just might have + to adapt the commands below slightly, in particular the commands for + installing additional software. ## Installing Nominatim The first step is to install Nominatim itself. Please follow the installation instructions in the [Admin section](../admin/Installation.md). You don't need -to set up a webserver for development, the webserver that is included with PHP -is sufficient. +to set up a webserver for development, the webserver that can be started +via `nominatim serve` is sufficient. -If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup. +If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup. Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to install an NFS daemon to enable directory sharing between host and guest. The following packages should get you started: @@ -25,39 +25,61 @@ following packages should get you started: ## Prerequisites for testing and documentation -The Nominatim tests suite consists of behavioural tests (using behave) and -unit tests (using PHPUnit). It has the following additional requirements: +The Nominatim test suite consists of behavioural tests (using behave) and +unit tests (using PHPUnit for PHP code and pytest for Python code). +It has the following additional requirements: -* [behave test framework](https://github.com/behave/behave) >= 1.2.5 -* [nose](https://nose.readthedocs.org) -* [pytidylib](http://countergram.com/open-source/pytidylib) -* [phpunit](https://phpunit.de) >= 7.3 +* [behave test framework](https://behave.readthedocs.io) >= 1.2.6 +* [phpunit](https://phpunit.de) (9.5 is known to work) * [PHP CodeSniffer](https://github.com/squizlabs/PHP_CodeSniffer) +* [Pylint](https://pylint.org/) (CI always runs the latest version from pip) +* [mypy](http://mypy-lang.org/) (plus typing information for external libs) +* [Python Typing Extensions](https://github.com/python/typing_extensions) (for Python < 3.9) +* [pytest](https://pytest.org) +* [pytest-asyncio](https://pytest-asyncio.readthedocs.io) + +For testing the Python search frontend, you need to install extra dependencies +depending on your choice of webserver framework: + +* [httpx](https://www.python-httpx.org/) (starlette only) +* [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) (starlette only) The documentation is built with mkdocs: * [mkdocs](https://www.mkdocs.org/) >= 1.1.2 +* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25 +* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/) ### Installing prerequisites on Ubuntu/Debian -Some of the Python packages require the newest version which is not yet -available with the current distributions. Therefore it is recommended to -install pip to get the newest versions. +The Python tools should always be run with the most recent version. +In particular, pylint tends to have a lot of breaking changes between versions. +The easiest way, to handle these Python dependencies is to run your +development from within a virtual environment. -To install all necessary packages run: +To set up the virtual environment with all necessary packages run: ```sh -sudo apt install php-cgi phpunit php-codesniffer \ - python3-pip python3-setuptools python3-dev - -pip3 install --user behave nose mkdocs +virtualenv ~/nominatim-dev-venv +~/nominatim-dev-venv/bin/pip install\ + psycopg2-binary psutil psycopg[binary] PyICU SQLAlchemy \ + python-dotenv jinja2 pyYAML datree \ + behave mkdocs mkdocstrings pytest pytest-asyncio pylint \ + types-jinja2 types-markupsafe types-psutil types-psycopg2 \ + types-pygments types-pyyaml types-requests types-ujson \ + types-urllib3 typing-extensions unicorn falcon ``` -The `mkdocs` executable will be located in `.local/bin`. You may have to add -this directory to your path, for example by running: +Now enter the virtual environment whenever you want to develop: +```sh +. ~/nominatim-dev-venv/bin/activate ``` -echo 'export PATH=~/.local/bin:$PATH' > ~/.profile + +For installing the PHP development tools, run: + +```sh +sudo apt install php-cgi phpunit php-codesniffer ``` If your distribution does not have PHPUnit 7.3+, you can install it (as well @@ -70,66 +92,39 @@ composer global require "phpunit/phpunit=8.*" ``` The binaries are found in `.config/composer/vendor/bin`. You need to add this -to your PATH as well: +to your PATH: ``` echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile ``` +### Running Nominatim during development -## Executing Tests - -All tests are located in the `\test` directory. - -### Preparing the test database - -Some of the behavioural test expect a test database to be present. You need at -least 2GB RAM and 10GB disk space to create the database. - -First create a separate directory for the test DB and Fetch the test planet -data and the Tiger data for South Dakota: - -``` -mkdir testdb -cd testdb -wget https://www.nominatim.org/data/test/nominatim-api-testdata.pbf -wget -O - https://nominatim.org/data/tiger2018-nominatim-preprocessed.tar.gz | tar xz --wildcards --no-anchored '46*' -``` - -Configure and build Nominatim in the usual way: - -``` -cmake $USERNAME/Nominatim -make -``` - -Copy the test settings: +The source code for Nominatim can be found in the `src` directory and can +be run in-place. The source directory features a special script +`nominatim-cli.py` which does the same as the installed 'nominatim' binary +but executes against the code in the source tree. For example: ``` -cp $USERNAME/Nominatim/test/testdb/local.php settings/ +me@machine:~$ cd Nomiantim +me@machine:~Nomiantim$ ./nominatim-cli.py --version +Nominatim version 4.4.99-1 ``` -Inspect the file to check that all settings are correct for your local setup. - -Now you can import the test database: +Make sure you have activated the virtual environment that holds all +necessary dependencies. -``` -dropdb --if-exists test_api_nominatim -./utils/setup.php --all --osm-file nominatim-api-testdb.pbf 2>&1 | tee import.log -./utils/specialphrases.php --wiki-import | psql -d test_api_nominatim 2>&1 | tee -a import.log -./utils/setup.php --import-tiger-data 2>&1 | tee -a import.log -``` +## Executing Tests -### Running the tests +All tests are located in the `/test` directory. -To run all tests just go to the test directory and run make: +To run all tests, run make from the source root: ```sh -cd test -make +make tests ``` -To skip tests that require the test database, run `make no-test-db` instead. +There are also goals for executing parts of the test suite: mypy, lint, pytest, bdd. For more information about the structure of the tests and how to change and extend the test suite, see the [Testing chapter](Testing.md). @@ -155,7 +150,7 @@ symlinks (see `CMakeLists.txt` for the exact steps). Now you can start webserver for local testing ``` -build> mkdocs serve +build> make serve-doc [server:296] Serving on http://127.0.0.1:8000 [handlers:62] Start watching changes ``` @@ -164,7 +159,7 @@ If you develop inside a Vagrant virtual machine, use a port that is forwarded to your host: ``` -build> mkdocs serve --dev-addr 0.0.0.0:8088 +build> PYTHONPATH=$SRCDIR mkdocs serve --dev-addr 0.0.0.0:8088 [server:296] Serving on http://0.0.0.0:8088 [handlers:62] Start watching changes ```