Merge remote-tracking branch 'upstream/master'

[nominatim.git] / docs / develop / Development-Environment.md

diff --git a/docs/develop/Development-Environment.md b/docs/develop/Development-Environment.md
index fc5008c4f7ebebfe872fa06cfbc382036c46f176..fd7820c6abef5a0be686c6477572a51db22b7ec5 100644 (file)
--- a/docs/develop/Development-Environment.md
+++ b/docs/develop/Development-Environment.md
@@ -4,19 +4,19 @@ This chapter gives an overview how to set up Nominatim for development
 and how to run tests.
 
 !!! Important
 and how to run tests.
 
 !!! Important

-    This guide assumes that you develop under the latest version of Ubuntu. You
-    can of course also use your favourite distribution. You just might have to
-    adapt the commands below slightly, in particular the commands for installing
-    additional software.
+    This guide assumes you develop under the latest version of Debian/Ubuntu.
+    You can of course also use your favourite distribution. You just might have
+    to adapt the commands below slightly, in particular the commands for
+    installing additional software.

 
 ## Installing Nominatim
 
 The first step is to install Nominatim itself. Please follow the installation
 instructions in the [Admin section](../admin/Installation.md). You don't need
 
 ## Installing Nominatim
 
 The first step is to install Nominatim itself. Please follow the installation
 instructions in the [Admin section](../admin/Installation.md). You don't need

-to set up a webserver for development, the webserver that is included with PHP
-is sufficient.
+to set up a webserver for development, the webserver that can be started
+via `nominatim serve` is sufficient.

 
 

-If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
+If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup.

 Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
 install an NFS daemon to enable directory sharing between host and guest. The
 following packages should get you started:
 Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
 install an NFS daemon to enable directory sharing between host and guest. The
 following packages should get you started:


@@ -36,43 +36,60 @@ It has the following additional requirements:
 * [mypy](http://mypy-lang.org/) (plus typing information for external libs)
 * [Python Typing Extensions](https://github.com/python/typing_extensions) (for Python < 3.9)
 * [pytest](https://pytest.org)
 * [mypy](http://mypy-lang.org/) (plus typing information for external libs)
 * [Python Typing Extensions](https://github.com/python/typing_extensions) (for Python < 3.9)
 * [pytest](https://pytest.org)

+* [pytest-asyncio](https://pytest-asyncio.readthedocs.io)

 
 For testing the Python search frontend, you need to install extra dependencies
 depending on your choice of webserver framework:
 
 
 For testing the Python search frontend, you need to install extra dependencies
 depending on your choice of webserver framework:
 

-* [sanic-testing](https://sanic.dev/en/plugins/sanic-testing/getting-started.html) (sanic only)
-* [httpx](https://www.python-httpx.org/) (starlette only)
-* [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) (starlette only)
+* [httpx](https://www.python-httpx.org/) (Starlette only)
+* [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) (Starlette only)

 
 The documentation is built with mkdocs:
 
 * [mkdocs](https://www.mkdocs.org/) >= 1.1.2
 
 The documentation is built with mkdocs:
 
 * [mkdocs](https://www.mkdocs.org/) >= 1.1.2

-* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.16
-* [mkdocstrings-python-legacy](https://mkdocstrings.github.io/python-legacy/)
+* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25
+* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/)
+* [mkdocs-gen-files](https://oprypin.github.io/mkdocs-gen-files/)

 
 

-### Installing prerequisites on Ubuntu/Debian
+Please be aware that tests always run against the globally installed
+osm2pgsql, so you need to have this set up. If you want to test against
+the vendored version of osm2pgsql, you need to set the PATH accordingly.

 
 

-Some of the Python packages require the newest version which is not yet
-available with the current distributions. Therefore it is recommended to
-install pip to get the newest versions.
+### Installing prerequisites on Ubuntu/Debian

 
 

-To install all necessary packages run:
+The Python tools should always be run with the most recent version.
+In particular, pylint tends to have a lot of breaking changes between versions.
+The easiest way, to handle these Python dependencies is to run your
+development from within a virtual environment.

 
 ```sh
 
 ```sh

-sudo apt install php-cgi phpunit php-codesniffer \
-                 python3-pip python3-setuptools python3-dev
+sudo apt install libsqlite3-mod-spatialite php-cli
+```

 
 

-pip3 install --user behave mkdocs mkdocstrings pytest pylint \
-                    mypy types-PyYAML types-jinja2 types-psycopg2 types-psutil \
-                    types-ujosn \
-                    sanic-testing httpx asgi-lifespan
+To set up the virtual environment with all necessary packages run:
+
+```sh
+virtualenv ~/nominatim-dev-venv
+~/nominatim-dev-venv/bin/pip install\
+    psutil psycopg[binary] PyICU SQLAlchemy \
+    python-dotenv jinja2 pyYAML datrie behave \
+    mkdocs mkdocstrings mkdocs-gen-files pytest pytest-asyncio pylint \
+    types-jinja2 types-markupsafe types-psutil types-psycopg2 \
+    types-pygments types-pyyaml types-requests types-ujson \
+    types-urllib3 typing-extensions unicorn falcon starlette \
+    uvicorn mypy osmium aiosqlite

 ```
 
 ```
 

-The `mkdocs` executable will be located in `.local/bin`. You may have to add
-this directory to your path, for example by running:
+Now enter the virtual environment whenever you want to develop:

 
 

+```sh
+. ~/nominatim-dev-venv/bin/activate

 ```
 ```

-echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
+
+For installing the PHP development tools, run:
+
+```sh
+sudo apt install php-cgi phpunit php-codesniffer

 ```
 
 If your distribution does not have PHPUnit 7.3+, you can install it (as well
 ```
 
 If your distribution does not have PHPUnit 7.3+, you can install it (as well


@@ -85,24 +102,47 @@ composer global require "phpunit/phpunit=8.*"
 ```
 
 The binaries are found in `.config/composer/vendor/bin`. You need to add this
 ```
 
 The binaries are found in `.config/composer/vendor/bin`. You need to add this

-to your PATH as well:
+to your PATH:

 
 ```
 echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
 ```
 
 
 ```
 echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
 ```
 

+### Running Nominatim during development
+
+The source code for Nominatim can be found in the `src` directory and can
+be run in-place. The source directory features a special script
+`nominatim-cli.py` which does the same as the installed 'nominatim' binary
+but executes against the code in the source tree. For example:
+
+```
+me@machine:~$ cd Nominatim
+me@machine:~Nominatim$ ./nominatim-cli.py --version
+Nominatim version 4.4.99-1
+```
+
+Make sure you have activated the virtual environment holding all
+necessary dependencies.

 
 ## Executing Tests
 
 All tests are located in the `/test` directory.
 
 
 ## Executing Tests
 
 All tests are located in the `/test` directory.
 

-To run all tests just go to the build directory and run make:
+To run all tests, run make from the source root:

 
 ```sh
 
 ```sh

-cd build
-make test
+make tests

 ```
 
 ```
 

+There are also make targets for executing only parts of the test suite.
+For example to run linting only use:
+
+```sh
+make lint
+```
+
+The possible testing targets are: mypy, lint, pytest, bdd.
+

 For more information about the structure of the tests and how to change and
 extend the test suite, see the [Testing chapter](Testing.md).
 
 For more information about the structure of the tests and how to change and
 extend the test suite, see the [Testing chapter](Testing.md).
 


@@ -113,18 +153,14 @@ built using the [MkDocs](https://www.mkdocs.org/) static site generation
 framework. The master branch is automatically deployed every night on
 [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
 
 framework. The master branch is automatically deployed every night on
 [https://nominatim.org/release-docs/develop/](https://nominatim.org/release-docs/develop/)
 

-To build the documentation, go to the build directory and run
+To build the documentation run

 
 ```
 make doc
 
 ```
 make doc

-INFO - Cleaning site directory
-INFO - Building documentation to directory: /home/vagrant/build/site-html

 ```
 
 ```
 

-This runs `mkdocs build` plus extra transformation of some files and adds
-symlinks (see `CMakeLists.txt` for the exact steps).

 
 

-Now you can start webserver for local testing
+For local testing, you can start webserver:

 
 ```
 build> make serve-doc
 
 ```
 build> make serve-doc


@@ -136,7 +172,7 @@ If you develop inside a Vagrant virtual machine, use a port that is forwarded
 to your host:
 
 ```
 to your host:
 
 ```

-build> PYTHONPATH=$SRCDIR mkdocs serve --dev-addr 0.0.0.0:8088
+build> mkdocs serve --dev-addr 0.0.0.0:8088

 [server:296] Serving on http://0.0.0.0:8088
 [handlers:62] Start watching changes
 ```
 [server:296] Serving on http://0.0.0.0:8088
 [handlers:62] Start watching changes
 ```