# Tests
-tests: mypy lint pytest
+tests: mypy lint pytest bdd
mypy:
mypy --strict src
# Deploying the Nominatim Python frontend
-The Nominatim can be run as a Python-based
+Nominatim can be run as a Python-based
[ASGI web application](https://asgi.readthedocs.io/en/latest/). You have the
choice between [Falcon](https://falcon.readthedocs.io/en/stable/)
and [Starlette](https://www.starlette.io/) as the ASGI framework.
!!! Note
Throughout this page, we assume your Nominatim project directory is
- located in `/srv/nominatim-project` and you have installed Nominatim
- using the default installation prefix `/usr/local`. If you have put it
- somewhere else, you need to adjust the commands and configuration
- accordingly.
+ located in `/srv/nominatim-project`. If you have put it somewhere else,
+ you need to adjust the commands and configuration accordingly.
- We further assume that your web server runs as user `www-data`. Older
- versions of CentOS may still use the user name `apache`. You also need
- to adapt the instructions in this case.
### Installing the required packages
+The Nominatim frontend is best run from its own virtual environment. If
+you have already created one for the database backend during
+[installation](Installation.md#Building-Nominatim), you can use that. Otherwise
+create one now with:
+
+```sh
+sudo apt-get install virtualenv
+virtualenv /srv/nominatim-venv
+```
+
+The Nominatim frontend is contained in the 'nominatim-api' package. To
+install directly from the source tree run:
+
+```sh
+cd Nominatim
+/srv/nominatim-venv/bin/pip install packaging/nominatim-api
+```
+
The recommended way to deploy a Python ASGI application is to run
the ASGI runner [uvicorn](https://uvicorn.org/)
together with [gunicorn](https://gunicorn.org/) HTTP server. We use
Falcon here as the web framework.
-Create a virtual environment for the Python packages and install the necessary
-dependencies:
+Add the necessary packages to your virtual environment:
``` sh
-sudo apt install virtualenv
-virtualenv /srv/nominatim-venv
-/srv/nominatim-venv/bin/pip install SQLAlchemy PyICU psycopg[binary] \
- psycopg2-binary python-dotenv PyYAML falcon uvicorn gunicorn
+/srv/nominatim-venv/bin/pip install falcon uvicorn gunicorn
```
### Setting up Nominatim as a systemd job
[Service]
Type=simple
-Environment="PYTHONPATH=/usr/local/lib/nominatim/lib-python/"
User=www-data
Group=www-data
WorkingDirectory=/srv/nominatim-project
-ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim.server.falcon.server:run_wsgi
+ExecStart=/srv/nominatim-venv/bin/gunicorn -b unix:/run/nominatim.sock -w 4 -k uvicorn.workers.UvicornWorker nominatim_api.server.falcon.server:run_wsgi
ExecReload=/bin/kill -s HUP $MAINPID
StandardOutput=append:/var/log/gunicorn-nominatim.log
StandardError=inherit
#### Building the latest development version with pip
-To install Nominatim directly from the source tree, run:
+Nominatim is easiest to run from its own virtual environment. To create one, run:
- pip install packaging/nominatim-{core,db,api}
+ sudo apt-get install virtualenv
+ virtualenv /srv/nominatim-venv
+
+To install Nominatim directly from the source tree into the virtual environment, run:
+
+ /srv/nominatim-venv/bin/pip install packaging/nominatim-{core,db,api}
#### Building in legacy CMake mode
If you are migrating from a version <3.6, then you still have to follow
the manual migration steps up to 3.6.
+## 4.4.0 -> master
+
+### New structure for Python packages
+
+The nominatim Python package has been split into nominatim-db and nominatim-api.
+Any imports need to be adapted accordingly.
+
+If you are running the Python frontend, change the server module from
+`nominatim.server.falcon.server` to `nominatim_api.server.falcon.server`.
+
+If you are using the Nominatim library, all imports need to be changed
+from `nominatim.api.<module>` to `nominatim_api.<module>`.
+
+If you have written custom tokenizers or sanitizers, the appropriate modules
+are now found in `nominatim_db`.
+
## 4.2.0 -> 4.3.0
### New indexes for reverse lookup
* **phrase**: the keyword to look for
* **class**: key of the main tag of the place to find
- (see [principal tags in import style](../Import-Styles.md#set_main_tags-principal-tags)
+ (see [principal tags in import style](Import-Styles.md#set_main_tags-principal-tags)
* **type**: value of the main tag
* **operator**: type of special phrase, may be one of:
* *in*: place is within the place defined by the search term (e.g. "_Hotels in_ Berlin")
##### split-name-list
-::: nominatim.tokenizer.sanitizers.split_name_list
+::: nominatim_db.tokenizer.sanitizers.split_name_list
options:
members: False
heading_level: 6
##### strip-brace-terms
-::: nominatim.tokenizer.sanitizers.strip_brace_terms
+::: nominatim_db.tokenizer.sanitizers.strip_brace_terms
options:
members: False
heading_level: 6
##### tag-analyzer-by-language
-::: nominatim.tokenizer.sanitizers.tag_analyzer_by_language
+::: nominatim_db.tokenizer.sanitizers.tag_analyzer_by_language
options:
members: False
heading_level: 6
##### clean-housenumbers
-::: nominatim.tokenizer.sanitizers.clean_housenumbers
+::: nominatim_db.tokenizer.sanitizers.clean_housenumbers
options:
members: False
heading_level: 6
##### clean-postcodes
-::: nominatim.tokenizer.sanitizers.clean_postcodes
+::: nominatim_db.tokenizer.sanitizers.clean_postcodes
options:
members: False
heading_level: 6
##### clean-tiger-tags
-::: nominatim.tokenizer.sanitizers.clean_tiger_tags
+::: nominatim_db.tokenizer.sanitizers.clean_tiger_tags
options:
members: False
heading_level: 6
#### delete-tags
-::: nominatim.tokenizer.sanitizers.delete_tags
+::: nominatim_db.tokenizer.sanitizers.delete_tags
options:
members: False
heading_level: 6
#### tag-japanese
-::: nominatim.tokenizer.sanitizers.tag_japanese
+::: nominatim_db.tokenizer.sanitizers.tag_japanese
options:
members: False
heading_level: 6
and how to run tests.
!!! Important
- This guide assumes that you develop under the latest version of Ubuntu. You
- can of course also use your favourite distribution. You just might have to
- adapt the commands below slightly, in particular the commands for installing
- additional software.
+ This guide assumes that you develop under the latest version of Debain/Ubuntu.
+ You can of course also use your favourite distribution. You just might have
+ to adapt the commands below slightly, in particular the commands for
+ installing additional software.
## Installing Nominatim
The first step is to install Nominatim itself. Please follow the installation
instructions in the [Admin section](../admin/Installation.md). You don't need
-to set up a webserver for development, the webserver that is included with PHP
-is sufficient.
+to set up a webserver for development, the webserver that can be started
+via `nominatim serve` is sufficient.
-If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu` setup.
+If you want to run Nominatim in a VM via Vagrant, use the default `ubuntu24` setup.
Vagrant's libvirt provider runs out-of-the-box under Ubuntu. You also need to
install an NFS daemon to enable directory sharing between host and guest. The
following packages should get you started:
The documentation is built with mkdocs:
* [mkdocs](https://www.mkdocs.org/) >= 1.1.2
-* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.18
-* [mkdocstrings-python](https://mkdocstrings.github.io/python/)
+* [mkdocstrings](https://mkdocstrings.github.io/) >= 0.25
+* [mkdocs-material](https://squidfunk.github.io/mkdocs-material/)
### Installing prerequisites on Ubuntu/Debian
-Some of the Python packages require the newest version which is not yet
-available with the current distributions. Therefore it is recommended to
-install pip to get the newest versions.
+The Python tools should always be run with the most recent version.
+In particular, pylint tends to have a lot of breaking changes between versions.
+The easiest way, to handle these Python dependencies is to run your
+development from within a virtual environment.
-To install all necessary packages run:
+To set up the virtual environment with all necessary packages run:
```sh
-sudo apt install php-cgi phpunit php-codesniffer \
- python3-pip python3-setuptools python3-dev
-
-pip3 install --user behave mkdocs mkdocstrings pytest pytest-asyncio pylint \
- mypy types-PyYAML types-jinja2 types-psycopg2 types-psutil \
- types-ujson types-requests types-Pygments typing-extensions\
- httpx asgi-lifespan
+virtualenv ~/nominatim-dev-venv
+~/nominatim-dev-venv/bin/pip install\
+ psycopg2-binary psutil psycopg[binary] PyICU SQLAlchemy \
+ python-dotenv jinja2 pyYAML datree \
+ behave mkdocs mkdocstrings pytest pytest-asyncio pylint \
+ types-jinja2 types-markupsafe types-psutil types-psycopg2 \
+ types-pygments types-pyyaml types-requests types-ujson \
+ types-urllib3 typing-extensions unicorn falcon
```
-The `mkdocs` executable will be located in `.local/bin`. You may have to add
-this directory to your path, for example by running:
+Now enter the virtual environment whenever you want to develop:
+```sh
+. ~/nominatim-dev-venv/bin/activate
```
-echo 'export PATH=~/.local/bin:$PATH' > ~/.profile
+
+For installing the PHP development tools, run:
+
+```sh
+sudo apt install php-cgi phpunit php-codesniffer
```
If your distribution does not have PHPUnit 7.3+, you can install it (as well
```
The binaries are found in `.config/composer/vendor/bin`. You need to add this
-to your PATH as well:
+to your PATH:
```
echo 'export PATH=~/.config/composer/vendor/bin:$PATH' > ~/.profile
```
+### Running Nominatim during development
+
+The source code for Nominatim can be found in the `src` directory and can
+be run in-place. The source directory features a special script
+`nominatim-cli.py` which does the same as the installed 'nominatim' binary
+but executes against the code in the source tree. For example:
+
+```
+me@machine:~$ cd Nomiantim
+me@machine:~Nomiantim$ ./nominatim-cli.py --version
+Nominatim version 4.4.99-1
+```
+
+Make sure you have activated the virtual environment that holds all
+necessary dependencies.
## Executing Tests
All tests are located in the `/test` directory.
-To run all tests just go to the build directory and run make:
+To run all tests, run make from the source root:
```sh
-cd build
-make test
+make tests
```
+There are also goals for executing parts of the test suite: mypy, lint, pytest, bdd.
+
For more information about the structure of the tests and how to change and
extend the test suite, see the [Testing chapter](Testing.md).
### Sanitizer configuration
-::: nominatim.tokenizer.sanitizers.config.SanitizerConfig
+::: nominatim_db.tokenizer.sanitizers.config.SanitizerConfig
options:
heading_level: 6
#### PlaceInfo - information about the place
-::: nominatim.data.place_info.PlaceInfo
+::: nominatim_db.data.place_info.PlaceInfo
options:
heading_level: 6
#### PlaceName - extended naming information
-::: nominatim.data.place_name.PlaceName
+::: nominatim_db.data.place_name.PlaceName
options:
heading_level: 6
## Custom token analysis module
-::: nominatim.tokenizer.token_analysis.base.AnalysisModule
+::: nominatim_db.tokenizer.token_analysis.base.AnalysisModule
options:
heading_level: 6
-::: nominatim.tokenizer.token_analysis.base.Analyzer
+::: nominatim_db.tokenizer.token_analysis.base.Analyzer
options:
heading_level: 6
### Python Tokenizer Class
-All tokenizers must inherit from `nominatim.tokenizer.base.AbstractTokenizer`
+All tokenizers must inherit from `nominatim_db.tokenizer.base.AbstractTokenizer`
and implement the abstract functions defined there.
-::: nominatim.tokenizer.base.AbstractTokenizer
+::: nominatim_db.tokenizer.base.AbstractTokenizer
options:
heading_level: 6
### Python Analyzer Class
-::: nominatim.tokenizer.base.AbstractAnalyzer
+::: nominatim_db.tokenizer.base.AbstractAnalyzer
options:
heading_level: 6
## `Configuration` class
-::: nominatim.config.Configuration
+::: nominatim_core.config.Configuration
options:
members:
- get_bool
The library interface is currently in an experimental stage. There might
be some smaller adjustments to the public interface until the next version.
- The library also misses a proper installation routine, so some manipulation
- of the PYTHONPATH is required. At the moment, use is only recommended for
- developers with some experience in Python.
-
## Installation
To use the Nominatim library, you need access to a local Nominatim database.
Follow the [installation](../admin/Installation.md) and
[import](../admin/Import.md) instructions to set up your database.
-It is not yet possible to install it in the usual way via pip or inside a
-virtualenv. To get access to the library you need to set an appropriate
-`PYTHONPATH`. With the default installation, the python library can be found
-under `/usr/local/share/nominatim/lib-python`. If you have installed
-Nominatim under a different prefix, adapt the `/usr/local/` part accordingly.
-You can also point the `PYTHONPATH` to the Nominatim source code.
+The Nominatim frontend library is contained in the 'nominatim-api' package.
+To install the package from the source tree directly, run:
+
+ pip install packaging/nominatim-api
+
+Usually, you would want to run this in a virtual environment.
### A simple search example
from pathlib import Path
import asyncio
- import nominatim.api as napi
+ import nominatim_api as napi
async def search(query):
api = napi.NominatimAPIAsync(Path('.'))
``` python
from pathlib import Path
- import nominatim.api as napi
+ import nominatim_api as napi
api = napi.NominatimAPI(Path('.'))
from pathlib import Path
import asyncio
- import nominatim.api as napi
+ import nominatim_api as napi
config_params = {
'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium'
``` python
from pathlib import Path
- import nominatim.api as napi
+ import nominatim_api as napi
config_params = {
'NOMINATIM_DATABASE_DSN': 'pgsql:dbname=belgium'
from pathlib import Path
import asyncio
- import nominatim.api as napi
+ import nominatim_api as napi
async def search(query):
api = napi.NominatimAPIAsync(Path('.'))
``` python
from pathlib import Path
- import nominatim.api as napi
+ import nominatim_api as napi
api = napi.NominatimAPI(Path('.'))
## Place identification
-The [details](NominatimAPI.md#nominatim.api.core.NominatimAPI.details) and
-[lookup](NominatimAPI.md#nominatim.api.core.NominatimAPI.lookup) functions
+The [details](NominatimAPI.md#nominatim_api.core.NominatimAPI.details) and
+[lookup](NominatimAPI.md#nominatim_api.core.NominatimAPI.lookup) functions
require references to places in the database. Below the possible
types for place identification are listed. All types are dataclasses.
### PlaceID
-::: nominatim.api.PlaceID
+::: nominatim_api.PlaceID
options:
heading_level: 6
### OsmID
-::: nominatim.api.OsmID
+::: nominatim_api.OsmID
options:
heading_level: 6
## Geometry types
-::: nominatim.api.GeometryFormat
+::: nominatim_api.GeometryFormat
options:
heading_level: 6
members_order: source
### Point
-::: nominatim.api.Point
+::: nominatim_api.Point
options:
heading_level: 6
show_signature_annotations: True
### Bbox
-::: nominatim.api.Bbox
+::: nominatim_api.Bbox
options:
heading_level: 6
show_signature_annotations: True
geographic extent.
-::: nominatim.api.DataLayer
+::: nominatim_api.DataLayer
options:
heading_level: 6
members_order: source
-
-
import asyncio
from pathlib import Path
import sqlalchemy as sa
-from nominatim.api import NominatimAPIAsync
+from nominatim_api import NominatimAPIAsync
async def print_table_size():
api = NominatimAPIAsync(Path('.'))
## SearchConnection class
-::: nominatim.api.SearchConnection
+::: nominatim_api.SearchConnection
options:
members:
- scalar
### NominatimAPI
-::: nominatim.api.NominatimAPI
+::: nominatim_api.NominatimAPI
options:
members:
- __init__
### NominatimAPIAsync
-::: nominatim.api.NominatimAPIAsync
+::: nominatim_api.NominatimAPIAsync
options:
members:
- __init__
Nominatim takes the result data from multiple sources. The `source_table` field
in the result describes, from which source the result was retrieved.
-::: nominatim.api.SourceTable
+::: nominatim_api.SourceTable
options:
heading_level: 6
members_order: source
for the house number or the country code. This makes processing and creating
a full address easier.
-::: nominatim.api.AddressLine
+::: nominatim_api.AddressLine
options:
heading_level: 6
members_order: source
terms are split into terms for the name of the place and search terms for
its address.
-::: nominatim.api.WordInfo
+::: nominatim_api.WordInfo
options:
heading_level: 6
### Locale
-::: nominatim.api.Locales
+::: nominatim_api.Locales
options:
heading_level: 6
- mkdocstrings:
handlers:
python:
- paths: ["${PROJECT_SOURCE_DIR}"]
+ paths: ["${PROJECT_SOURCE_DIR}/src"]
options:
show_source: False
show_bases: False
projects / nominatim.git / commitdiff