Here’s how to install the OpenStreetMap Nominatim service on your own server. It can be used to geocode and reverse geocode addresses and map coordinates. You will also get a web interface which loads map tiles from openstreetmap.org while doing geocoding requests using your own server.
I was faced with thousands of geocoding requests per hour, and with such a high volume it is not wise to burden the global OSM servers. Even with aggressive caching it would have potentially been too much.
I will use Ubuntu 14.04 LTS as the platform. Just a basic install with ssh server. We will install Apache to serve http requests. Make sure you have enough disk space and RAM to hold the data and serve it efficiently. I used the Finland extract, which was about a 200 MB download. The resulting database was 26 GB after importing, indexing and adding Wikipedia data. The Wikipedia data probably actually took more disk space than the OSM data. My server has 4 GB RAM, which seems to be enough for this small data set.
Table of Contents
Sofware requirements
PostgreSQL with PostGIS extension:
1 2 3 4 |
sudo apt-get install \ postgresql postgis postgresql-contrib \ postgresql-server-dev-9.3 postgresql-9.3-postgis-2.1 \ postgresql-doc-9.3 postgis-doc |
Apache with PHP5:
1 2 |
apt-get install \ apache2 php5 php-pear php5-pgsql php5-json php-db |
Git and various tools:
1 2 3 |
apt-get install \ wget git autoconf-archive build-essential \ automake gcc proj-bin |
Tool for handling OpenStreetMap data:
1 |
apt-get install osmosis |
Needed libraries:
1 2 3 4 5 6 |
apt-get install \ libxml2-dev libgeos-dev libpq-dev libbz2-dev libtool \ automake libproj-dev libboost-dev libboost-system-dev \ libboost-filesystem-dev libboost-thread-dev \ libgeos-c1 libgeos++-dev lua5.2 liblua5.2-dev \ libprotobuf-c0-dev protobuf-c-compiler |
Kernel tuning
We must increase kernel shared memory limits. Also we’ll reduce swappiness and set the kernel memory overcommit to be a bit more conservative. Make sure you have anough swap space, more than physical memory is a good idea.
These have been estimated using 4 GB RAM.
1 2 3 4 5 6 7 8 9 |
sysctl -w kernel.shmmax=4404019200 sysctl -w kernel.shmall=1075200 sysctl vm.overcommit_memory=2 sysctl vm.swappiness=10 echo "kernel.shmmax=4404019200" >> /etc/sysctl.conf echo "kernel.shmall=1075200" >> /etc/sysctl.conf echo "vm.overcommit_memory=2" >> /etc/sysctl.conf echo "vm.swappiness=10" >> /etc/sysctl.conf |
PostgreSQL tuning
The following directives were set in /etc/postgresql/9.3/main/postgresql.conf:
1 2 3 4 5 6 7 8 9 10 |
shared_buffers = 2048MB work_mem = 50MB maintenance_work_mem = 1024M fsync = off # dangerous! synchronous_commit = off full_page_writes = off # dangerous! checkpoint_segments = 100 checkpoint_timeout = 10min checkpoint_completion_target = 0.9 effective_cache_size = 2048M |
The buffer and memory values depend on your available RAM so you should set sane values accordingly. These are good for 4 GB RAM.
There are two truly dangerous settings here. Setting fsync off will cause PostgreSQL to report successful commits before the disk has confirmed the data has been written successfully. Setting full_page_writes off exposes the database to the possibility of partially updated data in case of power failure. These options are used here only for the duration of the initial import, because setting them so makes the import much faster. Fsync and full page writes must be turned back on after the import to ensure database consistency.
The synchronous_commit also jeopardizes the durability of committed transactions by reporting successful commits to clients asynchronously, meaning that the transaction may not have been fully written to disk. It does not compromize consistency as the two previous directives discussed. It just means that in the case of power failure, some recent transactions that were already reported to the client as having been committed, may in fact be aborted and rolled back. The database will still be in a consistent state. We’ll leave it off because it speeds up the queries, and we don’t really care about it in this database. We can always rebuild it from scratch. But if you have other important databases in the same database cluster, I would recommend turning it back on as well, since the setting is cluster-wide.
The maintenance_work_mem will also be reduced to a lower value later, after the import.
Restart PostgreSQL to apply changes:
1 |
pg_ctlcluster 9.3 main restart |
Dedicated user
We’ll install the software under this user’s home directory (user id “nominatim”).
1 |
useradd -c 'OpenStreetMap Nominatim' -m -s /bin/bash nominatim |
Database users
We will also make “nominatim” a database user, as well as the already created “www-data” (created by the apache2 package).
Nominatim will be a superuser and www-data will be a regular one.
This must be done as a database administrator. The “postgres” user by default is one (root is not). Change user:
1 |
su - postgres |
Create database users:
1 2 3 |
createuser -sdRe nominatim createuser -SDRe www-data exit |
Download and Compile Nominatim
Su to user nominatim:
1 |
su - nominatim |
Set some environment variables. We’ll use these later. These are the download locations for files and updates. Lines 3 and 4 below are for Finland. Customize to your needs.
Also set the BASE_URL to point to your server and install directory. The last forward slash seems to be important.
1 2 3 4 5 |
WIKIPEDIA_ARTICLES="http://www.nominatim.org/data/wikipedia_article.sql.bin" WIKIPEDIA_REDIRECTS="http://www.nominatim.org/data/wikipedia_redirect.sql.bin" OSM_LATEST="http://download.geofabrik.de/europe/finland-latest.osm.pbf" OSM_UPDATES="http://download.geofabrik.de/europe/finland-updates" BASE_URL="http://maps.example.org/nominatim/" |
You can browse available areas at download.geofabrik.de.
Clone the Repository from Github
We will use the release 2.3 branch:
1 2 3 4 5 |
git clone --recursive \ https://github.com/twain47/Nominatim.git \ --branch release_2.3 cd Nominatim |
The –recursive option is needed to clone everything, because the repository contains submodules.
Compile Nominatim
Simply:
1 2 3 |
./autogen.sh ./configure make |
If there were no errors, we are good. In case of missing libraries, check that you installed all requirements earlier.
Configure Local Settings
Some required settings should go to ~/Nominatim/settings/local.php:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
cat >settings/local.php <<EOF <?php // Paths @define('CONST_Postgresql_Version', '9.3'); @define('CONST_Postgis_Version', '2.1'); @define('CONST_Website_BaseURL', '$BASE_URL'); // Update process @define('CONST_Replication_Url', '$OSM_UPDATES'); @define('CONST_Replication_MaxInterval', '86400'); @define('CONST_Replication_Update_Interval', '86400'); @define('CONST_Replication_Recheck_Interval', '900'); ?> EOF |
The last four definitions are only required if you are going to do incremental updates.
Download Data
OpenStreetMap data (about 200 MB for Finland):
1 |
wget -O data/latest.osm.pbf $OSM_LATEST |
The OSM data is all that is really required. The rest below are optional.
Wikipedia data (about 1.4 GB):
1 2 |
wget -O data/wikipedia_article.sql.bin $WIKIPEDIA_ARTICLES wget -O data/wikipedia_redirect.sql.bin $WIKIPEDIA_REDIRECTS |
Special phrases for country codes and names (very small):
1 |
./utils/specialphrases.php --countries >data/specialphrases_countries.sql |
Special search phrases (a few megabytes):
1 |
./utils/specialphrases.php --wiki-import >data/specialphrases.sql |
Next we’ll import all this stuff into the database.
Import Data
The utils/setup.php will create a new database called “nominatim” and import the given .pbf file into it. This will take a long time depending on your PostgreSQL settings, available memory, disk speed and size of dataset. The full planet can take days to import on modern hardware. My small dataset took a bit over two hours.
1 2 3 4 |
./utils/setup.php \ --osm-file data/latest.osm.pbf \ --all --osm2pgsql-cache 1024 2>&1 \ | tee setup.log |
The messages will be saved into setup.log in case you need to look at them later.
If you had the Wikipedia data downloaded, the setup should have imported that automatically and told you about it. You can import the special phrases data if you downloaded it earlier using these commands:
1 2 |
psql -d nominatim -f data/specialphrases_countries.sql psql -d nominatim -f data/specialphrases.sql |
Database Production Settings
Now that the import is done, it is time to configure the database to settings that are suitable for production use. The following changes were made in /etc/postgresql/9.3/main/postgresql.conf:
1 2 3 |
maintenance_work_mem = 128M fsync = on full_page_writes = on |
You should also set the synchronous_commit directive to on if you have other databases running on this same database cluster. See the PostgreSQL Tuning section earlier in this post.
Apply changes:
1 |
pg_ctlcluster 9.3 main restart |
Create the Web Site
The following commands have to be run as root.
Create a directory to install the site into and set permissions:
1 2 3 4 |
mkdir /var/www/html/nominatim chown nominatim:www-data /var/www/html/nominatim chmod 755 /var/www/html/nominatim chmod g+s /var/www/html/nominatim |
Ask bots to keep out:
1 2 3 4 |
cat >/var/www/html/nominatim/robots.txt <<'EOF' User-agent: * Disallow: / EOF |
Apache configuration
Edit the default site configuration file /etc/apache2/sites-enabled/000-default.conf and make it look like something like below:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
<VirtualHost *:80> ServerName maps.example.org ServerAdmin webmaster@example.org DocumentRoot /var/www/html ErrorLog ${APACHE_LOG_DIR}/error.log CustomLog ${APACHE_LOG_DIR}/access.log combined <Directory "/var/www/html/nominatim"> Options FollowSymLinks MultiViews AddType text/html .php </Directory> </VirtualHost> |
Apply changes by restarting web server:
1 |
apache2ctl restart |
Install the Nominatim Web Site
The installation should be done as the nominatim user:
1 2 |
su - nominatim cd Nominatim |
Run the setup.php with the option to create the web site:
1 |
./utils/setup.php --create-website /var/www/html/nominatim |
At this point, the site is ready and you can point your browser to the base URL and try it out. It should look something like this:
Enable OSM Updates
We will configure a cron job to have the database updated periodically using diffs from Geofabrik.de. As the nominatim user:
1 2 |
rm settings/configuration.txt ./utils/setup.php --osmosis-init |
Enable hierarchical updates in the database (these were off during import to speed things up):
1 |
./utils/setup.php --create-functions --enable-diff-updates |
Run once to get up to date:
1 |
./utils/update.php --import-osmosis --no-npi |
There seems to be a time check in the import function that prevents another run immediately. Instead it will wait until the interval configured in the “Configure Local Settings” section earlier in this post has passed (default is 1 day).
We will add the command to crontab to be executed every monday at 03:00. Run:
1 |
crontab -e |
Add a new line to the crontab file looking like this:
1 |
00 03 * * mon /home/nominatim/Nominatim/utils/update.php --import-osmosis --no-npi >>/home/nominatim/nominatim-update.log 2>&1 |
A log file will be created as /home/nominatim/nominatim-update.log.
Wrap Up
That’s it! You can get JSON or XML data out of the database with http requests. A couple of examples:
Forward geocoding:
- http://maps.example.org/nominatim/search.php?format=jsonv2&q=mannerheimintie, helsinki
Reverse geocoding:
- http://maps.example.org/nominatim/reverse.php?format=jsonv2&lat=60.1805&lon=24.9434
Check out the Nominatim documentation for details.
This post was adapted from my notes of such an installation process. Let me know in the comments section if I introduced any mistakes. Other comments are also welcome!
Hi, you used 32 or 64 bit version?
Thanks!
64-bit Ubuntu
Hello Kortsi, great article.
When I try to access the reverse.php get this: “{” error “:” Unable to geocode “}”. I followed all the steps of your article, which can be?
Thanks for your attention.
That may mean there’s something wrong with your query. Did you give the “lat” and “lon” parameters? Those are required.
i have an erro in import data to postgres
Using projection SRS 4326 (Latlong)
NOTICE: table “place” does not exist, skipping
NOTICE: type “keyvalue” does not exist, skipping
NOTICE: type “wordscore” does not exist, skipping
NOTICE: type “stringlanguagetype” does not exist, skipping
NOTICE: type “keyvaluetype” does not exist, skipping
NOTICE: function get_connected_ways(pg_catalog.int4[]) does not exist, skipping
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Out of memory for dense node cache, reduce –cache size
Error occurred, cleaning up
ERROR: Error executing external command: /home/nominatim/Nominatim/osm2pgsql/osm2pgsql -lsc -O gazetteer –hstore -C 1024 -P 5432 -d nominatim /home/nominatim/Nominatim/data/latest.osm.pbf
Error executing external command: /home/nominatim/Nominatim/osm2pgsql/osm2pgsql -lsc -O gazetteer –hstore -C 1024 -P 5432 -d nominatim /home/nominatim/Nominatim/data/latest.osm.pbf
my web site it in “visual”
imagen of website
http://i60.tinypic.com/2mrv413.png
error when i want to shearch “restaurant in helsinki”
http://i60.tinypic.com/1zmju5l.png
I think you are running out of RAM because it says “Out of memory for dense node cache, reduce –cache size”. That’s why the whole import fails.
You should try with a lower –osm2pgsql-cache size. Or increase swap size for the machine.
i put my ubuntu server in a computer with a cpu of 3 core with 4 ram
I solved my problem, it was the cache size, I put 1024
then i have the problem
——
Out of memory for dense node cache, reduce –cache size
Error occurred, cleaning up
——
i saw in a external post that i was the cache size, and i put 512 and it’s work….
this is the post
https://github.com/twain47/Nominatim/issues/105
——
Yes, you do because osm2pgsql reports it is out of memory. It is currently trying to use 4GB and that seems to be too much. If you have 8GB of memory, try with 2048 and if that does not work either with 1024. Also make sure you have a few dozen GB of swap enabled.
——
in URL: http://localhost/nominatim/
i got map, but in top it showing error
Data: DB Error: no such table
———————
You don’t need to change anything in any files, just run ./utils/setup.php –osm-file –all –osm2pgsql-cache 1024 for the import as described in the installation instructions.
——————-
the problem was solved
http://i58.tinypic.com/2e3n4ed.png
Setting postgresql’s shared_buffers to 2GB when you have only 4GB RAM is not the best of ideas because this memory will be permanently reserved by the database server. Postgres recommends to use around 25% of memory but something like 256MB is more to the point with so little RAM because there are other applications running next to postgres during import (notably osm2pgsql which hogs lots of memory).
Also, don’t forbid to overcommit memory unless you have really a lot of RAM. osm2pgsql relies on being able to overcommit.
Hello Kortsi, very interesting article.
As I read it to install our Nominatim DB, I give some feedback on our experience.
The server used is vm with 4*3.1 Ghz, 15 Go Ram and high speed harddrives of 400 Go.
We imported only some country (fr, uk, it, be, de, sp, it). To process them we merged all files in a single one of 9 Go. It took 8 hours.
The import ran for about 36 hours and consumed 180 Go of disk.
Best Regards
Al
kortsi hello , good night!
when I try to use reverse geocoding , Info Some of addressparts not appear at the moment only appears this for example :
Return , BrazilRetornoBrazilbr
You know what might be causing the lack of data ?
Hi,
I am at the importing phase and when I type “./utils/setup.php –osm-file data/latest.osm.pbf –all –osm2pgsql-cache 1024 2>&1 | tee setup.log” I get the error “the path supplied to –osm-file does not exist”
Worked it out I had downloaded osn file to root “./utils/setup.php –osm-file latest.osm.pbf –all –osm2pgsql-cache 1024 2>&1 | tee setup.log” fixed it
Muchas gracias por la excelente explicacion de como usar esta herramienta, voy a trabajar esta configuracion para OpenGTS. Saludos
sorry – got a big problem. as a newbie i dont know much about apache
after
./utils/setup.php –create-website /var/www/html/nominatim
i got that error:
WARNING: Unable to access the website at http://maps.example.org/nominatim/
You may want to update settings/local.php with @define(‘CONST_Website_BaseURL’, ‘http://[HOST]/[PATH]/’);
Setup finished.
it words now. this error is solved
How did you solved it? Thanks in advance!
i am getting the same error, how did you solved it ?
Same here… can anyone help me?
What more can I say but it works!
The most complete and accurate set up instructions I have found!
Where can u find the error log after installing the Nominatim server ?
Hi Kortsi.
This a best guide i ever read about Nominatim installation.
Even this time some changes need to be done that different from what you said.
Anyway, i am enjoying it, that’s great.
Thank you so much.
lieubv@gmail.com
Excellent instructions – these are the first ones that actually have worked first time. I have imported a small sample country to ensure that the platform is working as expected.
I will not be using the updating functionality. How do I now import another country? What is the command required?
Thanks!!
If I wanted to use say jtiledownloader to export tiles please could you tell me how?
Thanks
Andy.
Databases are different, you have to do a full install of a tileserver as well.
The –create-website option was removed last year and won’t work.
https://github.com/twain47/Nominatim/commit/85c72cdccdb6adcad7a78643c569935e88a73182
Thank you!! It solved my problem. Now nominatim is working.
Thank You !! It solved my problem, now nominatim is working.
I am looking at servicing 100 reverse geocoding requests per second.
Does anybody know what sort of machine could do this, or would multiple servers be required?
I have this running on a Virtualbox VM with 4 cores and 4GB RAM but can only do about 10 per second.
Thanks
Andy.
Hi guys,
We have been trying to setup a server for quite some time now and are getting database errors every time. After the import into Postgres (the last step) it’s an error every singl time.
Is there a way/place where I can get an indexed DB dump which I can import and skip the import process??
Hope you can help (i will be watching this threat)
I haven’t seen any DB dumps around. One issue might be size. I did a full planet import about a year ago and the resulting database took nearly a terabyte of disk space.
What kind of an error are you getting?
Hello, thanks for the information but i have a problem in import data to postgres:
Using projection SRS 4326 (Latlong)
NOTICE: table “place” does not exist, skipping
NOTICE: type “keyvalue” does not exist, skipping
NOTICE: type “wordscore” does not exist, skipping
NOTICE: type “stringlanguagetype” does not exist, skipping
NOTICE: type “keyvaluetype” does not exist, skipping
NOTICE: function get_connected_ways(pg_catalog.int4[]) does not exist, skipping
Allocating memory for dense node cache
Out of memory for node cache dense index, try using “–cache-strategy sparse” instead
Error occurred, cleaning up
ERROR: Error executing external command: /home/nominatim/Nominatim/osm2pgsql/osm2pgsql -lsc -O gazetteer –hstore -C 256 -P 5432 -d nominatim /home/nominatim/Nominatim/data/latest.osm.pbf
Error executing external command: /home/nominatim/Nominatim/osm2pgsql/osm2pgsql -lsc -O gazetteer –hstore -C 256 -P 5432 -d nominatim /home/nominatim/Nominatim/data/latest.osm.pbf
PD:I have 6 GB of RAM And lower the size of cache-size up to 256 MB but I do not solve my problem.
This is a nice piece work and I have used for about 5 years now on my Ubuntu16 box, now I want to upgrade from Nominatim 2.3 to 4xx. Do you know any good sites that deals with this migration please?
Thanks