[osmosis-dev] Challenge: Loading Europe-latest.pbf into pgsql using osmosis

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[osmosis-dev] Challenge: Loading Europe-latest.pbf into pgsql using osmosis

Sami Snunu
Hi,

I believe this issue many of us encountered,..
I'm using Osmosis to import full Europe (18.2 GB) into my postgreSQL db with pgsnapshot schema to build my own queries and analysis.

Command used:
osmosis --read-pbf-fast "D:\Data\OSM\europe-latest.osm.pbf" workers=30 
--log-progress 
--write-pgsql host="localhost" database="Europe" user=".." password=".."

Not sure how I can get it into the db. It's still running since 11 days ago, Osmosis generated temp files are stored on the SSD disk, yet nothing loaded into the database.

I've implemented several tweaks found on wiki's for the osmosis\PostgreSQL (postgresql.conf file), also Osmosis JAVACMD_OPTIONS to improve the performance, and temp files settings.

I know Osmosis by default drops the indexes before loading. 

Also tried different approaches such as:
- Divide and conquer approach, load country by country, but chances are high and happened for duplicated pk_relation_id, or pk_user_id violation errors.
- Used (--write-pgsql-dump) then psql copy, but got the same errors for duplicate violation of primary keys.
- Converted the file from .pbf into .osm, then use (--read-xml), hopeless case, so slow.

Any idea how to sort this out?

--OS
Windows 7 Pro (64-bit) on i7 @2.7GHz

--Appl
PostgreSQL 9.6.1.1 (x64) [latest version]
Postgis 2.3.1 [latest version]
Java 1.8-111 [latest version]
Osmosis 0.45 [latest version]

--Hardware:
32-GB RAM
1-TB SSD


Sami


_______________________________________________
osmosis-dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/osmosis-dev