|
|
Hello everyone,
I am trying to host my own OSM Overpass server for my master thesis research.
I downloaded the database with attic data and could successfully complete the
installation up to setting up the Web API. Until this point everything works
fine.
I would also like to setup the areas in the database, so I could query data by
an area. I started the area creation process two days ago, which is still
running. Is it normal to run it for two days (the official documentation says
it would take up to 24 hours). I included some details I though might
relevant to this case
=================================================================================
These are my installation directories
$DB_DIR = /data/osm/osm-3s_v0.7.55/bin/database_dir
$EXEC_DIR = /data/osm/osm-3s_v0.7.55/
The command I ran for area creation is
nohup /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/
database_dir &
output of ps -ef | grep rules
sameera 9990 9961 0 Jul21 pts/0 00:01:47 bash /data/osm/osm-3s_v0.7.55/
bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/database_dir
sameera 21333 9961 0 13:38 pts/0 00:00:00 grep --color=auto rules
Any help is appreciated.
Thank you very much
kind regards,
Sameera
|
|
hi !
as far as i have understood, the rules loop process is looping forever BUT after a while all the area are created (less than 2 hours for French country for example).
On my server, to known when it's ready i make a request using area on multiple point of the map : - if the request is not correct it's means that the area are note yet created.
hope this helps.
Yann. Hello everyone,
I am trying to host my own OSM Overpass server for my master thesis research.
I downloaded the database with attic data and could successfully complete the
installation up to setting up the Web API. Until this point everything works
fine.
I would also like to setup the areas in the database, so I could query data by
an area. I started the area creation process two days ago, which is still
running. Is it normal to run it for two days (the official documentation says
it would take up to 24 hours). I included some details I though might
relevant to this case
=================================================================================
These are my installation directories
$DB_DIR = /data/osm/osm-3s_v0.7.55/bin/database_dir
$EXEC_DIR = /data/osm/osm-3s_v0.7.55/
The command I ran for area creation is
nohup /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/
database_dir &
output of ps -ef | grep rules
sameera 9990 9961 0 Jul21 pts/0 00:01:47 bash /data/osm/osm-3s_v0.7.55/
bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/database_dir
sameera 21333 9961 0 13:38 pts/0 00:00:00 grep --color=auto rules
Any help is appreciated.
Thank you very much
kind regards,
Sameera
|
|
Hi Yann,
Thank you very much for the reply. I issued an area query and I am getting below error.
{version: 0.6, generator: "Overpass API 0.7.55.7 8b86ff77", osm3s: { timestamp_osm_base: "", copyright: "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
}, elements: [ ], remark: "runtime error: open64: 2 No such file or directory /data/osm/osm-3s_v0.7.55/bin/database_dir/areas.bin File_Blocks::File_Blocks::1" }
In fact, there is no such directory areas.bin created yet. So I think the rules loop is having some issues :(
kind regards, Sameera
hi !
as far as i have understood, the rules loop process is looping forever BUT after a while all the area are created (less than 2 hours for French country for example).
On my server, to known when it's ready i make a request using area on multiple point of the map : - if the request is not correct it's means that the area are note yet created.
hope this helps.
Yann.
Hello everyone,
I am trying to host my own OSM Overpass server for my master thesis research.
I downloaded the database with attic data and could successfully complete the
installation up to setting up the Web API. Until this point everything works
fine.
I would also like to setup the areas in the database, so I could query data by
an area. I started the area creation process two days ago, which is still
running. Is it normal to run it for two days (the official documentation says
it would take up to 24 hours). I included some details I though might
relevant to this case
=================================================================================
These are my installation directories
$DB_DIR = /data/osm/osm-3s_v0.7.55/bin/database_dir
$EXEC_DIR = /data/osm/osm-3s_v0.7.55/
The command I ran for area creation is
nohup /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/
database_dir &
output of ps -ef | grep rules
sameera 9990 9961 0 Jul21 pts/0 00:01:47 bash /data/osm/osm-3s_v0.7.55/
bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/database_dir
sameera 21333 9961 0 13:38 pts/0 00:00:00 grep --color=auto rules
Any help is appreciated.
Thank you very much
kind regards,
Sameera
|
|
Hi Sameera,
could you please check what the file nohup.out contains. There should be error messages there.
As a blind guess, please try the following:
- Stop the rules_loop.sh process, e.g. by "kill 9990".
- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
replace there the line
DB_DIR="`pwd`/$1"
with
DB_DIR="$1"
- Restart the process, i.e. run the command again please.
I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.
Kind regards,
Roland
|
|
Hi Roland,
Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.
/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory
Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.
After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0 After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0 After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0 After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0 After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0
</osm> runtime error: Query timed out in "make-area" at line 257 after 86401 seconds. <?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77"> <note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note> <meta osm_base=""/>
kind regards, Sameera Hi Sameera,
could you please check what the file nohup.out contains. There should be error messages there.
As a blind guess, please try the following:
- Stop the rules_loop.sh process, e.g. by "kill 9990".
- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
replace there the line
DB_DIR="`pwd`/$1"
with
DB_DIR="$1"
- Restart the process, i.e. run the command again please.
I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.
Kind regards,
Roland
|
|
whhooo 24H ! i use rules_loop on 2 server where i have priorize the process down and it always make it (for world db) within a few hours. Actually for France only db, but the server is use for a lot more processes using sometime 100% of the processor, and it take 1h20 to finish the loop. The server specification : intel Xeon E3 1230 v6, 16go Ram.
in the pre-requirement, the documentation said to have at least 4Go ram, a lot of hard disk and that processor is not revelant.
are you sure that processor and ram are free to use ?
sincerly,
Yann.
Le jeu. 25 juil. 2019 à 08:12, Sameera Bandara < [hidden email]> a écrit : Hi Roland,
Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.
/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory
Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.
After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0 After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0 After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0 After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0 After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0
</osm> runtime error: Query timed out in "make-area" at line 257 after 86401 seconds. <?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77"> <note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note> <meta osm_base=""/>
kind regards, Sameera
Hi Sameera,
could you please check what the file nohup.out contains. There should be error messages there.
As a blind guess, please try the following:
- Stop the rules_loop.sh process, e.g. by "kill 9990".
- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
replace there the line
DB_DIR="`pwd`/$1"
with
DB_DIR="$1"
- Restart the process, i.e. run the command again please.
I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.
Kind regards,
Roland
|
|
Hey Yann,
I have a dedicated server only for hosting OSM with 2 GHz 8 cores and 16 GB RAM. But I didn't prioritize down the processes as still I am not using the server and thought the processes would finish fast. Below are the consumption stats. It consumes 100% CPU :O
1 [ 0.0%] 5 [ 0.0%]
2 [ 0.0%] 6 [ 0.0%]
3 [|| 1.3%] 7 [ 0.0%]
4 [||||||||||||||||||||||||||||||||||||||99.3%] 8 [ 0.0%]
Mem[||||||||||||||||||||||||||||||||4.99G/15.7G] Tasks: 41, 103 thr; 2 running
Swp[|| 33.2M/861M] Load average: 1.00 1.00 1.00
Uptime: 9 days, 22:19:58
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
30013 sameera 20 0 4929M 4903M 2620 R 100. 30.6 10h24:01 ./osm3s_query --progress --rules
31945 sameera 20 0 29544 4012 3320 R 1.3 0.0 0:03.52 htop
8737 sameera 20 0 19240 1680 680 S 0.0 0.0 2:19.47 /data/osm/osm-3s_v0.7.55/bin/dispatcher --ar
8692 sameera 20 0 33476 14168 4 S 0.0 0.1 2:22.61 /data/osm/osm-3s_v0.7.55/bin/dispatcher --os
1757 root 20 0 283M 804 0 S 0.0 0.0 0:12.68 /usr/lib/accountsservice/accounts-daemon
8546 root 20 0 69960 4240 3036 S 0.0 0.0 0:11.26 /usr/sbin/apache2 -k start
1779 root 20 0 283M 804 0 S 0.0 0.0 0:12.58 /usr/lib/accountsservice/accounts-daemon
whhooo 24H ! i use rules_loop on 2 server where i have priorize the process down and it always make it (for world db) within a few hours. Actually for France only db, but the server is use for a lot more processes using sometime 100% of the processor, and it take 1h20 to finish the loop. The server specification : intel Xeon E3 1230 v6, 16go Ram.
in the pre-requirement, the documentation said to have at least 4Go ram, a lot of hard disk and that processor is not revelant.
are you sure that processor and ram are free to use ?
sincerly,
Yann.
Le jeu. 25 juil. 2019 à 08:12, Sameera Bandara < [hidden email]> a écrit : Hi Roland,
Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.
/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory
Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.
After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0 After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0 After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0 After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0 After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0
</osm> runtime error: Query timed out in "make-area" at line 257 after 86401 seconds. <?xml version="1.0" encoding="UTF-8"?> <osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77"> <note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note> <meta osm_base=""/>
kind regards, Sameera
Hi Sameera,
could you please check what the file nohup.out contains. There should be error messages there.
As a blind guess, please try the following:
- Stop the rules_loop.sh process, e.g. by "kill 9990".
- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
replace there the line
DB_DIR="`pwd`/$1"
with
DB_DIR="$1"
- Restart the process, i.e. run the command again please.
I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.
Kind regards,
Roland
|
|