Overpass API Area Creation in a Custom Server

[sameera.palage]

Hello everyone,

I am trying to host my own OSM Overpass server for my master thesis research.
I downloaded the database with attic data and could successfully complete the
installation up to setting up the Web API. Until this point everything works
fine.

I would also like to setup the areas in the database, so I could query data by
an area. I started the area creation process two days ago, which is still
running. Is it normal to run it for two days (the official documentation says
it would take up to 24 hours).  I included some details I though might
relevant to this case

=================================================================================

These are my installation directories

$DB_DIR = /data/osm/osm-3s_v0.7.55/bin/database_dir
$EXEC_DIR = /data/osm/osm-3s_v0.7.55/

The command I ran for area creation is

nohup /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/
database_dir &

output of ps -ef | grep rules

sameera   9990  9961  0 Jul21 pts/0    00:01:47 bash /data/osm/osm-3s_v0.7.55/
bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/database_dir
sameera  21333  9961  0 13:38 pts/0    00:00:00 grep --color=auto rules

Any help is appreciated.
Thank you very much

kind regards,
Sameera

[yann Guillerm]

hi !

as far as i have understood, the rules loop process is looping forever BUT after a while all the area are created  (less than 2 hours for French country for example).

On my server, to known when it's ready i make a request using area on multiple point of the map : 

  - if the request is not correct it's means that the area are note yet created.

hope this helps.

Yann.

Le mar. 23 juil. 2019 à 15:46, <[hidden email]> a écrit :

Hello everyone,

I am trying to host my own OSM Overpass server for my master thesis research.
I downloaded the database with attic data and could successfully complete the
installation up to setting up the Web API. Until this point everything works
fine.

I would also like to setup the areas in the database, so I could query data by
an area. I started the area creation process two days ago, which is still
running. Is it normal to run it for two days (the official documentation says
it would take up to 24 hours).  I included some details I though might
relevant to this case

=================================================================================

These are my installation directories

$DB_DIR = /data/osm/osm-3s_v0.7.55/bin/database_dir
$EXEC_DIR = /data/osm/osm-3s_v0.7.55/

The command I ran for area creation is

nohup /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/
database_dir &

output of ps -ef | grep rules

sameera   9990  9961  0 Jul21 pts/0    00:01:47 bash /data/osm/osm-3s_v0.7.55/
bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/database_dir
sameera  21333  9961  0 13:38 pts/0    00:00:00 grep --color=auto rules

Any help is appreciated.
Thank you very much

kind regards,
Sameera

[sameera.palage]

Hi Yann,

Thank you very much for the reply. I issued an area query and I am getting below error.

{

• version: 0.6,
• generator: "Overpass API 0.7.55.7 8b86ff77",
• osm3s: 

  {

  • timestamp_osm_base: "",
  • copyright: "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."

  },
• elements: [ ],
• remark: "runtime error: open64: 2 No such file or directory /data/osm/osm-3s_v0.7.55/bin/database_dir/areas.bin File_Blocks::File_Blocks::1"

}

In fact, there is no such directory areas.bin created yet. So I think the rules loop is having some issues :(

kind regards,

Sameera

On Tue, 23 Jul 2019 at 16:10, yann Guillerm <[hidden email]> wrote:

hi !

as far as i have understood, the rules loop process is looping forever BUT after a while all the area are created  (less than 2 hours for French country for example).

On my server, to known when it's ready i make a request using area on multiple point of the map : 

  - if the request is not correct it's means that the area are note yet created.

hope this helps.

Yann.

Le mar. 23 juil. 2019 à 15:46, <[hidden email]> a écrit :

Hello everyone,

I am trying to host my own OSM Overpass server for my master thesis research.
I downloaded the database with attic data and could successfully complete the
installation up to setting up the Web API. Until this point everything works
fine.

I would also like to setup the areas in the database, so I could query data by
an area. I started the area creation process two days ago, which is still
running. Is it normal to run it for two days (the official documentation says
it would take up to 24 hours).  I included some details I though might
relevant to this case

=================================================================================

These are my installation directories

$DB_DIR = /data/osm/osm-3s_v0.7.55/bin/database_dir
$EXEC_DIR = /data/osm/osm-3s_v0.7.55/

The command I ran for area creation is

nohup /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/
database_dir &

output of ps -ef | grep rules

sameera   9990  9961  0 Jul21 pts/0    00:01:47 bash /data/osm/osm-3s_v0.7.55/
bin/rules_loop.sh /data/osm/osm-3s_v0.7.55/bin/database_dir
sameera  21333  9961  0 13:38 pts/0    00:00:00 grep --color=auto rules

Any help is appreciated.
Thank you very much

kind regards,
Sameera

[Roland Olbricht]

Hi Sameera,

could you please check what the file nohup.out contains. There should be error messages there.

As a blind guess, please try the following:

- Stop the rules_loop.sh process, e.g. by "kill 9990".

- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
   replace there the line
     DB_DIR="`pwd`/$1"
   with
     DB_DIR="$1"

- Restart the process, i.e. run the command again please.

I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.

Kind regards,
Roland

[sameera.palage]

Hi Roland,

Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.

After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0

After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0

After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0

After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0

After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0

</osm>

runtime error: Query timed out in "make-area" at line 257 after 86401 seconds.

<?xml version="1.0" encoding="UTF-8"?>

<osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77">

<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>

<meta osm_base=""/>

kind regards,

Sameera

On Wed, 24 Jul 2019 at 21:55, Roland Olbricht <[hidden email]> wrote:

Hi Sameera,

could you please check what the file nohup.out contains. There should be error messages there.

As a blind guess, please try the following:

- Stop the rules_loop.sh process, e.g. by "kill 9990".

- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
   replace there the line
     DB_DIR="`pwd`/$1"
   with
     DB_DIR="$1"

- Restart the process, i.e. run the command again please.

I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.

Kind regards,
Roland

[yann Guillerm]

whhooo 24H ! 

i use rules_loop on 2 server where i have priorize the process down and it always make it (for world db) within a few hours.

Actually for France only db, but the server is use for a lot more processes using sometime 100% of the processor, and it take 1h20 to finish the loop. The server specification : intel Xeon E3 1230 v6, 16go Ram.

in the pre-requirement, the documentation said to have at least 4Go ram, a lot of hard disk and that processor is not revelant.

are you sure that processor and ram are free to use ? 

sincerly, 

Yann.

Le jeu. 25 juil. 2019 à 08:12, Sameera Bandara <[hidden email]> a écrit :

Hi Roland,

Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.

After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0

After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0

After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0

After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0

After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0

</osm>

runtime error: Query timed out in "make-area" at line 257 after 86401 seconds.

<?xml version="1.0" encoding="UTF-8"?>

<osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77">

<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>

<meta osm_base=""/>

kind regards,

Sameera

On Wed, 24 Jul 2019 at 21:55, Roland Olbricht <[hidden email]> wrote:

Hi Sameera,

could you please check what the file nohup.out contains. There should be error messages there.

As a blind guess, please try the following:

- Stop the rules_loop.sh process, e.g. by "kill 9990".

- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
   replace there the line
     DB_DIR="`pwd`/$1"
   with
     DB_DIR="$1"

- Restart the process, i.e. run the command again please.

I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.

Kind regards,
Roland

[sameera.palage]

Hey Yann,

I have a dedicated server only for hosting OSM with 2 GHz 8 cores and 16 GB RAM. But I didn't prioritize down the processes as still I am not using the server and thought the processes would finish fast. Below are the consumption stats. It consumes 100% CPU :O

  1  [                                       0.0%]   5  [                                       0.0%]

  2  [                                       0.0%]   6  [                                       0.0%]

  3  [||                                     1.3%]   7  [                                       0.0%]

  4  [||||||||||||||||||||||||||||||||||||||99.3%]   8  [                                       0.0%]

  Mem[||||||||||||||||||||||||||||||||4.99G/15.7G]   Tasks: 41, 103 thr; 2 running

  Swp[||                               33.2M/861M]   Load average: 1.00 1.00 1.00 

                                                     Uptime: 9 days, 22:19:58

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command

30013 sameera    20   0 4929M 4903M  2620 R 100. 30.6 10h24:01 ./osm3s_query --progress --rules

31945 sameera    20   0 29544  4012  3320 R  1.3  0.0  0:03.52 htop

 8737 sameera    20   0 19240  1680   680 S  0.0  0.0  2:19.47 /data/osm/osm-3s_v0.7.55/bin/dispatcher --ar

 8692 sameera    20   0 33476 14168     4 S  0.0  0.1  2:22.61 /data/osm/osm-3s_v0.7.55/bin/dispatcher --os

 1757 root       20   0  283M   804     0 S  0.0  0.0  0:12.68 /usr/lib/accountsservice/accounts-daemon

 8546 root       20   0 69960  4240  3036 S  0.0  0.0  0:11.26 /usr/sbin/apache2 -k start

 1779 root       20   0  283M   804     0 S  0.0  0.0  0:12.58 /usr/lib/accountsservice/accounts-daemon

On Thu, 25 Jul 2019 at 09:01, yann Guillerm <[hidden email]> wrote:

whhooo 24H ! 

i use rules_loop on 2 server where i have priorize the process down and it always make it (for world db) within a few hours.

Actually for France only db, but the server is use for a lot more processes using sometime 100% of the processor, and it take 1h20 to finish the loop. The server specification : intel Xeon E3 1230 v6, 16go Ram.

in the pre-requirement, the documentation said to have at least 4Go ram, a lot of hard disk and that processor is not revelant.

are you sure that processor and ram are free to use ? 

sincerly, 

Yann.

Le jeu. 25 juil. 2019 à 08:12, Sameera Bandara <[hidden email]> a écrit :

Hi Roland,

Thank you very much for the reply. I actually already sorted out the issue after checkin nohup.out. As you have mentioned the issue was something with absolute paths. I specified the relative path to the database_dir and it worked. Below is the error I got initially.

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 40: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 41: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules/areas.osm3s: No such file or directory

/data/osm/osm-3s_v0.7.55/bin/rules_loop.sh: line 42: /data/osm/osm-3s_v0.7.55/bin//data/osm/osm-3s_v0.7.55/bin/database_dir/rules_loop.log: No such file or directory

Now the areas are populated and I can issue queries for areas and get results, however the log says the query was timed-out. And it ran exactly 24 hours.

After 23h59m3s: in "make-area", part 0, on line 257. Stack: 0 of 0 8442756 of 0

After 23h59m18s: in "recurse", part 0, on line 255. Stack: 0 of 0 8445531 of 0 0 of 0

After 23h59m33s: in "recurse", part 0, on line 255. Stack: 0 of 0 8447843 of 0 0 of 0

After 23h59m48s: in "make-area", part 0, on line 257. Stack: 0 of 0 8450335 of 0

After 24h0m1s: in "make-area", part 0, on line 257. Stack: 0 of 0 8452832 of 0

</osm>

runtime error: Query timed out in "make-area" at line 257 after 86401 seconds.

<?xml version="1.0" encoding="UTF-8"?>

<osm version="0.6" generator="Overpass API 0.7.55.7 8b86ff77">

<note>The data included in this document is from www.openstreetmap.org. The data is made available under ODbL.</note>

<meta osm_base=""/>

kind regards,

Sameera

On Wed, 24 Jul 2019 at 21:55, Roland Olbricht <[hidden email]> wrote:

Hi Sameera,

could you please check what the file nohup.out contains. There should be error messages there.

As a blind guess, please try the following:

- Stop the rules_loop.sh process, e.g. by "kill 9990".

- Edit the file /data/osm/osm-3s_v0.7.55/bin/rules_loop.sh:
   replace there the line
     DB_DIR="`pwd`/$1"
   with
     DB_DIR="$1"

- Restart the process, i.e. run the command again please.

I coincidentally ran on the Russian instance into the problem that the script fails if its argument is an absolute path because of that line.

Kind regards,
Roland