Should my Overpass server always be using 100% CPU?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Should my Overpass server always be using 100% CPU?

James Chevalier
I've recently created my own Overpass server on an m5.large EC2 instance in
AWS
(2 vCPU, 8GB Memory). It's continually at 100% CPU usage, and I'm wondering if
that's normal or if I should be using a larger EC2 instance or if something
isn't working properly. I'm about to run many queries against this server for
thousands of city's street data, so I want to make sure it's as healthy as
possible.

Truncated output of top:
%CPU %MEM     TIME+ COMMAND
99.0 65.2 953:11.45 ./osm3s_query --progress --rules
73.4  1.1   0:20.51 ./update_from_dir --osc-dir=/tmp/osm-3s_update_O9Zfhz --
version=2019-09-06T17\:32\:01Z --meta --flush-size=0

Contents of my rules_loop.log file:
2019-09-04 00:41:05: update started
2019-09-05 00:42:02: update finished
2019-09-05 00:42:05: update started
2019-09-06 00:43:12: update finished
2019-09-06 00:43:15: update started
Reply | Threaded
Open this post in threaded view
|

Re: Should my Overpass server always be using 100% CPU?

James Chevalier
I had to resize my EC2 instance, so I stopped the server by running:

-    bin/dispatcher --osm-base --terminate
-    bin/dispatcher --areas --terminate
-    sudo shutdown now -h

Then I started the server back up and ran these commands:

-    rm -f db/osm3s_v0.7.55_osm_base
-    nohup bin/dispatcher --osm-base --meta --db-dir="db/" >> osm_base.out &
-    chmod 666 "db/osm3s_v0.7.55_osm_base"
-    nohup bin/fetch_osc.sh `cat db/replicate_id`
"http://planet.openstreetmap.org/replication/minute/" "diffs/" >>
fetch_osc.out &
-    nohup bin/apply_osc_to_db.sh "diffs/" `cat db/replicate_id` --meta=yes >>
apply_osc_to_db.out &
-    rm -f db/osm3s_v0.7.54_areas
-    nohup bin/dispatcher --areas --db-dir="db/" >> areas.out &

I did not re-run nohup bin/rules_loop.sh "db" & after reboot. I'm unsure if I
need to, and things seem to be working well without it. The server does not
run at continuous 100% CPU usage any more, after the reboot.

I based this process on the reboot script that I found in the bin directory.
I'm unsure if what I did is correct...
https://github.com/drolbr/Overpass-API/blob/master/src/bin/reboot.sh
Reply | Threaded
Open this post in threaded view
|

Re: Should my Overpass server always be using 100% CPU?

Igor Brejc
Hi James,

Out of curiosity: what kind of EC2 instance are you running now?

Igor



On Wed, Sep 11, 2019, 01:02 James Chevalier <[hidden email]> wrote:
I had to resize my EC2 instance, so I stopped the server by running:

-    bin/dispatcher --osm-base --terminate
-    bin/dispatcher --areas --terminate
-    sudo shutdown now -h

Then I started the server back up and ran these commands:

-    rm -f db/osm3s_v0.7.55_osm_base
-    nohup bin/dispatcher --osm-base --meta --db-dir="db/" >> osm_base.out &
-    chmod 666 "db/osm3s_v0.7.55_osm_base"
-    nohup bin/fetch_osc.sh `cat db/replicate_id`
"http://planet.openstreetmap.org/replication/minute/" "diffs/" >>
fetch_osc.out &
-    nohup bin/apply_osc_to_db.sh "diffs/" `cat db/replicate_id` --meta=yes >>
apply_osc_to_db.out &
-    rm -f db/osm3s_v0.7.54_areas
-    nohup bin/dispatcher --areas --db-dir="db/" >> areas.out &

I did not re-run nohup bin/rules_loop.sh "db" & after reboot. I'm unsure if I
need to, and things seem to be working well without it. The server does not
run at continuous 100% CPU usage any more, after the reboot.

I based this process on the reboot script that I found in the bin directory.
I'm unsure if what I did is correct...
https://github.com/drolbr/Overpass-API/blob/master/src/bin/reboot.sh
Reply | Threaded
Open this post in threaded view
|

Re: Should my Overpass server always be using 100% CPU?

James Chevalier
I started with an m5.large (2 vCPU, 8GiB RAM) and increased it to an
m5.4xlarge (16 vCPU, 64GiB RAM).

In doing so, I've found that I can't get the server to exceed 50% CPU usage,
and I'm doing a lot of queries .. mmd is helping me with that over in github,
if you're interested in that issue:
https://github.com/drolbr/Overpass-API/issues/540
Reply | Threaded
Open this post in threaded view
|

Re: Should my Overpass server always be using 100% CPU?

Igor Brejc
Thanks!
I'm running an Overpass instance on my home server and I was wondering what kind of infrastructure (and cost) is needed to run Overpass on AWS. I'm not running that many queries though (only an occasional query over a city/region area size).

Igor

On Wed, Sep 11, 2019 at 4:43 PM James Chevalier <[hidden email]> wrote:
I started with an m5.large (2 vCPU, 8GiB RAM) and increased it to an
m5.4xlarge (16 vCPU, 64GiB RAM).

In doing so, I've found that I can't get the server to exceed 50% CPU usage,
and I'm doing a lot of queries .. mmd is helping me with that over in github,
if you're interested in that issue:
https://github.com/drolbr/Overpass-API/issues/540