API a lot slower?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

API a lot slower?

Maarten Deen
Is it just me or just today or is the API a lot slower after the move on
sunday? I'm downloading a number of busrelations and it takes a lot
longer than before the weekend.

Regards,
Maarten

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

James-2

On Tue, Jul 17, 2018, 5:46 AM Maarten Deen, <[hidden email]> wrote:
Is it just me or just today or is the API a lot slower after the move on
sunday? I'm downloading a number of busrelations and it takes a lot
longer than before the weekend.

Regards,
Maarten

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Grant Slater
Hi All,

Yes, the API will be slower. We are running on our backup hardware for
the next few weeks.

Our primary hardware is moving to a new data centre next week, and
will take some time to get up and running.
It is a major move for the server operations group. Will announce more
details in the next few days.

Kind regards,

Grant

On 17 July 2018 at 10:50, James <[hidden email]> wrote:

> https://twitter.com/OSM_Tech/status/1018447091423744000?s=20
>
> On Tue, Jul 17, 2018, 5:46 AM Maarten Deen, <[hidden email]> wrote:
>>
>> Is it just me or just today or is the API a lot slower after the move on
>> sunday? I'm downloading a number of busrelations and it takes a lot
>> longer than before the weekend.
>>
>> Regards,
>> Maarten
>>
>> _______________________________________________
>> talk mailing list
>> [hidden email]
>> https://lists.openstreetmap.org/listinfo/talk
>
>
> _______________________________________________
> talk mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/talk
>

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Daniel Koć
W dniu 17.07.2018 o 12:22, Grant Slater pisze:
> Our primary hardware is moving to a new data centre next week, and
> will take some time to get up and running.

What data center do you mean? Is it the one which OSMF was looking for
at the beginning of the year?

https://blog.openstreetmap.org/2018/02/19/osmf-request-for-proposals-data-centre-2018/

Could you give some more details about it?

--
"My method is uncertain/ It's a mess but it's working" [F. Apple]


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Grant Slater
On 17 July 2018 at 13:42, Daniel Koć <daniel@koć.pl> wrote:

> W dniu 17.07.2018 o 12:22, Grant Slater pisze:
>> Our primary hardware is moving to a new data centre next week, and
>> will take some time to get up and running.
>
> What data center do you mean? Is it the one which OSMF was looking for
> at the beginning of the year?
>
> https://blog.openstreetmap.org/2018/02/19/osmf-request-for-proposals-data-centre-2018/
>
> Could you give some more details about it?
>

Yes. We are moving some core servers to an Equinix data centre in Amsterdam, NL.

OpenStreetMap's Brexit ;-) </sarcasm>

Kind regards,

Grant
Part of the OSM Operations team.

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Daniel Koć
W dniu 17.07.2018 o 14:55, Grant Slater pisze:

> Yes. We are moving some core servers to an Equinix data centre in
> Amsterdam, NL.

Thanks for the message! However given how big this change is and that it
directly affects comfort of using OSM services, I would be glad to hear
much more details what are the plans and why there?

As I understand, OSMF technical department (I mean Operations Working
Group and Engineering Working Group) wants to engage more people
(https://blog.openstreetmap.org/category/organisation/osmf-working-groups/)
and giving some general informations in public is a basic tool to
encourage them.

--
"My method is uncertain/ It's a mess but it's working" [F. Apple]



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Frederik Ramm
Daniel,

On 18.07.2018 14:18, Daniel Koć wrote:
> Thanks for the message! However given how big this change is and that it
> directly affects comfort of using OSM services, I would be glad to hear
> much more details what are the plans and why there?

The server move is currently eating up all the extra resources that OWG
have (and more). We'll certainly be able to document it after the fact,
but there is no free time available to provide any running commentary
for the public while the move is being planned and executed.

The decision for Amsterdam was made after carefully reviewing the
bids/offers that were received in response to the request for proposals
put out in February this year. The request also explained why the move
was necessary.

> As I understand, OSMF technical department (I mean Operations Working
> Group and Engineering Working Group) wants to engage more people
> (https://blog.openstreetmap.org/category/organisation/osmf-working-groups/)
> and giving some general informations in public is a basic tool to
> encourage them.

Yes, the situation would be more relaxed if work were distributed on
more shoulders but at this precise point in time I'd like our operations
group to focus on getting the move done with minimum fuss, and not on
recruiting new people!

Bye
Frederik

--
Frederik Ramm  ##  eMail [hidden email]  ##  N49°00'09" E008°23'33"

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Tom Hughes-3
In reply to this post by Maarten Deen
On 17/07/18 10:42, Maarten Deen wrote:

> Is it just me or just today or is the API a lot slower after the move on
> sunday? I'm downloading a number of busrelations and it takes a lot
> longer than before the weekend.

I know the cause of this has been explained already but I'd be
interested to know exactly which API call you are referring to
here as it may be that (long term at least) there is something
here which should be fixed but it rather depends if it is a call
being handled by rails or one being handled by cgimap.

Tom

--
Tom Hughes ([hidden email])
http://compton.nu/

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Daniel Koć
In reply to this post by Frederik Ramm
W dniu 18.07.2018 o 14:40, Frederik Ramm pisze:

> Yes, the situation would be more relaxed if work were distributed on
> more shoulders but at this precise point in time I'd like our operations
> group to focus on getting the move done with minimum fuss, and not on
> recruiting new people!

First of all - I appreciate the work behind the hardware and software
support in OSMF. That is a huge bonus for me that in general I can focus
on developing default style without special care for the infrastructure
it will run on. Kudos for you all!

I also absolutely don't mean this precise moment. Being system
administrator myself, I can imagine how stressful and complicated that
could be. I just thought that there could be some message after the
solution has been selected and before the move has started. I try to
warn my users beforehand when doing anything that could directly affect
their work or if there's some risk involved. At least that's a Best
Practice TM. :-)

What I wanted to express was just a friendly moniker that OWG/EWG work
is still not visible from my perspective (even if I'm an active member
of OSM community) and that it could help to gain some people this way.
Especially as this aim was explicitly stated by technical team
(precisely I mean
https://blog.gravitystorm.co.uk/2016/07/28/getting-involved-in-owg/ -
BTW the same person that has successfully started and supported OSM
Carto team building!).

It's always hard to set the right priorities how much would one spend on
communicating and finding new people versus doing actual work, but my
experience is that it pays off to risk some evangelizing - now we have a
team of active people in OSM Carto again, but it required putting some
effort into making the style more available without clear outcome what
exactly might work. And we still keep trying - see this fresh announcement:

https://www.openstreetmap.org/user/Tomasz_W/diary/44420

--
"My method is uncertain/ It's a mess but it's working" [F. Apple]



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Maarten Deen
In reply to this post by Tom Hughes-3
On 2018-07-18 15:50, Tom Hughes wrote:

> On 17/07/18 10:42, Maarten Deen wrote:
>
>> Is it just me or just today or is the API a lot slower after the move
>> on sunday? I'm downloading a number of busrelations and it takes a lot
>> longer than before the weekend.
>
> I know the cause of this has been explained already but I'd be
> interested to know exactly which API call you are referring to
> here as it may be that (long term at least) there is something
> here which should be fixed but it rather depends if it is a call
> being handled by rails or one being handled by cgimap.

Nothing special really, I was just downloading the data from a lot of
relations sequentially in JOSM. I downloaded relation 1360154 with its
members which are bus master_relations that then also load the bus
relations but without their contents, and then I selected all the bus
relations and downloaded the members (in JOSM: select all relations and
do "download members").

 From the Java console:
2018-07-18 19:13:55.553 INFO: GET
https://api.openstreetmap.org/api/0.6/relation/7449166/full -> 200
and that for about 320 relations.

Regards,
Maarten

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Tom Hughes-3
On 18/07/18 18:15, Maarten Deen wrote:

> Nothing special really, I was just downloading the data from a lot of
> relations sequentially in JOSM. I downloaded relation 1360154 with its
> members which are bus master_relations that then also load the bus
> relations but without their contents, and then I selected all the bus
> relations and downloaded the members (in JOSM: select all relations and
> do "download members").
>
>  From the Java console:
> 2018-07-18 19:13:55.553 INFO: GET
> https://api.openstreetmap.org/api/0.6/relation/7449166/full -> 200
> and that for about 320 relations.

Thanks. That's a call handled by cgimap so this is likely related
to the fact that there is currently increased latency between the
application server and the database for those calls.

There is a pending pull request for cgimap that may help.

Tom

--
Tom Hughes ([hidden email])
http://compton.nu/

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Andrew Hain
In reply to this post by Grant Slater
Will there be a local team or does whatever can’t be done remotely need a visit?

--
Andrew
From: Grant Slater <[hidden email]>
Sent: 17 July 2018 13:55:05
To: Daniel Koć
Cc: Talk Openstreetmap
Subject: Re: [OSM-talk] API a lot slower?
 
On 17 July 2018 at 13:42, Daniel Koć <daniel@koć.pl> wrote:
> W dniu 17.07.2018 o 12:22, Grant Slater pisze:
>> Our primary hardware is moving to a new data centre next week, and
>> will take some time to get up and running.
>
> What data center do you mean? Is it the one which OSMF was looking for
> at the beginning of the year?
>
> https://blog.openstreetmap.org/2018/02/19/osmf-request-for-proposals-data-centre-2018/
>
> Could you give some more details about it?
>

Yes. We are moving some core servers to an Equinix data centre in Amsterdam, NL.

OpenStreetMap's Brexit ;-) </sarcasm>

Kind regards,

Grant
Part of the OSM Operations team.

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Paul Norman
On 2018-07-18 11:04 AM, Andrew Hain wrote:
> Will there be a local team or does whatever can’t be done remotely
> need a visit?

For the install there will be two people traveling with the equipment,
and I've reached out to some locals that were recommended. We should
have enough people to get it done in the two days scheduled, but more
hands are welcome.

If you're a local with the technical background to help install servers
or experience moving equipment and interested and available on Wed July
25th or Thurs July 26, please let me know off-list. The data centre is
Equinix AM6 in De Omval, Oost, Amsterdam:
https://www.openstreetmap.org/way/460515888

Long-term, the remote hands are capable of performing most tasks,
including installing new servers. Local help would still be useful, so
if you're interested, also let me know off-list.

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Roland Olbricht
In reply to this post by Maarten Deen
Hi Maarten,

> Nothing special really, I was just downloading the data from a lot of
> relations sequentially in JOSM. I downloaded relation 1360154 with its
> members which are bus master_relations that then also load the bus
> relations but without their contents, and then I selected all the bus
> relations and downloaded the members (in JOSM: select all relations and
> do "download members").

By the way: Please consider to use "Download data > from Overpass"
with the query

( rel(1360154); >>; );
out meta;

The runtime is expected to be 3 to 10 seconds, data is usually less than two minutes off. As a side effect, this takes load off the Main API.

Regards,
Roland


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: API a lot slower?

Maarten Deen
On 2018-07-19 09:04, Roland Olbricht wrote:

> Hi Maarten,
>
>> Nothing special really, I was just downloading the data from a lot of
>> relations sequentially in JOSM. I downloaded relation 1360154 with its
>> members which are bus master_relations that then also load the bus
>> relations but without their contents, and then I selected all the bus
>> relations and downloaded the members (in JOSM: select all relations
>> and
>> do "download members").
>
> By the way: Please consider to use "Download data > from Overpass"
> with the query
>
> ( rel(1360154); >>; );
> out meta;
>
> The runtime is expected to be 3 to 10 seconds, data is usually less
> than two minutes off. As a side effect, this takes load off the Main
> API.

Thanks, that is a good tip. I am in a habit of doing that when
downloading nodes or ways with a certain k/v combination in an area, but
haven't yet done it with relations in that way.

Regards,
Maarten

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk