Serious JOSM performance degradation

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Serious JOSM performance degradation

Safwat Halaby
I deal with a huge dataset (about 30k bus stops nodes) almost daily.
This was never a problem before, but since recently (I cannot pinpoint
the exact version), JOSM often hangs when I edit the bus stop tags.
Waiting for a few seconds (can be up to ~30) always resolves this.

I suspect it has to do with JOSM's autocomplete, because it seems to
happen when editing massively used keys. For instance, all stops have a
"ref" key, and when I click "edit" on a single stop's "ref", JOSM
stalls. A similar issue occurs when adding "description" to changeset
uploads. Description is also used by all stops.

Any tips or pointers?

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
On Sat, 2017-11-11 at 22:43 +0200, Safwat Halaby wrote:

> I deal with a huge dataset (about 30k bus stops nodes) almost daily.
> This was never a problem before, but since recently (I cannot
> pinpoint
> the exact version), JOSM often hangs when I edit the bus stop tags.
> Waiting for a few seconds (can be up to ~30) always resolves this.
>
> I suspect it has to do with JOSM's autocomplete, because it seems to
> happen when editing massively used keys. For instance, all stops have
> a
> "ref" key, and when I click "edit" on a single stop's "ref", JOSM
> stalls. A similar issue occurs when adding "description" to changeset
> uploads. Description is also used by all stops.
>
> Any tips or pointers?
>
> _______________________________________________
> talk mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/talk

I will soon test this out with various versions.

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

john whelan-2
In reply to this post by Safwat Halaby
java -Xmx8210000000 -jar G:\josm\josm-tested.jar 

in a .bat file seems to be happy with 2Mb files.  It can be sluggish though so I try to keep it below 250kb for best performance.

Cheerio John

On 11 November 2017 at 15:43, Safwat Halaby <[hidden email]> wrote:
I deal with a huge dataset (about 30k bus stops nodes) almost daily.
This was never a problem before, but since recently (I cannot pinpoint
the exact version), JOSM often hangs when I edit the bus stop tags.
Waiting for a few seconds (can be up to ~30) always resolves this.

I suspect it has to do with JOSM's autocomplete, because it seems to
happen when editing massively used keys. For instance, all stops have a
"ref" key, and when I click "edit" on a single stop's "ref", JOSM
stalls. A similar issue occurs when adding "description" to changeset
uploads. Description is also used by all stops.

Any tips or pointers?

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

john whelan-2
If you give it more memory sometimes that compensates on the CPU side by not requiring so much disk reading and writing.  Also there are almost certainly internal tables which work better in memory.  JAVA is a strange world of its own and with so many contributors and plugins I'm not sure anyone has a clear idea of exactly how JOSM works with in it.

Not guaranteed but worth a try.

Cheerio John

On 11 November 2017 at 16:03, Safwat Halaby <[hidden email]> wrote:
More details:

JOSM 12712 has no issues.
Latest JOSM 13053 has issues.
Not sure how to download the intermediate version (12921).
It's a CPU starvation issue and not a memory overuse issue. (Core at
100% for 10-30 seconds).

Test case:

Open up the attached file, try editing the ref or name of a stop.

12712 works flawlessly.
13053 hangs for 10-30 seconds.


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Jo-2
Did you start using PT_Assistant? Or a MapCSS style dedicated to PT?

Polyglot

2017-11-11 22:10 GMT+01:00 john whelan <[hidden email]>:
If you give it more memory sometimes that compensates on the CPU side by not requiring so much disk reading and writing.  Also there are almost certainly internal tables which work better in memory.  JAVA is a strange world of its own and with so many contributors and plugins I'm not sure anyone has a clear idea of exactly how JOSM works with in it.

Not guaranteed but worth a try.

Cheerio John

On 11 November 2017 at 16:03, Safwat Halaby <[hidden email]> wrote:
More details:

JOSM 12712 has no issues.
Latest JOSM 13053 has issues.
Not sure how to download the intermediate version (12921).
It's a CPU starvation issue and not a memory overuse issue. (Core at
100% for 10-30 seconds).

Test case:

Open up the attached file, try editing the ref or name of a stop.

12712 works flawlessly.
13053 hangs for 10-30 seconds.


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
I don't know what these are.

The test described below was done without plugins, by the way.

On Sat, 2017-11-11 at 22:14 +0100, Jo wrote:

> Did you start using PT_Assistant? Or a MapCSS style dedicated to PT?
>
> Polyglot
>
> 2017-11-11 22:10 GMT+01:00 john whelan <[hidden email]>:
>
> > If you give it more memory sometimes that compensates on the CPU
> > side by
> > not requiring so much disk reading and writing.  Also there are
> > almost
> > certainly internal tables which work better in memory.  JAVA is a
> > strange
> > world of its own and with so many contributors and plugins I'm not
> > sure
> > anyone has a clear idea of exactly how JOSM works with in it.
> >
> > Not guaranteed but worth a try.
> >
> > Cheerio John
> >
> > On 11 November 2017 at 16:03, Safwat Halaby <[hidden email]>
> > wrote:
> >
> > > More details:
> > >
> > > JOSM 12712 has no issues.
> > > Latest JOSM 13053 has issues.
> > > Not sure how to download the intermediate version (12921).
> > > It's a CPU starvation issue and not a memory overuse issue. (Core
> > > at
> > > 100% for 10-30 seconds).
> > >
> > > Test case:
> > >
> > > Open up the attached file, try editing the ref or name of a stop.
> > >
> > > 12712 works flawlessly.
> > > 13053 hangs for 10-30 seconds.
> >
> >
> >
> > _______________________________________________
> > talk mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/talk
> >
> >

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
In reply to this post by Safwat Halaby
Apparently one of my messages did not get through due to a huge file.
Here it is once more:

More details:

JOSM 12712 has no issues.
Latest JOSM 13053 has issues.
Not sure how to download the intermediate version (12921).
It's a CPU starvation issue and not a memory overuse issue. (Core at
100% for 10-30 seconds).

Test case:
Open up the attached file, try editing the ref or name of a stop.
12712 works flawlessly.
13053 hangs for 10-30 seconds.

original message ends here.

As for the file, I cannot attach it, but use this Overpass query to
obtain the same dataset:

[out:xml][timeout:90][bbox:29.4013195,33.8818359,33.4131022,36.0791016]
;
(
area(3601473946); // Israel
area(3601803010); // Judea and Samaria district
)->.a;
(
  node["highway"="bus_stop"](area.a);
  way["highway"="bus_stop"](area.a);
)->.b;
(rel(bw.b);rel(bn.b))->.routes;
(.b;way(bn.b);)->.b;
(.b;node(w.b);)->.b;
(.b;.routes;);
out meta;

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Jo-2
In reply to this post by Safwat Halaby
When working with public transport, the PT_Assistant plugin might be interesting for you. On the other hand, it might cause additional overhead.

Jo

2017-11-11 22:23 GMT+01:00 Safwat Halaby <[hidden email]>:
I don't know what these are.

The test described below was done without plugins, by the way.

On Sat, 2017-11-11 at 22:14 +0100, Jo wrote:
> Did you start using PT_Assistant? Or a MapCSS style dedicated to PT?
>
> Polyglot
>
> 2017-11-11 22:10 GMT+01:00 john whelan <[hidden email]>:
>
> > If you give it more memory sometimes that compensates on the CPU
> > side by
> > not requiring so much disk reading and writing.  Also there are
> > almost
> > certainly internal tables which work better in memory.  JAVA is a
> > strange
> > world of its own and with so many contributors and plugins I'm not
> > sure
> > anyone has a clear idea of exactly how JOSM works with in it.
> >
> > Not guaranteed but worth a try.
> >
> > Cheerio John
> >
> > On 11 November 2017 at 16:03, Safwat Halaby <[hidden email]>
> > wrote:
> >
> > > More details:
> > >
> > > JOSM 12712 has no issues.
> > > Latest JOSM 13053 has issues.
> > > Not sure how to download the intermediate version (12921).
> > > It's a CPU starvation issue and not a memory overuse issue. (Core
> > > at
> > > 100% for 10-30 seconds).
> > >
> > > Test case:
> > >
> > > Open up the attached file, try editing the ref or name of a stop.
> > >
> > > 12712 works flawlessly.
> > > 13053 hangs for 10-30 seconds.
> >
> >
> >
> > _______________________________________________
> > talk mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/talk
> >
> >


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
In reply to this post by john whelan-2
That's interestingly low. I'm talking about a 17MiB file here, and it
was never slugish before this bug. My laptop is an average thinkpad...

On Sat, 2017-11-11 at 15:52 -0500, john whelan wrote:

> java -Xmx8210000000 -jar G:\josm\josm-tested.jar
>
> in a .bat file seems to be happy with 2Mb files.  It can be sluggish
> though
> so I try to keep it below 250kb for best performance.
>
> Cheerio John
>
> On 11 November 2017 at 15:43, Safwat Halaby <[hidden email]>
> wrote:
>
> > I deal with a huge dataset (about 30k bus stops nodes) almost
> > daily.
> > This was never a problem before, but since recently (I cannot
> > pinpoint
> > the exact version), JOSM often hangs when I edit the bus stop tags.
> > Waiting for a few seconds (can be up to ~30) always resolves this.
> >
> > I suspect it has to do with JOSM's autocomplete, because it seems
> > to
> > happen when editing massively used keys. For instance, all stops
> > have a
> > "ref" key, and when I click "edit" on a single stop's "ref", JOSM
> > stalls. A similar issue occurs when adding "description" to
> > changeset
> > uploads. Description is also used by all stops.
> >
> > Any tips or pointers?
> >
> > _______________________________________________
> > talk mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/talk
> >

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

john whelan-2
>17MiB

Interesting file size.  What is a MiB?

If you are working with large file sizes and using the results elsewhere you should be using ECC memory.  Anything else and you stand a chance of corruption.

How much memory does your laptop have?

Thanks John

On 11 Nov 2017 4:48 pm, "Safwat Halaby" <[hidden email]> wrote:
That's interestingly low. I'm talking about a 17MiB file here, and it
was never slugish before this bug. My laptop is an average thinkpad...

On Sat, 2017-11-11 at 15:52 -0500, john whelan wrote:
> java -Xmx8210000000 -jar G:\josm\josm-tested.jar
>
> in a .bat file seems to be happy with 2Mb files.  It can be sluggish
> though
> so I try to keep it below 250kb for best performance.
>
> Cheerio John
>
> On 11 November 2017 at 15:43, Safwat Halaby <[hidden email]>
> wrote:
>
> > I deal with a huge dataset (about 30k bus stops nodes) almost
> > daily.
> > This was never a problem before, but since recently (I cannot
> > pinpoint
> > the exact version), JOSM often hangs when I edit the bus stop tags.
> > Waiting for a few seconds (can be up to ~30) always resolves this.
> >
> > I suspect it has to do with JOSM's autocomplete, because it seems
> > to
> > happen when editing massively used keys. For instance, all stops
> > have a
> > "ref" key, and when I click "edit" on a single stop's "ref", JOSM
> > stalls. A similar issue occurs when adding "description" to
> > changeset
> > uploads. Description is also used by all stops.
> >
> > Any tips or pointers?
> >
> > _______________________________________________
> > talk mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/talk
> >

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
mmd
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

mmd
In reply to this post by Safwat Halaby
Am 11.11.2017 um 22:25 schrieb Safwat Halaby:

> Apparently one of my messages did not get through due to a huge file.
> Here it is once more:
>
> More details:
>
> JOSM 12712 has no issues.
> Latest JOSM 13053 has issues.
> Not sure how to download the intermediate version (12921).
> It's a CPU starvation issue and not a memory overuse issue. (Core at
> 100% for 10-30 seconds).
>
> Test case:
> Open up the attached file, try editing the ref or name of a stop.
> 12712 works flawlessly.
> 13053 hangs for 10-30 seconds.
>

I can reproduce this locally on JOSM 13106. According to jvisualvm cpu
profiling (on Oracle JDK), most time is spent in method
org.openstreetmap.josm.data.tagging.ac.AutoCompletionSet.add.

Best bet would be to create an issue on
https://josm.openstreetmap.de/newticket


--


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Daniel Koć
In reply to this post by john whelan-2
W dniu 11.11.2017 o 22:55, john whelan pisze:
>17MiB

Interesting file size.  What is a MiB?

https://en.wikipedia.org/wiki/Mebibyte

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Jan Martinec
In reply to this post by john whelan-2
You can download somewhat-recent JOSM versions from https://josm.openstreetmap.de/download/ , I do see 12921 there. Also josm-latest, which is the testing version (currently 13101).

Cheers,
Honza "Piškvor" Martinec 

Dne 11. 11. 2017 22:15 napsal uživatel "john whelan" <[hidden email]>:
If you give it more memory sometimes that compensates on the CPU side by not requiring so much disk reading and writing.  Also there are almost certainly internal tables which work better in memory.  JAVA is a strange world of its own and with so many contributors and plugins I'm not sure anyone has a clear idea of exactly how JOSM works with in it.

Not guaranteed but worth a try.

Cheerio John

On 11 November 2017 at 16:03, Safwat Halaby <[hidden email]> wrote:
More details:

JOSM 12712 has no issues.
Latest JOSM 13053 has issues.
Not sure how to download the intermediate version (12921).
It's a CPU starvation issue and not a memory overuse issue. (Core at
100% for 10-30 seconds).

Test case:

Open up the attached file, try editing the ref or name of a stop.

12712 works flawlessly.
13053 hangs for 10-30 seconds.


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

James-2
have you tried with wireframe mode(ctrl+w)?

On Nov 11, 2017 5:11 PM, "Jan Martinec" <[hidden email]> wrote:
You can download somewhat-recent JOSM versions from https://josm.openstreetmap.de/download/ , I do see 12921 there. Also josm-latest, which is the testing version (currently 13101).

Cheers,
Honza "Piškvor" Martinec 

Dne 11. 11. 2017 22:15 napsal uživatel "john whelan" <[hidden email]>:
If you give it more memory sometimes that compensates on the CPU side by not requiring so much disk reading and writing.  Also there are almost certainly internal tables which work better in memory.  JAVA is a strange world of its own and with so many contributors and plugins I'm not sure anyone has a clear idea of exactly how JOSM works with in it.

Not guaranteed but worth a try.

Cheerio John

On 11 November 2017 at 16:03, Safwat Halaby <[hidden email]> wrote:
More details:

JOSM 12712 has no issues.
Latest JOSM 13053 has issues.
Not sure how to download the intermediate version (12921).
It's a CPU starvation issue and not a memory overuse issue. (Core at
100% for 10-30 seconds).

Test case:

Open up the attached file, try editing the ref or name of a stop.

12712 works flawlessly.
13053 hangs for 10-30 seconds.


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

john whelan-2
In reply to this post by Daniel Koć
>but it is not widely acknowledged within the industry or media.

That probably explains why in fifty years of working with computers I hadn't come across the term.

Cheerio John

On 11 Nov 2017 5:03 pm, "Daniel Koć" <[hidden email]> wrote:
W dniu 11.11.2017 o 22:55, john whelan pisze:
>17MiB

Interesting file size.  What is a MiB?

https://en.wikipedia.org/wiki/Mebibyte

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
In reply to this post by Safwat Halaby
On Sat, 2017-11-11 at 22:57 +0100, mmd wrote:
> I can reproduce this locally on JOSM 13106. According to
> cpu jvisualvm
> profiling (on Oracle JDK), most time is spent in method
> org.openstreetmap.josm.data.tagging.ac.AutoCompletionSet.add.

> Best bet would be to create an issue on
> https://josm.openstreetmap.de/newticket

Thank you! So it's autocomplete as I suspected. I will file a ticket.
Is there a way to disable autocomplete in the meantime?

On Sat, 2017-11-11 at 22:27 +0100, Jo wrote:
> When working with public transport, the PT_Assistant plugin might be
> interesting for you. On the other hand, it might cause additional
> overhead.

Seems useful!

On Sat, 2017-11-11 at 23:09 +0100, Jan Martinec wrote:
> You can download somewhat-recent JOSM versions from https://josm.
> openstreetmap.de/download/ , I do see 12921 there. Also josm-latest,
> which
> is the testing version (currently 13101).

Thank you.

> Interesting file size.  What is a MiB?

> If you are working with large file sizes and using the results
> elsewhere
> you should be using ECC memory.  Anything else and you stand a chance
> of
> corruption.

> How much memory does your laptop have?

MiB is practically the same as MB in for the needs of this
conversation.

MB can be  vague nowadays, sometimes people use it when they mean
1000^2, and sometimes it means 1024^2.  MiB is explicitly 1024^2.
(lowercase mb is even more vague as it could also mean Mega-bit, which
is 256 bytes).

I am not working with astronomically huge memory, just a mundane
Overpass fetch-edit*-upload cycle. But perhaps it's all been smooth
because the dataset is entirely nodes, and no ways or relations. The
laptop is 4 GiB. Pretty average hardware.

*The edit is actually several custom GTFS scripts followed by some
manual touches but that's irrelevant for the performance issue. For
anyone interested in what I do with 30k stops:
https://wiki.openstreetmap.org/wiki/User:SafwatHalaby/scripts/gtfs



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Jo-2
In reply to this post by john whelan-2
It has something to do with the difference between 10**3 and 2**10

2017-11-11 23:21 GMT+01:00 john whelan <[hidden email]>:
>but it is not widely acknowledged within the industry or media.

That probably explains why in fifty years of working with computers I hadn't come across the term.

Cheerio John

On 11 Nov 2017 5:03 pm, "Daniel Koć" <[hidden email]> wrote:
W dniu 11.11.2017 o 22:55, john whelan pisze:
>17MiB

Interesting file size.  What is a MiB?

https://en.wikipedia.org/wiki/Mebibyte

-- 
"My method is uncertain/ It's a mess but it's working" [F. Apple]

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
In reply to this post by Safwat Halaby

> MB can be  vague nowadays, sometimes people use it when they mean
> 1000^2, and sometimes it means 1024^2.  MiB is explicitly 1024^2.
> (lowercase mb is even more vague as it could also mean Mega-bit,
> which
> is 256 bytes).
>

Sorry, 131072 bytes.

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Safwat Halaby
In reply to this post by mmd
Ticket: https://josm.openstreetmap.de/ticket/15547

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Serious JOSM performance degradation

Bob Hawkins
In reply to this post by Safwat Halaby
I posted a topic on this matter in OpenStreetMap Forum>Editors on the very same day as this thread was started, by coincidence, and directed to this mailing list by SomeoneElse.  I received helpful replies and believe I have succeeded in overcoming the slow responses we were experiencing as a result.  My reply is here: https://forum.openstreetmap.org/viewtopic.php?id=60403.  I should be interested to learn if it helps others. 

Virus-free. www.avast.com

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
12