Infinite loop splitter

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Infinite loop splitter

ligfietser
Hi,
I was trying to update my Benelux map but the splitter refused to finish the process.
Normally, splitter is finished in 2 minutes, now it gets in an infinite loop with Full GC.
What can be the cause for this, a corrupt download of the europe extract?


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

splitter_error.log (114K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Bernd Weigelt
Am Samstag, 8. März 2014, 10:44:16 schrieb Minko:
> I was trying to update my Benelux map but the splitter refused to finish the
> process. Normally, splitter is finished in 2 minutes, now it gets in an
> infinite loop with Full GC. What can be the cause for this, a corrupt
> download of the europe extract?


I have seen this error last week, too.

I had to recreate my dach extract.

Bernd


--
amarok2 now playing:




_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
What have you done to solve this Bernd, should I just wait for another extract tomorrow?

> I have seen this error last week, too.
>
> I had to recreate my dach extract.
>
> Bernd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
Hi Minko,

please keep the files that produce this problem.
I'd like to have them to be able to reproduce it.

Gerd

> Date: Sat, 8 Mar 2014 11:09:03 +0100

> From: [hidden email]
> To: [hidden email]
> Subject: Re: [mkgmap-dev] Infinite loop splitter
>
> What have you done to solve this Bernd, should I just wait for another extract tomorrow?
>
> > I have seen this error last week, too.
> >
> > I had to recreate my dach extract.
> >
> > Bernd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
Hi Gerd,
I always mix the osm data with address nodes for Vlaanderen and the Netherlands with osmconvert.
This part of my batch file looks like this:

:osmconvert
%DIR%\Tools\osmconvert --drop-version %DIR%\%date%\Splitter\%extract% %DIR%\resources\adres.o5m %DIR%\resources\vlaanderen-adres.o5m -B=%DIR%\resources\bnl_01.poly -o=%DIR%\%date%\Splitter\benelux.o5m

%extract% is europe.osm.pbf from today

NL adres.o5m:
lon min: 3.3590433
lon max: 7.2259690
lat min: 50.7510383
lat max: 53.4972667
nodes: 8551053
ways: 0
relations: 0
node id min: 100000000000
node id max: 100008551052
keyval pairs max: 2
keyval pairs max object: node 100000000000

Vlaanderen vlaanderen-adres.o5m

lon min: 2.3066895
lon max: 5.8980601
lat min: 49.2933351
lat max: 51.5047683
nodes: 2449726
ways: 0
relations: 0
node id min: 200000000001
node id max: 200002449726
keyval pairs max: 4
keyval pairs max object: node 200000000001


After this I split the resulting file as follows

:Splitter
java -Xmx8000m -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -jar  %SPLITTER% --output-dir=%date%\Splitter --max-threads=8 --problem-file=%DIR%\resources\problem_areas.txt --precomp-sea=%seafile% --keep-complete --overlap=0 --mapid=10010001 --polygon-file=%DIR%\resources\benelux.poly --max-nodes=1750000 --write-kml=%areas%.kml --output=o5m --geonames-file=%DIR%\resources\cities5000.txt --description=OFM_BNL %DIR%\%date%\Splitter\benelux.o5m > splitter2.log 2>err2.log

Normally it all goes well until this morning.
I've tried the latest splitter version too but it didnt help.
I also tried to make the first polygon bnl_01.poly much bigger ( it is a little bigger than the second benelux.poly in the splitter step 2). This also didnt help.

Now if I skip the first step and split it without merging the address nodes it goes well without problems. Could it be that the splitter creates fake ID's that overlap my address data nodes?
Or is there an overlap in osm id's with the fake address nodes?

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
In reply to this post by ligfietser
Hi Minko,
ligfietser wrote
Now if I skip the first step and split it without merging the address nodes it goes well without problems. Could it be that the splitter creates fake ID's that overlap my address data nodes?
Or is there an overlap in osm id's with the fake address nodes?
I have no idea. Download of Europe will take a while...

Gerd
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Colin Smale
In reply to this post by ligfietser
I see this very often as well, when I use osmupdate on an earlier download. The only solution I have found is to download a new pbf, then it works again. I suspect therefore that osmupdate or one of the underlying tools is corrupting the data.

Colin

On 8 March 2014 10:44:16 CET, Minko <[hidden email]> wrote:
Hi,
I was trying to update my Benelux map but the splitter refused to finish the process.
Normally, splitter is finished in 2 minutes, now it gets in an infinite loop with Full GC.
What can be the cause for this, a corrupt download of the europe extract?



mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
Colin,

thanks for the hint.
No matter what other tools do: infinite loop in splitter is the worst case
as a reaction on errors ;-)

Gerd

Colin Smale wrote
I see this very often as well, when I use osmupdate on an earlier download. The only solution I have found is to download a new pbf, then it works again. I suspect therefore that osmupdate or one of the underlying tools is corrupting the data.

Colin

On 8 March 2014 10:44:16 CET, Minko <[hidden email]> wrote:
>Hi,
>I was trying to update my Benelux map but the splitter refused to
>finish the process.
>Normally, splitter is finished in 2 minutes, now it gets in an infinite
>loop with Full GC.
>What can be the cause for this, a corrupt download of the europe
>extract?
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>mkgmap-dev mailing list
>[hidden email]
>http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
Gerd,
If I skip merging the Vlaanderen address data then splitting goes well.
I have uploaded my 'corrupt' Benelux data which goes wrong (incl the Vlaanderen addresses) here:
http://mijndev.openstreetmap.nl/~ligfietser/test/benelux.o5m


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

osm-8
In reply to this post by ligfietser
Hi Minko,
is there a reason for merging the files before splitting? Did you try to
just pass all the files to splitter? I'm doing so with osm and
srtm-data. Don't know, if this helps in this case, but in general I
think it should be much faster.

Henning

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
Hi Henning,
I didnt know that you could feed multiple files to the splitter?
How do you do this?

> Hi Minko,
> is there a reason for merging the files before splitting? Did you try
> to
> just pass all the files to splitter? I'm doing so with osm and
> srtm-data. Don't know, if this helps in this case, but in general I
> think it should be much faster.
>
> Henning
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
In reply to this post by ligfietser
Hi Minko,

thanks, I can reproduce the problem with this file.
Looking for a fix now.

Gerd

ligfietser wrote
Gerd,
If I skip merging the Vlaanderen address data then splitting goes well.
I have uploaded my 'corrupt' Benelux data which goes wrong (incl the Vlaanderen addresses) here:
http://mijndev.openstreetmap.nl/~ligfietser/test/benelux.o5m


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
In reply to this post by ligfietser
Hi Minko,

I've committed r320 to fix the loop problem.
 
You can specify multiplie input files, but you have to make
sure that each file contains ids which are higher than all
previous ones. I think this is the case with your address files.

The reason for the loop was a way in a mp-relation
for which no nodes were found.
I guess this is caused by using osmconvert without the parameters
--complete-ways --complex-ways

I am not sure what's faster: Using osmconvert with polygon first or
only splitter with multiple input files. When all files are in o5m format,
single run with splitter should be quite fast.

Gerd
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
Hi Gerd,
Thanks for the fix, I will test it out.
I tried Hennings approach but splitting seemed to last much longer because it analyses the whole EU extract and I only need the Benelux plus a small area of France and Germany.

Splitting wasn't completed and threw out this error:

warning: --keep-complete is only used for the first input file.
New way id 195314 is not higher than last id 264955883
Maybe the IDs are not sorted. This is not supported with keep-complete=true or --problem-list

I'll try now  r320.


> I've committed r320 to fix the loop problem.
>
> You can specify multiplie input files, but you have to make
> sure that each file contains ids which are higher than all
> previous ones. I think this is the case with your address files.
>
> The reason for the loop was a way in a mp-relation
> for which no nodes were found.
> I guess this is caused by using osmconvert without the parameters
> --complete-ways --complex-ways
>
> I am not sure what's faster: Using osmconvert with polygon first or
> only splitter with multiple input files. When all files are in o5m
> format,
> single run with splitter should be quite fast.
>
> Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser

Gerd wrote:
> > The reason for the loop was a way in a mp-relation
> > for which no nodes were found.
> > I guess this is caused by using osmconvert without the parameters
> > --complete-ways --complex-ways


If I use --complete-ways or --complex-ways with osmconvert, I didnt get any output at all so I always skip this.
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
In reply to this post by ligfietser
Hi Minko,

the order of input files would be
java ... splitter.jar [options] europe.o5m  adres.o5m vlaanderen-adres.o5m

Your approach with osmconvert is probably faster because you have to use
osmconvert anyway to convert to o5m.

Gerd

ligfietser wrote
Hi Gerd,
Thanks for the fix, I will test it out.
I tried Hennings approach but splitting seemed to last much longer because it analyses the whole EU extract and I only need the Benelux plus a small area of France and Germany.

Splitting wasn't completed and threw out this error:

warning: --keep-complete is only used for the first input file.
New way id 195314 is not higher than last id 264955883
Maybe the IDs are not sorted. This is not supported with keep-complete=true or --problem-list

I'll try now  r320.


> I've committed r320 to fix the loop problem.
>
> You can specify multiplie input files, but you have to make
> sure that each file contains ids which are higher than all
> previous ones. I think this is the case with your address files.
>
> The reason for the loop was a way in a mp-relation
> for which no nodes were found.
> I guess this is caused by using osmconvert without the parameters
> --complete-ways --complex-ways
>
> I am not sure what's faster: Using osmconvert with polygon first or
> only splitter with multiple input files. When all files are in o5m
> format,
> single run with splitter should be quite fast.
>
> Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

ligfietser
Gerd wrote:
> the order of input files would be
> java ... splitter.jar [options] europe.o5m adres.o5m
> vlaanderen-adres.o5m

Yes I did that too.
Anyway, with r320 it seems to work, mkgmap is running now thanks!

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
Hi Minko,

fine:-)

I think the only good reason to use splitter with an input file like
europe.o5m or planet.o5m in combination with a poly file
that extracts a small part is that it is able to keep ferry lines complete.

osmconvert --complete-ways --complex-ways -B=bnl_01.poly europe.o5m -o=benelux.o5m
fails for me as well. I'll ask the author about it.

Gerd

ligfietser wrote
Gerd wrote:
> the order of input files would be
> java ... splitter.jar [options] europe.o5m adres.o5m
> vlaanderen-adres.o5m

Yes I did that too.
Anyway, with r320 it seems to work, mkgmap is running now thanks!

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop splitter

Gerd Petermann
Hi all,

I've contacted Markus Weber, the author of osmconvert and osmfilter.
It seems that the problem with complex-ways occurs only
in the Windows version, maybe because something goes wrong with
files > 4GB. Markus doesn't have access to a Win64 system.
I use a mingw-64 installation to compile my own
binary for 64 bit, maybe an update will help, but I doubt it,
and unfortunately I forgot how the installation works,
I just remember it wasn't simple.

So, if you use osmconvert/osmfilter to update your files
it is probably better to use a 64 bit Linux installation.

Gerd


GerdP wrote
Hi Minko,

fine:-)

I think the only good reason to use splitter with an input file like
europe.o5m or planet.o5m in combination with a poly file
that extracts a small part is that it is able to keep ferry lines complete.

osmconvert --complete-ways --complex-ways -B=bnl_01.poly europe.o5m -o=benelux.o5m
fails for me as well. I'll ask the author about it.

Gerd

ligfietser wrote
Gerd wrote:
> the order of input files would be
> java ... splitter.jar [options] europe.o5m adres.o5m
> vlaanderen-adres.o5m

Yes I did that too.
Anyway, with r320 it seems to work, mkgmap is running now thanks!

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
12