Quantcast

mkgmap-r3906 (optimize-index)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

mkgmap-r3906 (optimize-index)

Gerd Petermann
Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case,  --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
 
If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Gerd Petermann
Hi all,

as a follow up:
In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
r3906 failed with OutOfMemoryError even with -Xmx5000 .
So I think memory is no longer a problem unless you want to create an index  for planet ;-)
No other changes, means r3908 can be used to create index for *.img files created with r3906.

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Montag, 17. April 2017 20:05:15
An: [hidden email]
Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case,  --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.

If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Gerd Petermann
In reply to this post by Gerd Petermann
Hi Carlos,

thanks for testing. I am now looking again at the details for the prefix/suffix support.

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Carlos Dávila <[hidden email]>
Gesendet: Dienstag, 18. April 2017 23:29:37
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

El 17/04/17 a las 20:05, Gerd Petermann escribió:

> Hi all,
>
> with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
>
> I think address / road search works very well now, at least with west european languages.
> I tried various combinations of options like --latin1 / --unicode, --lower-case,  --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
>
> IMPORTANT:
> If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
>
> The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
> The created index is a bit smaller although it now also contains roads with an empty string as first label.
>
> If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
> keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
>
> If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
> I think it is a big step forward, but there may still be special cases with other languages.
>
> I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
>
> Gerd
I could not yet test it in deep, but first few searches seem to work
pretty well both on MapSource and nuvi
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Arndt Röhrig
In reply to this post by Gerd Petermann

Thank you for that great work!

Now my old nostalgia PC is able to create the index for Speiche_Europa!

My PC has only 6GB RAM.

-Xmx5300M -> fails with "java heap space error"

-Xmx10500M -> fails with "overflowed directory with max block".

- Xmx10500M and Option --block-size=65536 -> That works!

Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.

The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.

Special thanks to Gerd, who show me many things to improve my map building procedere!

Best regards

Arndt

speichenkarte.de



.


Gerd Petermann <[hidden email]> hat am 18. April 2017 um 16:41 geschrieben:

Hi all,

as a follow up:
In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
r3906 failed with OutOfMemoryError even with -Xmx5000 .
So I think memory is no longer a problem unless you want to create an index for planet ;-)
No other changes, means r3908 can be used to create index for *.img files created with r3906.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Montag, 17. April 2017 20:05:15
An: [hidden email]
Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.

If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Gerd Petermann
Hi Arndt,

thanks for the flowers ;-)
I am still surprised that your style requires so much more memory compared to the default style.
Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
Gesendet: Mittwoch, 19. April 2017 17:04:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Thank you for that great work!

Now my old nostalgia PC is able to create the index for Speiche_Europa!

My PC has only 6GB RAM.

-Xmx5300M -> fails with "java heap space error"

-Xmx10500M -> fails with "overflowed directory with max block".

- Xmx10500M and Option --block-size=65536 -> That works!

Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.

The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.

Special thanks to Gerd, who show me many things to improve my map building procedere!

Best regards

Arndt

speichenkarte.de



.


Gerd Petermann <[hidden email]> hat am 18. April 2017 um 16:41 geschrieben:

Hi all,

as a follow up:
In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
r3906 failed with OutOfMemoryError even with -Xmx5000 .
So I think memory is no longer a problem unless you want to create an index for planet ;-)
No other changes, means r3908 can be used to create index for *.img files created with r3906.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Montag, 17. April 2017 20:05:15
An: [hidden email]
Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.

If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Arndt Röhrig

Hi Gerd,

yes, i use 170419.

The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter.

Arndt


Gerd Petermann <[hidden email]> hat am 21. April 2017 um 07:34 geschrieben:

Hi Arndt,

thanks for the flowers ;-)
I am still surprised that your style requires so much more memory compared to the default style.
Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
Gesendet: Mittwoch, 19. April 2017 17:04:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Thank you for that great work!

Now my old nostalgia PC is able to create the index for Speiche_Europa!

My PC has only 6GB RAM.

-Xmx5300M -> fails with "java heap space error"

-Xmx10500M -> fails with "overflowed directory with max block".

  • Xmx10500M and Option --block-size=65536 -> That works!

Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.

The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.

Special thanks to Gerd, who show me many things to improve my map building procedere!

Best regards

Arndt

speichenkarte.de

.

Gerd Petermann <[hidden email]> hat am 18. April 2017 um 16:41 geschrieben:

Hi all,

as a follow up:
In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
r3906 failed with OutOfMemoryError even with -Xmx5000 .
So I think memory is no longer a problem unless you want to create an index for planet ;-)
No other changes, means r3908 can be used to create index for *.img files created with r3906.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Montag, 17. April 2017 20:05:15
An: [hidden email]
Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.

If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Gerd Petermann
Hi Arndt,

strange, the index itself  is ~ 1GB, that's very close to the size for the default style in r3908.
Did you use r3908 or an older version?

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
Gesendet: Freitag, 21. April 2017 10:28:25
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi Gerd,

yes, i use 170419.

The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter.

Arndt


Gerd Petermann <[hidden email]> hat am 21. April 2017 um 07:34 geschrieben:

Hi Arndt,

thanks for the flowers ;-)
I am still surprised that your style requires so much more memory compared to the default style.
Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
Gesendet: Mittwoch, 19. April 2017 17:04:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Thank you for that great work!

Now my old nostalgia PC is able to create the index for Speiche_Europa!

My PC has only 6GB RAM.

-Xmx5300M -> fails with "java heap space error"

-Xmx10500M -> fails with "overflowed directory with max block".

  *   Xmx10500M and Option --block-size=65536 -> That works!

Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.

The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.

Special thanks to Gerd, who show me many things to improve my map building procedere!

Best regards

Arndt

speichenkarte.de

.

Gerd Petermann <[hidden email]> hat am 18. April 2017 um 16:41 geschrieben:

Hi all,

as a follow up:
In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
r3906 failed with OutOfMemoryError even with -Xmx5000 .
So I think memory is no longer a problem unless you want to create an index for planet ;-)
No other changes, means r3908 can be used to create index for *.img files created with r3906.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Montag, 17. April 2017 20:05:15
An: [hidden email]
Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi all,

with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).

I think address / road search works very well now, at least with west european languages.
I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.

IMPORTANT:
If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.

The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
The created index is a bit smaller although it now also contains roads with an empty string as first label.

If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.

If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
I think it is a big step forward, but there may still be special cases with other languages.

I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index

Gerd

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Arndt Röhrig
Hi Gerd,
the map tiles from Speiche_Europa 190419 was build with mkgmap-r3906. The index with mkgmap-r3908. (I start it before 3908 was available)
Arndt

> Gerd Petermann <[hidden email]> hat am 21. April 2017 um 10:32 geschrieben:
>
>
> Hi Arndt,
>
> strange, the index itself is ~ 1GB, that's very close to the size for the default style in r3908.
> Did you use r3908 or an older version?
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
> Gesendet: Freitag, 21. April 2017 10:28:25
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Hi Gerd,
>
> yes, i use 170419.
>
> The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter.
>
> Arndt
>
>
> Gerd Petermann <[hidden email]> hat am 21. April 2017 um 07:34 geschrieben:
>
> Hi Arndt,
>
> thanks for the flowers ;-)
> I am still surprised that your style requires so much more memory compared to the default style.
> Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
> Gesendet: Mittwoch, 19. April 2017 17:04:43
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Thank you for that great work!
>
> Now my old nostalgia PC is able to create the index for Speiche_Europa!
>
> My PC has only 6GB RAM.
>
> -Xmx5300M -> fails with "java heap space error"
>
> -Xmx10500M -> fails with "overflowed directory with max block".
>
> * Xmx10500M and Option --block-size=65536 -> That works!
>
> Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.
>
> The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.
>
> Special thanks to Gerd, who show me many things to improve my map building procedere!
>
> Best regards
>
> Arndt
>
> speichenkarte.de
>
> .
>
> Gerd Petermann <[hidden email]> hat am 18. April 2017 um 16:41 geschrieben:
>
> Hi all,
>
> as a follow up:
> In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
> r3906 failed with OutOfMemoryError even with -Xmx5000 .
> So I think memory is no longer a problem unless you want to create an index for planet ;-)
> No other changes, means r3908 can be used to create index for *.img files created with r3906.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
> Gesendet: Montag, 17. April 2017 20:05:15
> An: [hidden email]
> Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Hi all,
>
> with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
>
> I think address / road search works very well now, at least with west european languages.
> I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
>
> IMPORTANT:
> If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
>
> The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
> The created index is a bit smaller although it now also contains roads with an empty string as first label.
>
> If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
> keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
>
> If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
> I think it is a big step forward, but there may still be special cases with other languages.
>
> I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
>
> Gerd
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: mkgmap-r3906 (optimize-index)

Gerd Petermann
Hi Arndt,

seems that -Xmx5300 was just a little too less, on my machine it worked fine  with -Xmx6000m and also crashed with -Xmx5300m.
I've downloaded your map and used MapReverseConverter
from http://www.javawa.nl/cne2009_multilang.html
to convert the .gmap format back to *.img files so that I could use mkgmap to re-calculate the index.
java -Xmx6000m -jar d:\mkgmap\dist\mkgmap_r3910.jar -c mkgmap_options_mdrmdx.args  --block-size=65536 ..\Speiche_Europa_170419\88*.img
(r3910 is exactly like r3908 for this purpose)

I am working on some improvements and corrections for the prefix/suffix support now, maybe this will require less memory, maybe a bit more.

Gerd
________________________________________
Von: Arndt Röhrig <[hidden email]>
Gesendet: Freitag, 21. April 2017 10:40:48
An: Development list for mkgmap; Gerd Petermann
Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)

Hi Gerd,
the map tiles from Speiche_Europa 190419 was build with mkgmap-r3906. The index with mkgmap-r3908. (I start it before 3908 was available)
Arndt

> Gerd Petermann <[hidden email]> hat am 21. April 2017 um 10:32 geschrieben:
>
>
> Hi Arndt,
>
> strange, the index itself is ~ 1GB, that's very close to the size for the default style in r3908.
> Did you use r3908 or an older version?
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
> Gesendet: Freitag, 21. April 2017 10:28:25
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Hi Gerd,
>
> yes, i use 170419.
>
> The OSM-Data is downloaded by "Geofabrik" (europe). To be added 5GB SRTM data. osm and srtm data merged with the splitter.
>
> Arndt
>
>
> Gerd Petermann <[hidden email]> hat am 21. April 2017 um 07:34 geschrieben:
>
> Hi Arndt,
>
> thanks for the flowers ;-)
> I am still surprised that your style requires so much more memory compared to the default style.
> Did you use the scripts in SpeicheFabrik_Steuerdateien170419.zip for this?
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von Arndt Röhrig <[hidden email]>
> Gesendet: Mittwoch, 19. April 2017 17:04:43
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Thank you for that great work!
>
> Now my old nostalgia PC is able to create the index for Speiche_Europa!
>
> My PC has only 6GB RAM.
>
> -Xmx5300M -> fails with "java heap space error"
>
> -Xmx10500M -> fails with "overflowed directory with max block".
>
> * Xmx10500M and Option --block-size=65536 -> That works!
>
> Its better to run mkgmap 2 times. First step builds the maptiles with Xmx3000M, so that java not use the harddisk to swap, cause that makes the PC very slow. The ovm-work files may not be deleted for the overview map.
>
> The second step with -Xmx10500M builds the index and the overview map. The taskmanager show maximal ~9GB in use. The Speiche_Europa map has ~20GB.
>
> Special thanks to Gerd, who show me many things to improve my map building procedere!
>
> Best regards
>
> Arndt
>
> speichenkarte.de
>
> .
>
> Gerd Petermann <[hidden email]> hat am 18. April 2017 um 16:41 geschrieben:
>
> Hi all,
>
> as a follow up:
> In r3907 and r3908 I have coded the merge sort for roads and pois. Now -Xmx4000 was easily enough to create the index for Europe.
> r3906 failed with OutOfMemoryError even with -Xmx5000 .
> So I think memory is no longer a problem unless you want to create an index for planet ;-)
> No other changes, means r3908 can be used to create index for *.img files created with r3906.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
> Gesendet: Montag, 17. April 2017 20:05:15
> An: [hidden email]
> Betreff: [mkgmap-dev] mkgmap-r3906 (optimize-index)
>
> Hi all,
>
> with the help of Steve I fixed some problems with the index, esp. sorting of road names with different speliing of Straße caused a lot of problems, like Ahornstraße (Germany) and Ahornstrasse (Switzerland).
>
> I think address / road search works very well now, at least with west european languages.
> I tried various combinations of options like --latin1 / --unicode, --lower-case, --x-split-name-index both in MapSource and on my Oregon 600 and always got what I expected.
>
> IMPORTANT:
> If you try this version, please make sure that you also compile the img files with this version so that the changes in the sort are used everywhere.
>
> The branch also reduces peak memory compared to trunk and because of that it is faster when creating large indexes, but speed is probably not so important here.
> The created index is a bit smaller although it now also contains roads with an empty string as first label.
>
> If memory is still an issue for you when compiling the index for large maps I can try to implement a merge sort which would only create the - heap consuming - sort
> keys for a rather small number (e.g. 100.000) roads and sort those and finally merge the parts.
>
> If you know special cases which don't work with r3890 trunk please try the branch and let me know if something might be improved.
> I think it is a big step forward, but there may still be special cases with other languages.
>
> I used a small set of only 4 tiles to test functionality and compiled index for Europe (compiled with default style) (>1600 tiles) with -Xmx6800m and --x-split-name-index
>
> Gerd
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Loading...