[Patch v3] reduce peak memory usage for global index

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[Patch v3] reduce peak memory usage for global index

Gerd Petermann
Hi all,

the attached patch is a further optimization. Peak memory for --x-split-name-index is now ~ 57% of r3873, without it ~47 % compared to r3873.
The run time is a bit better and the unit tests are passed again.
I've removed the debug reports for now, so that this patch is ready for commit.
A binary is here:
http://files.mkgmap.org.uk/download/343/mkgmap.jar

Worth to mention:
The --x-split-name-index option requires a lot more memory in my test sample with only 51 tiles I see 250 MB instead of 101MB (437 instead of 214 with r3873),
so the relative increase for the option is even higher with the patch.

A lot of the memory is required to store so called sort keys. Those are created from the street name(s). A possible alternative would be
to calculate the keys only only when required. This would increase run time for sorting but should keep memory much lower.
I'll experiment with this later.

Gerd





_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

mdr-sort-v3.patch (25K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [Patch v3] reduce peak memory usage for global index

Gerd Petermann
Hi all,

something goes wrong with this patch. Please ignore it.

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Freitag, 31. März 2017 12:32:46
An: [hidden email]
Betreff: [mkgmap-dev] [Patch v3] reduce peak memory usage for global index

Hi all,

the attached patch is a further optimization. Peak memory for --x-split-name-index is now ~ 57% of r3873, without it ~47 % compared to r3873.
The run time is a bit better and the unit tests are passed again.
I've removed the debug reports for now, so that this patch is ready for commit.
A binary is here:
http://files.mkgmap.org.uk/download/343/mkgmap.jar

Worth to mention:
The --x-split-name-index option requires a lot more memory in my test sample with only 51 tiles I see 250 MB instead of 101MB (437 instead of 214 with r3873),
so the relative increase for the option is even higher with the patch.

A lot of the memory is required to store so called sort keys. Those are created from the street name(s). A possible alternative would be
to calculate the keys only only when required. This would increase run time for sorting but should keep memory much lower.
I'll experiment with this later.

Gerd




_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev