r4033: new dem options are now documented

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

r4033: new dem options are now documented

Gerd Petermann
Hi all,

see http://www.mkgmap.org.uk/doc/options
I've added them in a chapter headed "Hill Shading (DEM) options"
This also means that you no longer need the --x- prefix when using r4033 or newer.

Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
Maybe option --dem should be changed to --dem-source before merging to trunk?

I think these are the other open problems now (no particular order):
-- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
-- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
-- Problems with java heap when creating large or highly detailed maps:
Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
@Frank: I think BuildDEMFile also has this problem, right?
-- The code needs further clean up , javadoc and unit tests.

Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

ligfietser

Gerd, I don't know why but mkgmap >v4025 does not work for my maps anymore.

It is doing hours to compile one tile!


My (experimental) DEM settings are

dem-dists=3312,3312,3312,6628,9942
overview-dem-dist=16560

Map levels:
levels = 0:24, 1:23, 2:22, 3:20, 4:18
overview-levels = 5:17, 6:15, 7:14

v4025 compiles ok, I updated my OFM Alps, Benelux and Germany recently.


Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 03:09:39
Aan: [hidden email]
Onderwerp: [mkgmap-dev] r4033: new dem options are now documented
 
Hi all,

see http://www.mkgmap.org.uk/doc/options
I've added them in a chapter headed "Hill Shading (DEM) options"
This also means that you no longer need the --x- prefix when using r4033 or newer.

Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
Maybe option --dem should be changed to --dem-source before merging to trunk?

I think these are the other open problems now (no particular order):
-- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
-- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
-- Problems with java heap when creating large or highly detailed maps:
Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
@Frank: I think BuildDEMFile also has this problem, right?
-- The code needs further clean up , javadoc and unit tests.

Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
Hi Minko,

okay, your options look strange, but I cannot reproduce a run time problem here.
Do you see this for all tiles? Maybe you have a tile with special boundaries. Please post the input file that takes so long.
What is your value for JRE option -Xmx ?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 14:30:04
An: [hidden email]
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Gerd, I don't know why but mkgmap >v4025 does not work for my maps anymore.

It is doing hours to compile one tile!


My (experimental) DEM settings are

dem-dists=3312,3312,3312,6628,9942
overview-dem-dist=16560

Map levels:
levels = 0:24, 1:23, 2:22, 3:20, 4:18
overview-levels = 5:17, 6:15, 7:14


v4025 compiles ok, I updated my OFM Alps, Benelux and Germany recently.

________________________________
Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 03:09:39
Aan: [hidden email]
Onderwerp: [mkgmap-dev] r4033: new dem options are now documented

Hi all,

see http://www.mkgmap.org.uk/doc/options
I've added them in a chapter headed "Hill Shading (DEM) options"
This also means that you no longer need the --x- prefix when using r4033 or newer.

Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
Maybe option --dem should be changed to --dem-source before merging to trunk?

I think these are the other open problems now (no particular order):
-- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
-- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
-- Problems with java heap when creating large or highly detailed maps:
Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
@Frank: I think BuildDEMFile also has this problem, right?
-- The code needs further clean up , javadoc and unit tests.

Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

ligfietser

Hi Gerd, I use -Xmx4000m

All tiles get stuck, doesnt matter which one.

I have tried one of the Benelux, Germany, Canary Islands.


I guess my splitter version is too old??

I have still r-321....(2014!). Strange because mkgmap 4025 performed well.





_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Arndt Röhrig
In reply to this post by ligfietser

Perhaps it helps to set the value from  --max-nodes lower in splitter. On my old PC it is save to use 55555.

lig fietser <[hidden email]> hat am 6. Januar 2018 um 14:30 geschrieben:

Gerd, I don't know why but mkgmap >v4025 does not work for my maps anymore.

It is doing hours to compile one tile!


My (experimental) DEM settings are

dem-dists=3312,3312,3312,6628,9942
overview-dem-dist=16560

Map levels:
levels = 0:24, 1:23, 2:22, 3:20, 4:18
overview-levels = 5:17, 6:15, 7:14

 

v4025 compiles ok, I updated my OFM Alps, Benelux and Germany recently.


Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 03:09:39
Aan: [hidden email]
Onderwerp: [mkgmap-dev] r4033: new dem options are now documented
 
Hi all,

see http://www.mkgmap.org.uk/doc/options
I've added them in a chapter headed "Hill Shading (DEM) options"
This also means that you no longer need the --x- prefix when using r4033 or newer.

Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
Maybe option --dem should be changed to --dem-source before merging to trunk?

I think these are the other open problems now (no particular order):
-- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
-- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
-- Problems with java heap when creating large or highly detailed maps:
Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
@Frank: I think BuildDEMFile also has this problem, right?
-- The code needs further clean up , javadoc and unit tests.

Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


 

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
In reply to this post by ligfietser
Hi Minko,

I don't think that splitter is the problem.
The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 15:47:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Gerd, I use -Xmx4000m

All tiles get stuck, doesnt matter which one.

I have tried one of the Benelux, Germany, Canary Islands.


I guess my splitter version is too old??

I have still r-321....(2014!). Strange because mkgmap 4025 performed well.



_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

ligfietser

How can I check that? And why does it go wrong with every random tile?

BTW My max-nodes are 1600000


Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:52:32
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented
 
Hi Minko,

I don't think that splitter is the problem.
The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 15:47:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Gerd, I use -Xmx4000m

All tiles get stuck, doesnt matter which one.

I have tried one of the Benelux, Germany, Canary Islands.


I guess my splitter version is too old??

I have still r-321....(2014!). Strange because mkgmap 4025 performed well.



_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

ligfietser

BTW my DEM tiles for my Atlantic map are unzipped hgt's so that is not the case...



Van: mkgmap-dev <[hidden email]> namens lig fietser <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:54:09
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented
 

How can I check that? And why does it go wrong with every random tile?

BTW My max-nodes are 1600000


Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:52:32
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented
 
Hi Minko,

I don't think that splitter is the problem.
The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 15:47:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Gerd, I use -Xmx4000m

All tiles get stuck, doesnt matter which one.

I have tried one of the Benelux, Germany, Canary Islands.


I guess my splitter version is too old??

I have still r-321....(2014!). Strange because mkgmap 4025 performed well.



_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
Are you sure that r4026 was the first version that did not work for you?

Do you use logging with -Dlog.config=... ? If yes, please try without.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 15:55:42
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

BTW my DEM tiles for my Atlantic map are unzipped hgt's so that is not the case...


________________________________
Van: mkgmap-dev <[hidden email]> namens lig fietser <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:54:09
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented


How can I check that? And why does it go wrong with every random tile?

BTW My max-nodes are 1600000

________________________________
Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:52:32
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Minko,

I don't think that splitter is the problem.
The major change in r4026 was for handling of zipped hgt files. Maybe you have an unexpected zip archive ?

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 15:47:06
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Gerd, I use -Xmx4000m

All tiles get stuck, doesnt matter which one.

I have tried one of the Benelux, Germany, Canary Islands.


I guess my splitter version is too old??

I have still r-321....(2014!). Strange because mkgmap 4025 performed well.



_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

ligfietser

I dont use logging Gerd.

And I saw this behaviour in 4028 and 4033.

Here are some of those o5m tiles, maybe they are too big too handle?

http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/







Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:59
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented
 
Are you sure that r4026 was the first version that did not work for you?

Do you use logging with -Dlog.config=... ? If yes, please try without.

Gerd


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
Hi Minko,

can't reproduce the problem with your 10010001.o5m.
Maybe you use a poly file with --dem-poly ?

Please try to reproduce the problem with 10010001.o5m and a directory containing only N50E002.hgt
(as this is the only file needed) and a small set of options (default style)

If you can reproduce the problem please post the options and N50E002.hgt .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 16:05:49
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

I dont use logging Gerd.

And I saw this behaviour in 4028 and 4033.

Here are some of those o5m tiles, maybe they are too big too handle?

http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/





________________________________
Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:59
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented

Are you sure that r4026 was the first version that did not work for you?

Do you use logging with -Dlog.config=... ? If yes, please try without.

Gerd
<http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

ligfietser

Hi Gerd,

I think you are right, it might be the zipped hgt.

If I use it unzipped there seems no delay.

I posted it here:

http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/




Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 07:48:14
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented
 
Hi Minko,

can't reproduce the problem with your 10010001.o5m.
Maybe you use a poly file with --dem-poly ?

Please try to reproduce the problem with 10010001.o5m and a directory containing only N50E002.hgt
(as this is the only file needed) and a small set of options (default style)

If you can reproduce the problem please post the options and N50E002.hgt .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 16:05:49
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

I dont use logging Gerd.

And I saw this behaviour in 4028 and 4033.

Here are some of those o5m tiles, maybe they are too big too handle?

http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/





________________________________
Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:59
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented

Are you sure that r4026 was the first version that did not work for you?

Do you use logging with -Dlog.config=... ? If yes, please try without.

Gerd
<http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

EthnicFood IsGreat
In reply to this post by Gerd Petermann

> Date: Sat, 6 Jan 2018 11:09:39 +0000
> From: Gerd Petermann <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Subject: [mkgmap-dev] r4033: new dem options are now documented
>
> Hi all,
>
> see http://www.mkgmap.org.uk/doc/options
> I've added them in a chapter headed "Hill Shading (DEM) options"
> This also means that you no longer need the --x- prefix when using r4033 or newer.
>
> Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
> If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
> Maybe option --dem should be changed to --dem-source before merging to trunk?
>
> I think these are the other open problems now (no particular order):
> -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
> -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
> -- Problems with java heap when creating large or highly detailed maps:
> Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
> are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
> http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
> One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
> so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
> so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
> @Frank: I think BuildDEMFile also has this problem, right?
> -- The code needs further clean up , javadoc and unit tests.
>
> Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.
>
> Gerd
>
> ------------------------------

Thanks for the documentation Gerd.  I look forward to creating my first
map with the DEM included.  You mention that some sources of .hgt files
contain voids.  Can you specify which ones you're aware of that should
be avoided?

Mark

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
Hi Mark,

my understanding is that files with version 2.1 have many voids. You find them here:
https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/
The files from here seem to have few voids : http://viewfinderpanoramas.org/dem3.html

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von EthnicFood IsGreat <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 17:42:41
An: [hidden email]
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented


> Date: Sat, 6 Jan 2018 11:09:39 +0000
> From: Gerd Petermann <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Subject: [mkgmap-dev] r4033: new dem options are now documented
>
> Hi all,
>
> see http://www.mkgmap.org.uk/doc/options
> I've added them in a chapter headed "Hill Shading (DEM) options"
> This also means that you no longer need the --x- prefix when using r4033 or newer.
>
> Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
> If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
> Maybe option --dem should be changed to --dem-source before merging to trunk?
>
> I think these are the other open problems now (no particular order):
> -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
> -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
> -- Problems with java heap when creating large or highly detailed maps:
> Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
> are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
> http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
> One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
> so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
> so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
> @Frank: I think BuildDEMFile also has this problem, right?
> -- The code needs further clean up , javadoc and unit tests.
>
> Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.
>
> Gerd
>
> ------------------------------

Thanks for the documentation Gerd.  I look forward to creating my first
map with the DEM included.  You mention that some sources of .hgt files
contain voids.  Can you specify which ones you're aware of that should
be avoided?

Mark

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
In reply to this post by ligfietser
Hi Minko,
yes, seems I've never tested this again since r4026 :-(

Now I can reproduce the problem.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 17:37:36
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Gerd,

I think you are right, it might be the zipped hgt.

If I use it unzipped there seems no delay.

I posted it here:

http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/



________________________________
Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 07:48:14
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Minko,

can't reproduce the problem with your 10010001.o5m.
Maybe you use a poly file with --dem-poly ?

Please try to reproduce the problem with 10010001.o5m and a directory containing only N50E002.hgt
(as this is the only file needed) and a small set of options (default style)

If you can reproduce the problem please post the options and N50E002.hgt .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von lig fietser <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 16:05:49
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

I dont use logging Gerd.

And I saw this behaviour in 4028 and 4033.

Here are some of those o5m tiles, maybe they are too big too handle?

http://mijndev.openstreetmap.nl/~ligfietser/mkgmap/





________________________________
Van: mkgmap-dev <[hidden email]> namens Gerd Petermann <[hidden email]>
Verzonden: zaterdag 6 januari 2018 06:59
Aan: Development list for mkgmap
Onderwerp: Re: [mkgmap-dev] r4033: new dem options are now documented

Are you sure that r4026 was the first version that did not work for you?

Do you use logging with -Dlog.config=... ? If yes, please try without.

Gerd
<http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev>
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Henning Scholland
In reply to this post by Gerd Petermann
Hi Gerd,

do you think using tif (as Frank pointed out even has better compression
ratio) instead of zipped hgt will help regarding the RAM-footprint?

Henning
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

Gerd Petermann
Hi Henning,

I don't think so.
Did anybody try to store hgt files in a compressing file system (or a compressed folder)?


Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Henning Scholland <[hidden email]>
Gesendet: Samstag, 6. Januar 2018 19:19:35
An: [hidden email]
Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented

Hi Gerd,

do you think using tif (as Frank pointed out even has better compression
ratio) instead of zipped hgt will help regarding the RAM-footprint?

Henning
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: r4033: new dem options are now documented

EthnicFood IsGreat
In reply to this post by Gerd Petermann

> Date: Sat, 6 Jan 2018 16:51:43 +0000
> From: Gerd Petermann <[hidden email]>
> To: Development list for mkgmap <[hidden email]>
> Subject: Re: [mkgmap-dev] r4033: new dem options are now documented
> Message-ID:
> <[hidden email]>
>
> Content-Type: text/plain; charset="us-ascii"
>
> Hi Mark,
>
> my understanding is that files with version 2.1 have many voids. You find them here:
> https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/
> The files from here seem to have few voids : http://viewfinderpanoramas.org/dem3.html
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von EthnicFood IsGreat <[hidden email]>
> Gesendet: Samstag, 6. Januar 2018 17:42:41
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] r4033: new dem options are now documented
>
>
>> Date: Sat, 6 Jan 2018 11:09:39 +0000
>> From: Gerd Petermann <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Subject: [mkgmap-dev] r4033: new dem options are now documented
>>
>> Hi all,
>>
>> see http://www.mkgmap.org.uk/doc/options
>> I've added them in a chapter headed "Hill Shading (DEM) options"
>> This also means that you no longer need the --x- prefix when using r4033 or newer.
>>
>> Please suggest improvements, esp. if you have found rules to determine "good" dem-dists values.
>> If possible I'd like to do the calculations in mkgmap instead of asking each user to find out what is good.
>> Maybe option --dem should be changed to --dem-source before merging to trunk?
>>
>> I think these are the other open problems now (no particular order):
>> -- determine optimal block-size to avoid MapFailedExeption with large DEM files, if that is too complex we'll change the default from 512 to 2048.
>> -- find possible encoding error for low resolutions, esp. in overview maps. Frank Stinner tries to find a solution for this :-)
>> -- Problems with java heap when creating large or highly detailed maps:
>> Zipped files require more heap memory because the current implementation has to keep a copy of the unzipped content while unzipped files
>> are kept in MappedByteBuffers which don't require java heap. On the other hand they maybe cause other problems as those reported by Carlos:
>> http://gis.19327.n8.nabble.com/mkgmap-r4025-implements-x-dem-poly-option-tp5909038p5909257.html
>> One work around would be to keep the unzipped data in temporary files (more I/O), another solution might be to further change the algorithm
>> so that it doesn't require full random access to all(!) hgt files "touched" by the overview map. I've already tried to change that with r4027
>> so that only some rows are required, but that code is not very elegant. I hope I find a better solution for this.
>> @Frank: I think BuildDEMFile also has this problem, right?
>> -- The code needs further clean up , javadoc and unit tests.
>>
>> Thanks for the good feedback so far, I think this will be another big improvement if merged into trunk.
>>
>> Gerd
>>
>> ------------------------------
> Thanks for the documentation Gerd.  I look forward to creating my first
> map with the DEM included.  You mention that some sources of .hgt files
> contain voids.  Can you specify which ones you're aware of that should
> be avoided?
>
> Mark
>

Thanks Gerd!

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev