DEM File Format and mkgmap

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

DEM File Format and mkgmap

Gerd Petermann
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
        at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
        at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
        at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
        at test.display.DemDisplay.print(DemDisplay.java:845)
        at test.display.CommonDisplay.display(CommonDisplay.java:171)
        at test.display.CommonDisplay.display(CommonDisplay.java:196)
        at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink
Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:

> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>          at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>          at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>          at test.display.DemDisplay.print(DemDisplay.java:845)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:

> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>          at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>          at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>          at test.display.DemDisplay.print(DemDisplay.java:845)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink
Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
>> Hi all,
>>
>> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
>> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
>> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
>> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>>
>> Reg. the format:
>> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
>> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
>> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
>> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
>> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
>> A 2nd trick to save space is to use run length encoding for repeated values.
>> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
>> Frank has done really great work here.
>>
>> For those who want to try the display tool:
>> You have to compile on your own and you also need an installed mgkmap.jar.
>> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
>> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>>
>> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>>           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>>           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>>           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>>           at test.display.DemDisplay.print(DemDisplay.java:845)
>>           at test.display.CommonDisplay.display(CommonDisplay.java:171)
>>           at test.display.CommonDisplay.display(CommonDisplay.java:196)
>>           at test.display.DemDisplay.main(DemDisplay.java:903)
>>
>> Gerd
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
>> Hi all,
>>
>> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
>> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
>> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
>> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>>
>> Reg. the format:
>> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
>> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
>> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
>> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
>> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
>> A 2nd trick to save space is to use run length encoding for repeated values.
>> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
>> Frank has done really great work here.
>>
>> For those who want to try the display tool:
>> You have to compile on your own and you also need an installed mgkmap.jar.
>> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
>> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>>
>> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>>           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>>           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>>           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>>           at test.display.DemDisplay.print(DemDisplay.java:845)
>>           at test.display.CommonDisplay.display(CommonDisplay.java:171)
>>           at test.display.CommonDisplay.display(CommonDisplay.java:196)
>>           at test.display.DemDisplay.main(DemDisplay.java:903)
>>
>> Gerd
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped   with repeated plateau len
64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go
using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:

> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
>> Hi Nick,
>>
>> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
>> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
>> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>>
>> A compiled display.jar r505 is here:
>> http://files.mkgmap.org.uk/download/361/display.jar
>> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
>> you can run e.g.
>> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>>
>> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
>> which were the input for BuildDemFile .
>>
>> Gerd
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 13:00:27
>> An: [hidden email]
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Gerd
>>
>> Presumably some hgt data gets stored in a tile from which other data is
>> predicted? Perhaps the first 'row'?
>>
>> Would be great if you could  compile the DemDisplay.java so we can do
>> ome testing as well.
>>
>> I'm glad you are almost 'getting there'!
>>
>> r
>>
>> Nick
>>
>>
>> On 03/12/2017 11:16, Gerd Petermann wrote:
>>> Hi all,
>>>
>>> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
>>> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
>>> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
>>> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>>>
>>> Reg. the format:
>>> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
>>> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
>>> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
>>> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
>>> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
>>> A 2nd trick to save space is to use run length encoding for repeated values.
>>> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
>>> Frank has done really great work here.
>>>
>>> For those who want to try the display tool:
>>> You have to compile on your own and you also need an installed mgkmap.jar.
>>> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
>>> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>>>
>>> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
>>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>>>            at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>>>            at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>>>            at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>>>            at test.display.DemDisplay.print(DemDisplay.java:845)
>>>            at test.display.CommonDisplay.display(CommonDisplay.java:171)
>>>            at test.display.CommonDisplay.display(CommonDisplay.java:196)
>>>            at test.display.DemDisplay.main(DemDisplay.java:903)
>>>
>>> Gerd
>>> _______________________________________________
>>> mkgmap-dev mailing list
>>> [hidden email]
>>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick


On 04/12/2017 15:02, osm@pinns wrote:
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev [hidden email] im Auftrag von osm@pinns [hidden email]
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev [hidden email] im Auftrag von osm@pinns [hidden email]
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
           at test.display.DemDisplay.print(DemDisplay.java:845)
           at test.display.CommonDisplay.display(CommonDisplay.java:171)
           at test.display.CommonDisplay.display(CommonDisplay.java:196)
           at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi Nick,
yes, I know that.

I've fixed some problem in DemDisplay, so here is r507:
http://files.mkgmap.org.uk/download/363/display.jar

It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
the problem with that data is that I don't know for sure what values they contain.
If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
reproduce the DEM file and finally trace what values Franks program creates.
With a bit of luck this will help to fix the problems with the demo data.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Montag, 4. Dezember 2017 16:27:20
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick

On 04/12/2017 15:02, osm@pinns wrote:
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
           at test.display.DemDisplay.print(DemDisplay.java:845)
           at test.display.CommonDisplay.display(CommonDisplay.java:171)
           at test.display.CommonDisplay.display(CommonDisplay.java:196)
           at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi all,

I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 09:26:37
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Nick,
yes, I know that.

I've fixed some problem in DemDisplay, so here is r507:
http://files.mkgmap.org.uk/download/363/display.jar

It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
the problem with that data is that I don't know for sure what values they contain.
If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
reproduce the DEM file and finally trace what values Franks program creates.
With a bit of luck this will help to fix the problems with the demo data.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Montag, 4. Dezember 2017 16:27:20
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick

On 04/12/2017 15:02, osm@pinns wrote:
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
           at test.display.DemDisplay.print(DemDisplay.java:845)
           at test.display.CommonDisplay.display(CommonDisplay.java:171)
           at test.display.CommonDisplay.display(CommonDisplay.java:196)
           at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink

Gert

I think I may have a dem that causes a problem

http://files.mkgmap.org.uk/download/364/12345684.zip

C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
.DemDisplay 12345684.DEM  1>log12345684.txt
Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
e bytes
        at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
98)
        at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
02)
        at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
        at test.display.DemDisplay.print(DemDisplay.java:816)
        at test.display.CommonDisplay.display(CommonDisplay.java:171)
        at test.display.CommonDisplay.display(CommonDisplay.java:196)
        at test.display.DemDisplay.main(DemDisplay.java:875)

It is part of OSM Greece

r

Nick



On 05/12/2017 10:17, Gerd Petermann wrote:
Hi all,

I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.

Gerd

________________________________________
Von: mkgmap-dev [hidden email] im Auftrag von Gerd Petermann [hidden email]
Gesendet: Dienstag, 5. Dezember 2017 09:26:37
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Nick,
yes, I know that.

I've fixed some problem in DemDisplay, so here is r507:
http://files.mkgmap.org.uk/download/363/display.jar

It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
the problem with that data is that I don't know for sure what values they contain.
If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
reproduce the DEM file and finally trace what values Franks program creates.
With a bit of luck this will help to fix the problems with the demo data.

Gerd

________________________________________
Von: mkgmap-dev [hidden email] im Auftrag von osm@pinns [hidden email]
Gesendet: Montag, 4. Dezember 2017 16:27:20
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick

On 04/12/2017 15:02, osm@pinns wrote:
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev [hidden email][hidden email] im Auftrag von osm@pinns [hidden email][hidden email]
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email][hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev [hidden email][hidden email] im Auftrag von osm@pinns [hidden email][hidden email]
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email][hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
           at test.display.DemDisplay.print(DemDisplay.java:845)
           at test.display.CommonDisplay.display(CommonDisplay.java:171)
           at test.display.CommonDisplay.display(CommonDisplay.java:196)
           at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email][hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email][hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email][hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email][hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email][hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email][hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi Nick,

yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 13:33:16
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Gert

I think I may have a dem that causes a problem

http://files.mkgmap.org.uk/download/364/12345684.zip

C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
.DemDisplay 12345684.DEM  1>log12345684.txt
Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
e bytes
        at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
98)
        at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
02)
        at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
        at test.display.DemDisplay.print(DemDisplay.java:816)
        at test.display.CommonDisplay.display(CommonDisplay.java:171)
        at test.display.CommonDisplay.display(CommonDisplay.java:196)
        at test.display.DemDisplay.main(DemDisplay.java:875)

It is part of OSM Greece

r

Nick


On 05/12/2017 10:17, Gerd Petermann wrote:

Hi all,

I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 09:26:37
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Nick,
yes, I know that.

I've fixed some problem in DemDisplay, so here is r507:
http://files.mkgmap.org.uk/download/363/display.jar

It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
the problem with that data is that I don't know for sure what values they contain.
If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
reproduce the DEM file and finally trace what values Franks program creates.
With a bit of luck this will help to fix the problems with the demo data.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 16:27:20
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick

On 04/12/2017 15:02, osm@pinns wrote:
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could  compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
           at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
           at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
           at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
           at test.display.DemDisplay.print(DemDisplay.java:845)
           at test.display.CommonDisplay.display(CommonDisplay.java:171)
           at test.display.CommonDisplay.display(CommonDisplay.java:196)
           at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

andreas.schmidt.hetschbach
In reply to this post by Gerd Petermann

Hi Gerd,

 

I added http://files.mkgmap.org.uk/detail/365

 

Which includes two dem/tre files which cause

 

Exception in thread "main" java.lang.AssertionError

        at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:470)

        at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:802)

        at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)

        at test.display.DemDisplay.print(DemDisplay.java:816)

        at test.display.CommonDisplay.display(CommonDisplay.java:171)

        at test.display.CommonDisplay.display(CommonDisplay.java:196)

        at test.display.DemDisplay.main(DemDisplay.java:875)

 

Regards

 

Andreas

 

Gesendet von Mail für Windows 10

 

Von: [hidden email]
Gesendet: Dienstag, 5. Dezember 2017 11:17
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

 

Hi all,

I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von Gerd Petermann <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 09:26:37
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Nick,
yes, I know that.

I've fixed some problem in DemDisplay, so here is r507:
http://files.mkgmap.org.uk/download/363/display.jar

It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
the problem with that data is that I don't know for sure what values they contain.
If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
reproduce the DEM file and finally trace what values Franks program creates.
With a bit of luck this will help to fix the problems with the demo data.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Montag, 4. Dezember 2017 16:27:20
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

I think all the dem files from the very old GBTopo can only be parsed up to a certain point

I've included 3 excmples with tre

http://files.mkgmap.org.uk/download/362/00145336.zip

Nick

On 04/12/2017 15:02, osm@pinns wrote:
Hi Gerd

Thanks for letting me know

That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created

I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar

r

Nick



On 04/12/2017 13:49, Gerd Petermann wrote:
Hi Nick,

I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 14:35:15
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Many thanks Gerd

Will have a go at it today and let you know my findings, if any !

Nick


On 04/12/2017 13:27, Gerd Petermann wrote:
Hi Nick,

My understanding is that the prediction starts with the values from the header (min and max height in the tile)
The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.

A compiled display.jar r505 is here:
http://files.mkgmap.org.uk/download/361/display.jar
If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
you can run e.g.
java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt

I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
which were the input for BuildDemFile .

Gerd
________________________________________
Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
Gesendet: Montag, 4. Dezember 2017 13:00:27
An: [hidden email]<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Presumably some hgt data gets stored in a tile from which other data is
predicted? Perhaps the first 'row'?

Would be great if you could compile the DemDisplay.java so we can do
ome testing as well.

I'm glad you are almost 'getting there'!

r

Nick


On 03/12/2017 11:16, Gerd Petermann wrote:
Hi all,

I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile.
My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.

Reg. the format:
It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
needed to store the value. If the delta is larger than expected the so called BigBin encoding is used.
The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
A 2nd trick to save space is to use run length encoding for repeated values.
So, the tricky part is to find out exactly how the prediction algo by Garmin works.
Frank has done really great work here.

For those who want to try the display tool:
You have to compile on your own and you also need an installed mgkmap.jar.
If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt

I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this
Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
at test.display.DemDisplay.print(DemDisplay.java:845)
at test.display.CommonDisplay.display(CommonDisplay.java:171)
at test.display.CommonDisplay.display(CommonDisplay.java:196)
at test.display.DemDisplay.main(DemDisplay.java:903)

Gerd
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]<mailto:[hidden email]>
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink
In reply to this post by Gerd Petermann
Hi Gerd

Excellent, will do so more hunting!

Nick


On 05/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM  1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
>          at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
>          at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
>          at test.display.DemDisplay.print(DemDisplay.java:816)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: [hidden email]<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm@pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>             at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>             at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>             at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>             at test.display.DemDisplay.print(DemDisplay.java:845)
>             at test.display.CommonDisplay.display(CommonDisplay.java:171)
>             at test.display.CommonDisplay.display(CommonDisplay.java:196)
>             at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi Nick,

you can stop for a while. With the data from Andreas I have enough special cases for now. Please wait for a new DemDisplay version.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 14:33:14
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Excellent, will do so more hunting!

Nick


On 05/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM  1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
>          at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
>          at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
>          at test.display.DemDisplay.print(DemDisplay.java:816)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: [hidden email]<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm@pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>             at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>             at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>             at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>             at test.display.DemDisplay.print(DemDisplay.java:845)
>             at test.display.CommonDisplay.display(CommonDisplay.java:171)
>             at test.display.CommonDisplay.display(CommonDisplay.java:196)
>             at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
In reply to this post by nwillink
OK , with r508 I fixed the problems that showed up in the files from Nick and Andreas, bot were related to the meaning of header info.
I've also changed the program so that it doesn't produce so much output unless option --verbose is given as 1st arg to speed up the error hunt ;-)
See http://files.mkgmap.org.uk/download/366/display.jar

Unfortunately it still fails to read all Garmin demo files. Obviously the prediction implemented in DemDisplay is not always
in sync with Garmins algo. Maybe this is related to the fact that Garmin maps use values in feet instead of metres.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 14:33:14
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Excellent, will do so more hunting!

Nick


On 05/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM  1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
>          at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
>          at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
>          at test.display.DemDisplay.print(DemDisplay.java:816)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: [hidden email]<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm@pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped   with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could  compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by  BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc.  In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value.  If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile  which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>             at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>             at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>             at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>             at test.display.DemDisplay.print(DemDisplay.java:845)
>             at test.display.CommonDisplay.display(CommonDisplay.java:171)
>             at test.display.CommonDisplay.display(CommonDisplay.java:196)
>             at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

andreas.schmidt.hetschbach

Hi Gerd,

 

added http://files.mkgmap.org.uk/detail/367 which cause

 

C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710081\63710081.DEM  1>63710081\dem_decode.txt )

Exception in thread "main" java.lang.ArithmeticException: / by zero

       at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)

        at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)

        at test.display.DemDisplay.print(DemDisplay.java:882)

        at test.display.CommonDisplay.display(CommonDisplay.java:171)

        at test.display.CommonDisplay.display(CommonDisplay.java:196)

        at test.display.DemDisplay.main(DemDisplay.java:946)

 

C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710082\63710082.DEM  1>63710082\dem_decode.txt )

Exception in thread "main" java.lang.ArithmeticException: / by zero

        at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)

        at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)

        at test.display.DemDisplay.print(DemDisplay.java:882)

        at test.display.CommonDisplay.display(CommonDisplay.java:171)

        at test.display.CommonDisplay.display(CommonDisplay.java:196)

        at test.display.DemDisplay.main(DemDisplay.java:946)

 

regards

 

Andreas

 

Gesendet von Mail für Windows 10

 

Von: [hidden email]
Gesendet: Mittwoch, 6. Dezember 2017 10:52
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

 

OK , with r508 I fixed the problems that showed up in the files from Nick and Andreas, bot were related to the meaning of header info.
I've also changed the program so that it doesn't produce so much output unless option --verbose is given as 1st arg to speed up the error hunt ;-)
See http://files.mkgmap.org.uk/download/366/display.jar

Unfortunately it still fails to read all Garmin demo files. Obviously the prediction implemented in DemDisplay is not always
in sync with Garmins algo. Maybe this is related to the fact that Garmin maps use values in feet instead of metres.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 14:33:14
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Excellent, will do so more hunting!

Nick


On 05/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM 1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
> at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
> at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
> at test.display.DemDisplay.print(DemDisplay.java:816)
> at test.display.CommonDisplay.display(CommonDisplay.java:171)
> at test.display.CommonDisplay.display(CommonDisplay.java:196)
> at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: [hidden email]<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm@pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value. If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
> at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
> at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
> at test.display.DemDisplay.print(DemDisplay.java:845)
> at test.display.CommonDisplay.display(CommonDisplay.java:171)
> at test.display.CommonDisplay.display(CommonDisplay.java:196)
> at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann
Hi Andreas,

thanks for that. fixed with r509:
http://files.mkgmap.org.uk/download/368/display.jar

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von [hidden email] <[hidden email]>
Gesendet: Mittwoch, 6. Dezember 2017 13:10:52
An: mkgmap-dev; Development list for mkgmap
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd,

added http://files.mkgmap.org.uk/detail/367 which cause

C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710081\63710081.DEM  1>63710081\dem_decode.txt )
Exception in thread "main" java.lang.ArithmeticException: / by zero
       at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
        at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
        at test.display.DemDisplay.print(DemDisplay.java:882)
        at test.display.CommonDisplay.display(CommonDisplay.java:171)
        at test.display.CommonDisplay.display(CommonDisplay.java:196)
        at test.display.DemDisplay.main(DemDisplay.java:946)

C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710082\63710082.DEM  1>63710082\dem_decode.txt )
Exception in thread "main" java.lang.ArithmeticException: / by zero
        at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
        at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
        at test.display.DemDisplay.print(DemDisplay.java:882)
        at test.display.CommonDisplay.display(CommonDisplay.java:171)
        at test.display.CommonDisplay.display(CommonDisplay.java:196)
        at test.display.DemDisplay.main(DemDisplay.java:946)

regards

Andreas

Gesendet von Mail<https://go.microsoft.com/fwlink/?LinkId=550986> für Windows 10

Von: Gerd Petermann<mailto:[hidden email]>
Gesendet: Mittwoch, 6. Dezember 2017 10:52
An: Development list for mkgmap<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

OK , with r508 I fixed the problems that showed up in the files from Nick and Andreas, bot were related to the meaning of header info.
I've also changed the program so that it doesn't produce so much output unless option --verbose is given as 1st arg to speed up the error hunt ;-)
See http://files.mkgmap.org.uk/download/366/display.jar

Unfortunately it still fails to read all Garmin demo files. Obviously the prediction implemented in DemDisplay is not always
in sync with Garmins algo. Maybe this is related to the fact that Garmin maps use values in feet instead of metres.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 14:33:14
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Excellent, will do so more hunting!

Nick


On 05/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM 1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
> at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
> at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
> at test.display.DemDisplay.print(DemDisplay.java:816)
> at test.display.CommonDisplay.display(CommonDisplay.java:171)
> at test.display.CommonDisplay.display(CommonDisplay.java:196)
> at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: [hidden email]<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm@pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value. If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
> at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
> at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
> at test.display.DemDisplay.print(DemDisplay.java:845)
> at test.display.CommonDisplay.display(CommonDisplay.java:171)
> at test.display.CommonDisplay.display(CommonDisplay.java:196)
> at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

nwillink
Hi Gerd

have just done Nepal area with Himalayas - thanks to the new display.jar
it's much quicker to check

But, no issues found with the latest display.jar, except it has
highlighted one major issue creating DEM files.

At one stage using Franks program I was told there were 443977 tiles to
be created - after five minutes it seemed to trickle to 1 tile a second
- on that basis it would take all day !

I resplit the pfb using small maxnodes; this halved the number of tiles
to 270000 and only used 1 layer- but even that took 40 mins.

I have a suspicion that most people only want to create dem files on the
hoof when the time penalty is acceptable.

You can always create one off maps containing only say contours and dem
data.

This might have some relevance if ,  when (!) mkgmap is capable of
creating dem files.

Nick


On 06/12/2017 13:55, Gerd Petermann wrote:

> Hi Andreas,
>
> thanks for that. fixed with r509:
> http://files.mkgmap.org.uk/download/368/display.jar
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von [hidden email] <[hidden email]>
> Gesendet: Mittwoch, 6. Dezember 2017 13:10:52
> An: mkgmap-dev; Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd,
>
> added http://files.mkgmap.org.uk/detail/367 which cause
>
> C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710081\63710081.DEM  1>63710081\dem_decode.txt )
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>         at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
>          at test.display.DemDisplay.print(DemDisplay.java:882)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:946)
>
> C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710082\63710082.DEM  1>63710082\dem_decode.txt )
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>          at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
>          at test.display.DemDisplay.print(DemDisplay.java:882)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:946)
>
> regards
>
> Andreas
>
> Gesendet von Mail<https://go.microsoft.com/fwlink/?LinkId=550986> für Windows 10
>
> Von: Gerd Petermann<mailto:[hidden email]>
> Gesendet: Mittwoch, 6. Dezember 2017 10:52
> An: Development list for mkgmap<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> OK , with r508 I fixed the problems that showed up in the files from Nick and Andreas, bot were related to the meaning of header info.
> I've also changed the program so that it doesn't produce so much output unless option --verbose is given as 1st arg to speed up the error hunt ;-)
> See http://files.mkgmap.org.uk/download/366/display.jar
>
> Unfortunately it still fails to read all Garmin demo files. Obviously the prediction implemented in DemDisplay is not always
> in sync with Garmins algo. Maybe this is related to the fact that Garmin maps use values in feet instead of metres.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 14:33:14
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Excellent, will do so more hunting!
>
> Nick
>
>
> On 05/12/2017 13:27, Gerd Petermann wrote:
>> Hi Nick,
>>
>> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
>> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
>> An: [hidden email]
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Gert
>>
>> I think I may have a dem that causes a problem
>>
>> http://files.mkgmap.org.uk/download/364/12345684.zip
>>
>> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
>> .DemDisplay 12345684.DEM 1>log12345684.txt
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
>> e bytes
>> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
>> 98)
>> at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
>> 02)
>> at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
>> at test.display.DemDisplay.print(DemDisplay.java:816)
>> at test.display.CommonDisplay.display(CommonDisplay.java:171)
>> at test.display.CommonDisplay.display(CommonDisplay.java:196)
>> at test.display.DemDisplay.main(DemDisplay.java:875)
>>
>> It is part of OSM Greece
>>
>> r
>>
>> Nick
>>
>>
>> On 05/12/2017 10:17, Gerd Petermann wrote:
>>
>> Hi all,
>>
>> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
>> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
>> An: Development list for mkgmap
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Nick,
>> yes, I know that.
>>
>> I've fixed some problem in DemDisplay, so here is r507:
>> http://files.mkgmap.org.uk/download/363/display.jar
>>
>> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
>> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
>> the problem with that data is that I don't know for sure what values they contain.
>> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
>> reproduce the DEM file and finally trace what values Franks program creates.
>> With a bit of luck this will help to fix the problems with the demo data.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 16:27:20
>> An: [hidden email]<mailto:[hidden email]>
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Gerd
>>
>> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>>
>> I've included 3 excmples with tre
>>
>> http://files.mkgmap.org.uk/download/362/00145336.zip
>>
>> Nick
>>
>> On 04/12/2017 15:02, osm@pinns wrote:
>> Hi Gerd
>>
>> Thanks for letting me know
>>
>> That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created
>>
>> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>>
>> r
>>
>> Nick
>>
>>
>>
>> On 04/12/2017 13:49, Gerd Petermann wrote:
>> Hi Nick,
>>
>> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 14:35:15
>> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Many thanks Gerd
>>
>> Will have a go at it today and let you know my findings, if any !
>>
>> Nick
>>
>>
>> On 04/12/2017 13:27, Gerd Petermann wrote:
>> Hi Nick,
>>
>> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
>> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
>> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>>
>> A compiled display.jar r505 is here:
>> http://files.mkgmap.org.uk/download/361/display.jar
>> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
>> you can run e.g.
>> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>>
>> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
>> which were the input for BuildDemFile .
>>
>> Gerd
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 13:00:27
>> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Gerd
>>
>> Presumably some hgt data gets stored in a tile from which other data is
>> predicted? Perhaps the first 'row'?
>>
>> Would be great if you could compile the DemDisplay.java so we can do
>> ome testing as well.
>>
>> I'm glad you are almost 'getting there'!
>>
>> r
>>
>> Nick
>>
>>
>> On 03/12/2017 11:16, Gerd Petermann wrote:
>> Hi all,
>>
>> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
>> I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile.
>> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
>> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>>
>> Reg. the format:
>> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
>> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
>> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
>> needed to store the value. If the delta is larger than expected the so called BigBin encoding is used.
>> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
>> A 2nd trick to save space is to use run length encoding for repeated values.
>> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
>> Frank has done really great work here.
>>
>> For those who want to try the display tool:
>> You have to compile on your own and you also need an installed mgkmap.jar.
>> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
>> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>>
>> I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>> at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>> at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>> at test.display.DemDisplay.print(DemDisplay.java:845)
>> at test.display.CommonDisplay.display(CommonDisplay.java:171)
>> at test.display.CommonDisplay.display(CommonDisplay.java:196)
>> at test.display.DemDisplay.main(DemDisplay.java:903)
>>
>> Gerd
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

Gerd Petermann

Hi Nick,

I think Frank already pointed out that his code is not much optimized for speed.
The same is true for DemDisplay. Once DemDisplay is able to read all DEM files
I'll start looking at the conversion from HGT format into DEM format.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Mittwoch, 6. Dezember 2017 19:39:33
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

have just done Nepal area with Himalayas - thanks to the new display.jar
it's much quicker to check

But, no issues found with the latest display.jar, except it has
highlighted one major issue creating DEM files.

At one stage using Franks program I was told there were 443977 tiles to
be created - after five minutes it seemed to trickle to 1 tile a second
- on that basis it would take all day !

I resplit the pfb using small maxnodes; this halved the number of tiles
to 270000 and only used 1 layer- but even that took 40 mins.

I have a suspicion that most people only want to create dem files on the
hoof when the time penalty is acceptable.

You can always create one off maps containing only say contours and dem
data.

This might have some relevance if ,  when (!) mkgmap is capable of
creating dem files.

Nick


On 06/12/2017 13:55, Gerd Petermann wrote:

> Hi Andreas,
>
> thanks for that. fixed with r509:
> http://files.mkgmap.org.uk/download/368/display.jar
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von [hidden email] <[hidden email]>
> Gesendet: Mittwoch, 6. Dezember 2017 13:10:52
> An: mkgmap-dev; Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd,
>
> added http://files.mkgmap.org.uk/detail/367 which cause
>
> C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710081\63710081.DEM  1>63710081\dem_decode.txt )
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>         at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
>          at test.display.DemDisplay.print(DemDisplay.java:882)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:946)
>
> C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710082\63710082.DEM  1>63710082\dem_decode.txt )
> Exception in thread "main" java.lang.ArithmeticException: / by zero
>          at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
>          at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
>          at test.display.DemDisplay.print(DemDisplay.java:882)
>          at test.display.CommonDisplay.display(CommonDisplay.java:171)
>          at test.display.CommonDisplay.display(CommonDisplay.java:196)
>          at test.display.DemDisplay.main(DemDisplay.java:946)
>
> regards
>
> Andreas
>
> Gesendet von Mail<https://go.microsoft.com/fwlink/?LinkId=550986> für Windows 10
>
> Von: Gerd Petermann<mailto:[hidden email]>
> Gesendet: Mittwoch, 6. Dezember 2017 10:52
> An: Development list for mkgmap<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> OK , with r508 I fixed the problems that showed up in the files from Nick and Andreas, bot were related to the meaning of header info.
> I've also changed the program so that it doesn't produce so much output unless option --verbose is given as 1st arg to speed up the error hunt ;-)
> See http://files.mkgmap.org.uk/download/366/display.jar
>
> Unfortunately it still fails to read all Garmin demo files. Obviously the prediction implemented in DemDisplay is not always
> in sync with Garmins algo. Maybe this is related to the fact that Garmin maps use values in feet instead of metres.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 14:33:14
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Excellent, will do so more hunting!
>
> Nick
>
>
> On 05/12/2017 13:27, Gerd Petermann wrote:
>> Hi Nick,
>>
>> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
>> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
>> An: [hidden email]
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Gert
>>
>> I think I may have a dem that causes a problem
>>
>> http://files.mkgmap.org.uk/download/364/12345684.zip
>>
>> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
>> .DemDisplay 12345684.DEM 1>log12345684.txt
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
>> e bytes
>> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
>> 98)
>> at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
>> 02)
>> at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
>> at test.display.DemDisplay.print(DemDisplay.java:816)
>> at test.display.CommonDisplay.display(CommonDisplay.java:171)
>> at test.display.CommonDisplay.display(CommonDisplay.java:196)
>> at test.display.DemDisplay.main(DemDisplay.java:875)
>>
>> It is part of OSM Greece
>>
>> r
>>
>> Nick
>>
>>
>> On 05/12/2017 10:17, Gerd Petermann wrote:
>>
>> Hi all,
>>
>> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
>> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
>> An: Development list for mkgmap
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Nick,
>> yes, I know that.
>>
>> I've fixed some problem in DemDisplay, so here is r507:
>> http://files.mkgmap.org.uk/download/363/display.jar
>>
>> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
>> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
>> the problem with that data is that I don't know for sure what values they contain.
>> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
>> reproduce the DEM file and finally trace what values Franks program creates.
>> With a bit of luck this will help to fix the problems with the demo data.
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 16:27:20
>> An: [hidden email]<mailto:[hidden email]>
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Gerd
>>
>> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>>
>> I've included 3 excmples with tre
>>
>> http://files.mkgmap.org.uk/download/362/00145336.zip
>>
>> Nick
>>
>> On 04/12/2017 15:02, osm@pinns wrote:
>> Hi Gerd
>>
>> Thanks for letting me know
>>
>> That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created
>>
>> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>>
>> r
>>
>> Nick
>>
>>
>>
>> On 04/12/2017 13:49, Gerd Petermann wrote:
>> Hi Nick,
>>
>> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>>
>> Gerd
>>
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 14:35:15
>> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Many thanks Gerd
>>
>> Will have a go at it today and let you know my findings, if any !
>>
>> Nick
>>
>>
>> On 04/12/2017 13:27, Gerd Petermann wrote:
>> Hi Nick,
>>
>> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
>> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
>> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>>
>> A compiled display.jar r505 is here:
>> http://files.mkgmap.org.uk/download/361/display.jar
>> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
>> you can run e.g.
>> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>>
>> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
>> which were the input for BuildDemFile .
>>
>> Gerd
>> ________________________________________
>> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Gesendet: Montag, 4. Dezember 2017 13:00:27
>> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>>
>> Hi Gerd
>>
>> Presumably some hgt data gets stored in a tile from which other data is
>> predicted? Perhaps the first 'row'?
>>
>> Would be great if you could compile the DemDisplay.java so we can do
>> ome testing as well.
>>
>> I'm glad you are almost 'getting there'!
>>
>> r
>>
>> Nick
>>
>>
>> On 03/12/2017 11:16, Gerd Petermann wrote:
>> Hi all,
>>
>> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
>> I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile.
>> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
>> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>>
>> Reg. the format:
>> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
>> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
>> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
>> needed to store the value. If the delta is larger than expected the so called BigBin encoding is used.
>> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
>> A 2nd trick to save space is to use run length encoding for repeated values.
>> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
>> Frank has done really great work here.
>>
>> For those who want to try the display tool:
>> You have to compile on your own and you also need an installed mgkmap.jar.
>> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
>> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>>
>> I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this
>> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
>> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
>> at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
>> at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
>> at test.display.DemDisplay.print(DemDisplay.java:845)
>> at test.display.CommonDisplay.display(CommonDisplay.java:171)
>> at test.display.CommonDisplay.display(CommonDisplay.java:196)
>> at test.display.DemDisplay.main(DemDisplay.java:903)
>>
>> Gerd
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]<mailto:[hidden email]>
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>>
>> _______________________________________________
>> mkgmap-dev mailing list
>> [hidden email]
>> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: DEM File Format and mkgmap

andreas.schmidt.hetschbach
In reply to this post by Gerd Petermann

Hi Gerd,

 

checked

 

  • Germany
  • Mallorca
  • Sweden

 

without any fault.

 

Although for now it is some manual work  (produce the DEM-files with Franks program, correct the tdb-file and so on), the results showed up in basecamp (3d) and Oregon600 (hillshading) are  such amazing. I´m really looking forward getting it integrated into the mkgmap-process by you.

 

Great work!

 

Andreas

 

 

 

Gesendet von Mail für Windows 10

 

Von: [hidden email]
Gesendet: Mittwoch, 6. Dezember 2017 14:55
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

 

Hi Andreas,

thanks for that. fixed with r509:
http://files.mkgmap.org.uk/download/368/display.jar

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von [hidden email] <[hidden email]>
Gesendet: Mittwoch, 6. Dezember 2017 13:10:52
An: mkgmap-dev; Development list for mkgmap
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd,

added http://files.mkgmap.org.uk/detail/367 which cause

C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710081\63710081.DEM 1>63710081\dem_decode.txt )
Exception in thread "main" java.lang.ArithmeticException: / by zero
at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
at test.display.DemDisplay.print(DemDisplay.java:882)
at test.display.CommonDisplay.display(CommonDisplay.java:171)
at test.display.CommonDisplay.display(CommonDisplay.java:196)
at test.display.DemDisplay.main(DemDisplay.java:946)

C:\Garmin\FAMILY_6371.gmap\Product1>(java -ea -Xmx1000m -cp c:\garmin\bin\display.jar;c:\garmin\bin\mkgmap.jar test.display.DemDisplay 63710082\63710082.DEM 1>63710082\dem_decode.txt )
Exception in thread "main" java.lang.ArithmeticException: / by zero
at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:862)
at test.display.DemDisplay.readFileHeader(DemDisplay.java:925)
at test.display.DemDisplay.print(DemDisplay.java:882)
at test.display.CommonDisplay.display(CommonDisplay.java:171)
at test.display.CommonDisplay.display(CommonDisplay.java:196)
at test.display.DemDisplay.main(DemDisplay.java:946)

regards

Andreas

Gesendet von Mail<https://go.microsoft.com/fwlink/?LinkId=550986> für Windows 10

Von: Gerd Petermann<mailto:[hidden email]>
Gesendet: Mittwoch, 6. Dezember 2017 10:52
An: Development list for mkgmap<mailto:[hidden email]>
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

OK , with r508 I fixed the problems that showed up in the files from Nick and Andreas, bot were related to the meaning of header info.
I've also changed the program so that it doesn't produce so much output unless option --verbose is given as 1st arg to speed up the error hunt ;-)
See http://files.mkgmap.org.uk/download/366/display.jar

Unfortunately it still fails to read all Garmin demo files. Obviously the prediction implemented in DemDisplay is not always
in sync with Garmins algo. Maybe this is related to the fact that Garmin maps use values in feet instead of metres.

Gerd

________________________________________
Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
Gesendet: Dienstag, 5. Dezember 2017 14:33:14
An: [hidden email]
Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap

Hi Gerd

Excellent, will do so more hunting!

Nick


On 05/12/2017 13:27, Gerd Petermann wrote:

> Hi Nick,
>
> yes, that one was special because it contained 4 byte offsets. DemDisplay r507 supports only up to 3 bytes. Fixed soon.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]> im Auftrag von osm@pinns <[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 13:33:16
> An: [hidden email]
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Gert
>
> I think I may have a dem that causes a problem
>
> http://files.mkgmap.org.uk/download/364/12345684.zip
>
> C:\Users\Owner\Desktop\mkgmap-r3997>java -cp display.jar;mkgmap.jar test.display
> .DemDisplay 12345684.DEM 1>log12345684.txt
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaibl
> e bytes
> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:4
> 98)
> at test.display.DemDisplay$DemSection.decodeBitstreams(DemDisplay.java:8
> 02)
> at test.display.DemDisplay.readFileHeader(DemDisplay.java:859)
> at test.display.DemDisplay.print(DemDisplay.java:816)
> at test.display.CommonDisplay.display(CommonDisplay.java:171)
> at test.display.CommonDisplay.display(CommonDisplay.java:196)
> at test.display.DemDisplay.main(DemDisplay.java:875)
>
> It is part of OSM Greece
>
> r
>
> Nick
>
>
> On 05/12/2017 10:17, Gerd Petermann wrote:
>
> Hi all,
>
> I've changed my strategy now. I've modified DemDisplay to display the decoded height together with lat/lon. If a new value differs by more than a few meters from the value left or above it is probably wrong. Hope this helps me to find the remaining special cases.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von Gerd Petermann <[hidden email]><mailto:[hidden email]>
> Gesendet: Dienstag, 5. Dezember 2017 09:26:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Nick,
> yes, I know that.
>
> I've fixed some problem in DemDisplay, so here is r507:
> http://files.mkgmap.org.uk/download/363/display.jar
>
> It seems I was not clear about my strategy. I am searching for DEM files created by Franks program which display tool cannot parse.
> Please don't post data from Garmin if that doesn't come from free demo maps. I have quite a few DEM files from those demo maps, but
> the problem with that data is that I don't know for sure what values they contain.
> If you find a DEM produced by Franks program which r507 still cannot read I need the options for BuildDEMFile and the *.TRE file so that I can
> reproduce the DEM file and finally trace what values Franks program creates.
> With a bit of luck this will help to fix the problems with the demo data.
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 16:27:20
> An: [hidden email]<mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> I think all the dem files from the very old GBTopo can only be parsed up to a certain point
>
> I've included 3 excmples with tre
>
> http://files.mkgmap.org.uk/download/362/00145336.zip
>
> Nick
>
> On 04/12/2017 15:02, osm@pinns wrote:
> Hi Gerd
>
> Thanks for letting me know
>
> That is probably why eventually it stopped with repeated plateau len 64 messages on one of the dems builddemfile created
>
> I have some 'old' non nt Garmin Topo maps with dem and will have a go using your display.jar
>
> r
>
> Nick
>
>
>
> On 04/12/2017 13:49, Gerd Petermann wrote:
> Hi Nick,
>
> I just found out that display tool can not yet read DEM tiles with multiple zoom levels. Working on it now...
>
> Gerd
>
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 14:35:15
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Many thanks Gerd
>
> Will have a go at it today and let you know my findings, if any !
>
> Nick
>
>
> On 04/12/2017 13:27, Gerd Petermann wrote:
> Hi Nick,
>
> My understanding is that the prediction starts with the values from the header (min and max height in the tile)
> The first row is indeed rather easy to decode. Frank has described many more details in his pdf, and his program
> seems to create valid DEM data since Basecamp displays it, I just try to reduce the number of special cases now.
>
> A compiled display.jar r505 is here:
> http://files.mkgmap.org.uk/download/361/display.jar
> If you also download and unzip this mkgmap package http://www.mkgmap.org.uk/download/mkgmap-r3997.zip
> you can run e.g.
> java -cp display.jar;mkgmap-r3997\mkgmap.jar test.display.DemDisplay 12345678.DEM > log.txt
>
> I am interested in those files created by BuildDemFile which create a run time exception. If you find one or more, I need the original *.TRE file(s)
> which were the input for BuildDemFile .
>
> Gerd
> ________________________________________
> Von: mkgmap-dev <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]> im Auftrag von osm@pinns <[hidden email]><mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Gesendet: Montag, 4. Dezember 2017 13:00:27
> An: [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> Betreff: Re: [mkgmap-dev] DEM File Format and mkgmap
>
> Hi Gerd
>
> Presumably some hgt data gets stored in a tile from which other data is
> predicted? Perhaps the first 'row'?
>
> Would be great if you could compile the DemDisplay.java so we can do
> ome testing as well.
>
> I'm glad you are almost 'getting there'!
>
> r
>
> Nick
>
>
> On 03/12/2017 11:16, Gerd Petermann wrote:
> Hi all,
>
> I think I understand now much more about this format. I used Frank Stinners code for BuildDEMFile and his pdf to write a decoder in the display tool.
> I still fails to read some Garmin DEM files but seems to work with the files produced by BuildDEMFile.
> My current work flow is to generate DEM files with BuildDEMFile and read them with my decoder. If the decoder fails I can find out why because I can trace step by step what BuildDEMFile does.
> I guess that I did not yet hit all special cases identified by Frank, but I am making good progress now, so I hope I'll be able to start implementing a DEM encoder in mkgmap soon.
>
> Reg. the format:
> It is quite different to those used in other GARMIN sub files like NOD, NET, RGN etc. In short: The data is organized in tiles, typically 64*64 values. Instead of storing each value
> Garmin uses some statistics to predict the next value based on the already processed values. This predicted value is subtracted from the real value and only the delta is stored.
> The better the prediction, the closer to 0 is the delta. The prediction also calculates the error-probability of the prediction and uses this to select the number of bits which are
> needed to store the value. If the delta is larger than expected the so called BigBin encoding is used.
> The format to store the delta is optimized for values close to 0, so that many deltas are stored with only 1 or 2 bits.
> A 2nd trick to save space is to use run length encoding for repeated values.
> So, the tricky part is to find out exactly how the prediction algo by Garmin works.
> Frank has done really great work here.
>
> For those who want to try the display tool:
> You have to compile on your own and you also need an installed mgkmap.jar.
> If that is available a command like this (replace d:\... with your paths) should produce a large txt file:
> java -ea -Xmx1000m -cp d:\display\dist\display.jar;d:\mkgmap\dist\mkgmap.jar test.display.DemDisplay test.DEM > dem_decode.txt
>
> I am now searching for DEM files produced by BuildDEMFile which are not properly processed by display tool. When it fails you'll see a traceback like this
> Exception in thread "main" java.lang.RuntimeException: did not read all avalaible bytes
> at test.display.DemDisplay$DemTile.decodeBitStreamData(DemDisplay.java:532)
> at test.display.DemDisplay$DemSection.decodeBitstream(DemDisplay.java:831)
> at test.display.DemDisplay.readFileHeader(DemDisplay.java:887)
> at test.display.DemDisplay.print(DemDisplay.java:845)
> at test.display.CommonDisplay.display(CommonDisplay.java:171)
> at test.display.CommonDisplay.display(CommonDisplay.java:196)
> at test.display.DemDisplay.main(DemDisplay.java:903)
>
> Gerd
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]><mailto:[hidden email]><mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]<mailto:[hidden email]>
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
12