Mapnik render multiple geotiffs

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Mapnik render multiple geotiffs

tim martin
Hi

I have been recommended to use Mapnik to combine two geotiffs, one topogrpahic and one hillshade.

I have managed to do this using an XML mapfile and a short python script, just like in the tutorials.

However, I need to create a geotiff output of 132000 by 248000 pixels, even going above 6000 by 6000 i get an out of memory error.

So the other way I think I could do it is run the geotiff+hillshade on the smaller original geotiffs which are 4000 by 4000 each.

However there are 2000 of them so how can I program mapnik to step through each one rather than write 2000 entries in my map.xml file

thanks for your help

Tim

_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users
Reply | Threaded
Open this post in threaded view
|

Re: Mapnik render multiple geotiffs

dieterdreist
Am 21. Februar 2012 16:07 schrieb tim martin <[hidden email]>:

> Hi
>
> I have been recommended to use Mapnik to combine two geotiffs, one
> topogrpahic and one hillshade.
>
> I have managed to do this using an XML mapfile and a short python script,
> just like in the tutorials.
>
> However, I need to create a geotiff output of 132000 by 248000 pixels, even
> going above 6000 by 6000 i get an out of memory error.
>
> So the other way I think I could do it is run the geotiff+hillshade on the
> smaller original geotiffs which are 4000 by 4000 each.
>
> However there are 2000 of them so how can I program mapnik to step through
> each one rather than write 2000 entries in my map.xml file


have a look at vrt (virtual raster tables) from GDAL.

cheers,
Martin
_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users
Reply | Threaded
Open this post in threaded view
|

Re: Mapnik render multiple geotiffs

tim martin
Hi Martin

I tried that but it still had a memory errory.

In my map xml i pointed to the VRT and then in my python I had the following

#!/usr/bin/env python
import mapnik
mapfile = 'map.xml'
map_output = 'map.tif'
m = mapnik.Map(132000, 248000)
mapnik.load_map(m, mapfile)
bbox = mapnik.Box2d(mapnik.Coord(0, 0), mapnik.Coord(700000, 1300000))
m.zoom_to_box(bbox)
mapnik.render_to_file(m, map_output)

Should I be changing the map size or the box?

thanks



On 21 February 2012 15:42, Martin Koppenhoefer <[hidden email]> wrote:
Am 21. Februar 2012 16:07 schrieb tim martin <[hidden email]>:
> Hi
>
> I have been recommended to use Mapnik to combine two geotiffs, one
> topogrpahic and one hillshade.
>
> I have managed to do this using an XML mapfile and a short python script,
> just like in the tutorials.
>
> However, I need to create a geotiff output of 132000 by 248000 pixels, even
> going above 6000 by 6000 i get an out of memory error.
>
> So the other way I think I could do it is run the geotiff+hillshade on the
> smaller original geotiffs which are 4000 by 4000 each.
>
> However there are 2000 of them so how can I program mapnik to step through
> each one rather than write 2000 entries in my map.xml file


have a look at vrt (virtual raster tables) from GDAL.

cheers,
Martin


_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users
Reply | Threaded
Open this post in threaded view
|

Re: Mapnik render multiple geotiffs

dieterdreist
Am 21. Februar 2012 16:48 schrieb tim martin <[hidden email]>:

> #!/usr/bin/env python
> import mapnik
> mapfile = 'map.xml'
> map_output = 'map.tif'
> m = mapnik.Map(132000, 248000)
> mapnik.load_map(m, mapfile)
> bbox = mapnik.Box2d(mapnik.Coord(0, 0), mapnik.Coord(700000, 1300000))
> m.zoom_to_box(bbox)
> mapnik.render_to_file(m, map_output)
>
> Should I be changing the map size or the box?


I guess you'll have to change the map size. I am not sure if mapnik is
the best tool to combine huge tiffs of this dimension, what about
using GDAL? Are you sure you have enough space left? What kind of tif
are you creating? If it was an uncompressed 1-Bit tiff you'd already
need 3,81 GB, if it was 8 Bit it would need 30,5 GB and 24 Bit would
need 91,5 GB. You should also make sure that your filesystem supports
files of this size.

cheers,
Martin
_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users
Reply | Threaded
Open this post in threaded view
|

Re: Mapnik render multiple geotiffs

tim martin
I asked the gdal-dev forum this morning and a response said to use mapnik.

others have given me some other options, which I am trying also. 

the hillshade is 2gb and the topographic is 4.3gb, these are compessed

so thats why i wondered if I could loop through the smaller files which are only a few MB each an 4000x4000 in size.

but I am new to python and mapnik, so not sure if you can write a loop statement to step through each small rater.



On 21 February 2012 16:29, Martin Koppenhoefer <[hidden email]> wrote:
Am 21. Februar 2012 16:48 schrieb tim martin <[hidden email]>:
> #!/usr/bin/env python
> import mapnik
> mapfile = 'map.xml'
> map_output = 'map.tif'
> m = mapnik.Map(132000, 248000)
> mapnik.load_map(m, mapfile)
> bbox = mapnik.Box2d(mapnik.Coord(0, 0), mapnik.Coord(700000, 1300000))
> m.zoom_to_box(bbox)
> mapnik.render_to_file(m, map_output)
>
> Should I be changing the map size or the box?


I guess you'll have to change the map size. I am not sure if mapnik is
the best tool to combine huge tiffs of this dimension, what about
using GDAL? Are you sure you have enough space left? What kind of tif
are you creating? If it was an uncompressed 1-Bit tiff you'd already
need 3,81 GB, if it was 8 Bit it would need 30,5 GB and 24 Bit would
need 91,5 GB. You should also make sure that your filesystem supports
files of this size.

cheers,
Martin


_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users
Reply | Threaded
Open this post in threaded view
|

Re: Mapnik render multiple geotiffs

Iain Malcolm
In reply to this post by tim martin
Tim, mapnik is only sensible if you are aiming to end up with tiles anyway, which from what you said now is not the case.  It isn't the right tool if you want single large output files.

gdal copes well with large files (I have used with single files up to 11Gb, but you do need a reasonable amount of memory in your PC. (2Gb or more)

It could well be a good idea to do the preliminary work on the smaller files you mention, and then combine them, but you may end up with edge problems with this approach.  Are these input files SRTM data by any chance?

I produced contour coloured and hillsahded files for the british isles by merging with gdal, then using hillshade etc. to generate the relevant output files.

From what you have said it sounds like you have produced a file with the data you want in it, but because it lacked the geo bits it was no use, but you can put the geo info back into a tiff, or perhaps use a sidecar style file, although I haven't done that myself,I have seen mentions of it in my geotiff travels.  Also note that many applications will not handle very large tiff files.  The original tiff spec has been extended and not many tools have picked this up yet - (although gdal is one that has).  Original tiff only worked up to around 4Gb.

See this for how to get GDAL to put the geo back into a tiff

The best approach does depend on what tools / applications you finally want to use this info with. It would be helpful if you could briefly describe what your end goal is.

Iain

On 21 February 2012 15:07, tim martin <[hidden email]> wrote:
Hi

I have been recommended to use Mapnik to combine two geotiffs, one topogrpahic and one hillshade.

I have managed to do this using an XML mapfile and a short python script, just like in the tutorials.

However, I need to create a geotiff output of 132000 by 248000 pixels, even going above 6000 by 6000 i get an out of memory error.

So the other way I think I could do it is run the geotiff+hillshade on the smaller original geotiffs which are 4000 by 4000 each.

However there are 2000 of them so how can I program mapnik to step through each one rather than write 2000 entries in my map.xml file

thanks for your help

Tim

_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users



_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users
Reply | Threaded
Open this post in threaded view
|

Re: Mapnik render multiple geotiffs

tim martin
Hi Iain

After much playing I resorted back to using ImageMagik and setup a loop through all the files and apply a constant blend of the two files(topo and hillshade)

As you mentioned I then re-added the projects using gdal_translate and a world file.

I totally agree, support for BIGTIFFs is very low which is a shame when you think about the quality of imagery and other datasets now becoming avaliable.

I am surprised how complex the options are, when I set out I thought this would be relatively easy ;)

thanks again

Tim 

On 22 February 2012 02:05, Iain Malcolm <[hidden email]> wrote:
Tim, mapnik is only sensible if you are aiming to end up with tiles anyway, which from what you said now is not the case.  It isn't the right tool if you want single large output files.

gdal copes well with large files (I have used with single files up to 11Gb, but you do need a reasonable amount of memory in your PC. (2Gb or more)

It could well be a good idea to do the preliminary work on the smaller files you mention, and then combine them, but you may end up with edge problems with this approach.  Are these input files SRTM data by any chance?

I produced contour coloured and hillsahded files for the british isles by merging with gdal, then using hillshade etc. to generate the relevant output files.

From what you have said it sounds like you have produced a file with the data you want in it, but because it lacked the geo bits it was no use, but you can put the geo info back into a tiff, or perhaps use a sidecar style file, although I haven't done that myself,I have seen mentions of it in my geotiff travels.  Also note that many applications will not handle very large tiff files.  The original tiff spec has been extended and not many tools have picked this up yet - (although gdal is one that has).  Original tiff only worked up to around 4Gb.

See this for how to get GDAL to put the geo back into a tiff

The best approach does depend on what tools / applications you finally want to use this info with. It would be helpful if you could briefly describe what your end goal is.

Iain

On 21 February 2012 15:07, tim martin <[hidden email]> wrote:
Hi

I have been recommended to use Mapnik to combine two geotiffs, one topogrpahic and one hillshade.

I have managed to do this using an XML mapfile and a short python script, just like in the tutorials.

However, I need to create a geotiff output of 132000 by 248000 pixels, even going above 6000 by 6000 i get an out of memory error.

So the other way I think I could do it is run the geotiff+hillshade on the smaller original geotiffs which are 4000 by 4000 each.

However there are 2000 of them so how can I program mapnik to step through each one rather than write 2000 entries in my map.xml file

thanks for your help

Tim

_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users




_______________________________________________
Mapnik-users mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/mapnik-users