Precompiled boundary files in one zip file

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Precompiled boundary files in one zip file

Gerd Petermann
Hi WanMil,

I am still busy rewriting all the boundary code to allow reading bnd data from zip.
Most of it is done, and I also found a few errors contained in my patch
boundary_prep_quadtree_v2.patch

I have now a version that can read old bnd format, new quadtree bnd file format,
both from zip or normal directory, and its BoundaryQuadtree delivers the same results as the
version in the performance branch.
I did not see a big difference between reading from zip or directory, I think that is okay.
I did not see any multi-threading problems while reading from zip, so I did not code anything
special for this. You mentioned such problems, did you mean updating zip files?

I don't think that we need a routine to write / create zip files, do we?
I think we need a utility similar to BoundaryMerger to allow creating the bnd files for planet in multiple steps, so that's what I am working on now.


Gerd

Reply | Threaded
Open this post in threaded view
|

Re: Precompiled boundary files in one zip file

WanMil
> Hi WanMil,
>
> I am still busy rewriting all the boundary code to allow reading bnd data
> from zip.
> Most of it is done, and I also found a few errors contained in my patch
> boundary_prep_quadtree_v2.patch

Great!!

>
> I have now a version that can read old bnd format, new quadtree bnd file
> format,
> both from zip or normal directory, and its BoundaryQuadtree delivers the
> same results as the
> version in the performance branch.

Great!!

> I did not see a big difference between reading from zip or directory, I
> think that is okay.

I expected an improvement when using more threads because then reading
from disk is often a bottleneck. But obviously this is different between
systems.

> I did not see any multi-threading problems while reading from zip, so I did
> not code anything
> special for this. You mentioned such problems, did you mean updating zip
> files?

I remember that I read warnings that the Java zip file implementation is
not multithread capable. I'll have to search for them and will let you
know if I find them.

>
> I don't think that we need a routine to write / create zip files, do we?

There will be one real improvement: The LocationHook first checks if the
bounds directory contains bounds file. The time checking that has been
improved much by your patch. But I still see quite high timings when
using multiple threads. If using a zip file this timing should be much
better because one just have to check if the on zip file is there.

A 2nd thing is that it is much easier to handle for users. That's what
Klaus alias toc-rox metioned.

So decide yourself. I would vote for using zip files (and it would also
be ok for me to drop the support for non zip files).

> I think we need a utility similar to BoundaryMerger to allow creating the
> bnd files for planet in multiple steps, so that's what I am working on now.

Yes. Otherwise it is not possible to create boundaries for larger
regions if your computer does not have tons of GB.

WanMil

>
>
> Gerd
>
>
>
> --
> View this message in context: http://gis.19327.n5.nabble.com/Precompiled-boundary-files-in-one-zip-file-tp5477353p5477353.html
> Sent from the Mkgmap Development mailing list archive at Nabble.com.
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Precompiled boundary files in one zip file

Gerd Petermann
Hi WanMil,

> > I did not see any multi-threading problems while reading from zip, so I did
> > not code anything
> > special for this. You mentioned such problems, did you mean updating zip
> > files?
>
> I remember that I read warnings that the Java zip file implementation is
> not multithread capable. I'll have to search for them and will let you
> know if I find them.

OK, I just found general comments like "it is not threadsave"

>
> >
> > I don't think that we need a routine to write / create zip files, do we?
>
> There will be one real improvement: The LocationHook first checks if the
> bounds directory contains bounds file. The time checking that has been
> improved much by your patch. But I still see quite high timings when
> using multiple threads. If using a zip file this timing should be much
> better because one just have to check if the on zip file is there.
>
> A 2nd thing is that it is much easier to handle for users. That's what
> Klaus alias toc-rox metioned.
>
> So decide yourself. I would vote for using zip files (and it would also
> be ok for me to drop the support for non zip files).

I like to use zip file as input, my question is if there is a need to write
or update *.zip . I think it is sufficient to use a standalone zip program.

>
> > I think we need a utility similar to BoundaryMerger to allow creating the
> > bnd files for planet in multiple steps, so that's what I am working on now.
>
> Yes. Otherwise it is not possible to create boundaries for larger
> regions if your computer does not have tons of GB.

Well, a special program to calculate only the bnd files would be an alternative,
but I don't plan to code that.

Gerd

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev