Holes in the Sea

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Holes in the Sea

rheinskipper1000

Hello again!

 

I tried to follow all the hints, I got here. But I still cannot do proper sea generation.

 

So I provide as much information as I can, to track down the problem.

 

First I cut out a rectangular part from the osm file. I did not use the planet as source but the Europe extract from Geofabrik for time saving. It shouldn´t make any difference as my cutout is somewhere in the middle of the Geofabrik extract. I use the following command for this:

 

set read=--read-pbf D:\OSM\Geofabrik\europe.osm.pbf

set write=--write-pbf D:\OSM\planet\europe.osm.pbf

set bbox=--bounding-box left=-0.4 right=11.3 bottom=39 top=45.4

D:\osm\osmosis\bin\osmosis.bat %read% %bbox% completeRelations=yes --tf accept-relations type=boundary,multipolygon --used-way --used-node outPipe.0=MP %read% %bbox% --tf reject-relations type=boundary,multipolygon outPipe.0=bbox --merge inPipe.0=MP inPipe.1=bbox %write%

 

Then I run the splitter with --no-trim and –overlap options:

 

java -Xmx3000M -jar splitter\splitter.jar --max-nodes=500000 --no-trim --overlap=10000 --resolution=15 --geonames-file=cities15000.zip --output-dir=tiles planet\europe.osm.pbf 2>err1.txt

 

After this, I make the map:

 

cd img

java -Xmx3000m -enableassertions -jar ..\mkgmap\mkgmap.jar -c ..\mkgmap_settings.txt -c ..\tiles\template.args 2>..\err2.txt

cd ..

 

And this is my mkgmap_settings.txt:

 

style-file=..\styles

style=marine_mode

family-name=OSM_Marine

series-name=OSM_Marine

overview-mapname=OSM_Marine

latin1

remove-short-arcs

merge-lines

reduce-point-density-polygon=4

reduce-point-density=5.4

lower-case

add-pois-to-areas

#coastlinefile=..\coastlines_europe.osm.pbf

generate-sea=multipolygon,extend-sea-sectors,close-gaps=6000

name-tag-list='name,name:de,name:latin,name:en'

#index

tdbfile

nsis

 

The error file is attached. I understand that those ExtTypeAttributes errors come from unfinished code for the seamarks. I do not understand why I get the MapSplitter errors, as I already use very low –max-nodes to save memory. And there are still those Polyline errors resulting from sea generation, even on the rectangular cutout.

 

Also attached is a screenshot of those terrible holes in the sea.

 

Everything is running under Win 7 x64 with 4 GB memory. (I know, you´d prefer Linux, but I don´t have a Linux system running.)

 

If it´s a memory issue, which options can I modify to save memory?

 

 

 

http://wiki.openstreetmap.org/wiki/User:RheinSkipper


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

err2.txt (59K) Download Attachment
sea.JPG (141K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

Marko Mäkelä
On Fri, Feb 17, 2012 at 05:14:53PM +0100, RheinSkipper wrote:
>Also attached is a screenshot of those terrible holes in the sea.

I do not have idea about the max-nodes in the splitter, but the holes
are so regular that they should be relatively easy to figure out.

Can you determine the tiles corresponding to the holes? Are there any
natural=coastline ways in the tiles? If not, mkgmap should assume that
the whole tile is land. I would start by checking the hole south of
Nizza, as it is smaller. Try to convert the tile.osm.pbf to XML format
with Osmosis and then do grep 'natural.*coastline' tile.osm. If there is
no match, then my guess should be right.

I have had the opposite problem. Someone added an island to a lake with
natural=coastline. Previously, the tile had no natural=coastline ways
(it was all land). The result was that everything except the small
island was converted to water. mkgmap did not produce a warning;
technically, it could warn about overlap between natural=water,
waterway=riverbank and natural=coastline.

>If it´s a memory issue, which options can I modify to save memory?

As far as I understand, the 'area too small to split' and related
messages are due to limitations of the Garmin map format and of Garmin
devices. They do not have that much RAM to play with; that is why the
map tiles must be relatively small.

I hope that this helps.

        Marko
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
Hi,


I am a step further:

After commenting out
        #natural=coastline [0x15 resolution 16]
in my style, all those Polyline errors are away.

Most of the holes in the sea are also gone. Only the small hole south of
Nice and a big one west of Ireland are still there.

I suppose it´s all about memory. The coastline may be too complex to
process.

Now I ordered some more memory. I hope 8 GB can close those last remaining
sea holes.


http://wiki.openstreetmap.org/wiki/User:RheinSkipper


> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:mkgmap-dev-
> [hidden email]] Im Auftrag von Marko Mäkelä
> Gesendet: Freitag, 17. Februar 2012 22:29
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Holes in the Sea
>
> On Fri, Feb 17, 2012 at 05:14:53PM +0100, RheinSkipper wrote:
> >Also attached is a screenshot of those terrible holes in the sea.
>
> I do not have idea about the max-nodes in the splitter, but the holes are
so
> regular that they should be relatively easy to figure out.
>
> Can you determine the tiles corresponding to the holes? Are there any
> natural=coastline ways in the tiles? If not, mkgmap should assume that the
> whole tile is land. I would start by checking the hole south of Nizza, as
it is
> smaller. Try to convert the tile.osm.pbf to XML format with Osmosis and
then
> do grep 'natural.*coastline' tile.osm. If there is no match, then my guess
> should be right.
>
> I have had the opposite problem. Someone added an island to a lake with
> natural=coastline. Previously, the tile had no natural=coastline ways (it
was
> all land). The result was that everything except the small island was
> converted to water. mkgmap did not produce a warning; technically, it
could
> warn about overlap between natural=water, waterway=riverbank and
> natural=coastline.
>
> >If it´s a memory issue, which options can I modify to save memory?
>
> As far as I understand, the 'area too small to split' and related messages
are

> due to limitations of the Garmin map format and of Garmin devices. They do
> not have that much RAM to play with; that is why the map tiles must be
> relatively small.
>
> I hope that this helps.
>
> Marko
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
In reply to this post by Marko Mäkelä
Still holes in the sea
------------------------

First I bought some more memory to make sure, the holes in the sea are no
memory issue. But even with 8 GB the empty sea tiles are still there.

The empty tile south of Monaco is attached here. I converted it to XML and
there are 28 occurrences of natural=coastline in it.

After increasing --max-nodes from 1000000 to 1200000 this hole is also still
there (tile number changes off course).

I use --no-trim --overlap=5000. Higher overlap makes no difference.




http://wiki.openstreetmap.org/wiki/User:RheinSkipper




> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:mkgmap-dev-
> [hidden email]] Im Auftrag von Marko Mäkelä
> Gesendet: Freitag, 17. Februar 2012 22:29
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Holes in the Sea
>
> On Fri, Feb 17, 2012 at 05:14:53PM +0100, RheinSkipper wrote:
> >Also attached is a screenshot of those terrible holes in the sea.
>
> I do not have idea about the max-nodes in the splitter, but the holes are
so
> regular that they should be relatively easy to figure out.
>
> Can you determine the tiles corresponding to the holes? Are there any
> natural=coastline ways in the tiles? If not, mkgmap should assume that the
> whole tile is land. I would start by checking the hole south of Nizza, as
it is
> smaller. Try to convert the tile.osm.pbf to XML format with Osmosis and
then
> do grep 'natural.*coastline' tile.osm. If there is no match, then my guess
> should be right.
>

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

63240790.osm.pbf (19K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

Marko Mäkelä
On Sun, Feb 26, 2012 at 07:17:39PM +0100, RheinSkipper wrote:
>The empty tile south of Monaco is attached here. I converted it to XML
>and there are 28 occurrences of natural=coastline in it.

How did you convert it to XML? In my conversion, there is no
natural=coastline left. Can you upload the converted XML?

I used this command to convert:

osmosis --rb 63240790.osm.pbf --wx 63240790.osm

For the record, JOSM refused to load the file until I replaced
version="-1" with version="1" and changeset="-1" with changeset="1".

If the tile does not have any closed natural=coastline way, then mkgmap
is "allowed" to make the tile land-only. Maybe splitter could be
"wisened" to draw a natural=coastline rectangle around the otherwise
coastline-free tile?

        Marko
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
You can try to add --coastlinefile=coastlines_europe.osm.pbf
I downloaded the latest from http://fabianowski.eu/osm/coastlines/
and added the coastline of the North-African coast to it:
http://mijndev.openstreetmap.nl/~ligfietser/coast/coastlines_europe.osm.pbf

Works for my Europe map, without any holes.
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

toc-rox
In reply to this post by rheinskipper1000
I have a similar problem with portugal (unmodified geofabrik extract, coastlines).



Klaus

splitter:
--mixed --overlap=10000 --max-nodes=600000

mkgmap:
generate-sea:multipolygon,no-sea-sectors,extend-sea-sectors,close-gaps=5000,land-tag=natural=land
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
Klaus,
If you use the splitter with --no-trim and/or a higher max-nodes (100000) those holes are gone,
the sea tiles will touch the coastlines.
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
In reply to this post by Marko Mäkelä
Sorry Marko, I mixed up the tiles. 63240790 was a seahole west of Ireland
which may have no coastline.

Meanwhile  I re-splitted and re-compiled everything again with some
different options (so the tile names are different now). This time I have
exactly 3 seaholes. Again the one south of Monaco (63240484) has coastline
in it. The one in the Biscay also has.

http://files.mkgmap.org.uk/download/52/Mapsource.JPG
http://files.mkgmap.org.uk/download/53/63240484.osm.pbf
http://files.mkgmap.org.uk/download/54/63240484.xml
http://files.mkgmap.org.uk/download/55/63240320.osm.pbf

Strange is: The tiles with the holes have completely incorrect geonames.
Other geonames seem to be OK.

To make sure, coastline is OK, I extracted one from the planet and used this
as coastline-file this time.

The files linked above are from a split with max-nodes=1200000 and I used
default options for generate-sea. I also tested with max-nodes=1600000, then
the Monaco hole was away but the others were still there.

I had no errors from the splitter or mkgmap (except some corrupted seamarks
and a too long way (ferry route) in the Baltic).

Yes, I have the same problems loading the XML tiles in Josm but with
tweaking "version" and "changeset" to 1 it works.


http://wiki.openstreetmap.org/wiki/User:RheinSkipper



> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:mkgmap-dev-
> [hidden email]] Im Auftrag von Marko Mäkelä
> Gesendet: Montag, 27. Februar 2012 09:40
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] Holes in the Sea
>
> On Sun, Feb 26, 2012 at 07:17:39PM +0100, RheinSkipper wrote:
> >The empty tile south of Monaco is attached here. I converted it to XML
> >and there are 28 occurrences of natural=coastline in it.
>
> How did you convert it to XML? In my conversion, there is no
> natural=coastline left. Can you upload the converted XML?
>
> I used this command to convert:
>
> osmosis --rb 63240790.osm.pbf --wx 63240790.osm
>
> For the record, JOSM refused to load the file until I replaced
version="-1"
> with version="1" and changeset="-1" with changeset="1".
>
> If the tile does not have any closed natural=coastline way, then mkgmap is
> "allowed" to make the tile land-only. Maybe splitter could be "wisened" to
> draw a natural=coastline rectangle around the otherwise coastline-free
tile?
>
> Marko
> _______________________________________________
> mkgmap-dev mailing list
> [hidden email]
> http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
Hi RheinSkipper,

>Strange is: The tiles with the holes have completely incorrect geonames.
>Other geonames seem to be OK.

Use --description in your splitter args, otherwise if there are no places
in that tile, the splitter takes the  description of the previous tile.

Did you use --no-trim in your splitter parameters? Maybe this helps to cover some holes.

If you use --write-kml=filename.kml, the splitter will generate a kml file after the tiles are calculated,
so you can quickly see in Google Earth if there are still tiles in the sea that dont touch any coastline.
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

Marko Mäkelä
In reply to this post by rheinskipper1000
On Tue, Feb 28, 2012 at 02:19:28PM +0100, RheinSkipper wrote:
>To make sure, coastline is OK, I extracted one from the planet and used
>this as coastline-file this time.

How can you be sure that the planet coastline was fully OK on that day?  
Does anyone have validator statistics on good/bad coastline? The Finland
coastline is broken about once in every couple of weeks or months. I
would guess that the planet coastline must be broken every week or more
often, unless people are watching for and fixing bugs every day (like I
am in Finland).

Would it be reasonable to extract the natural=coastline ways from the
planet.osm and load it in JOSM to check it? Or translate only the
coastline in mkgmap, to get all coastline checks? Would it be possible?

For data validation, you might also try OsmAndMapCreator; it has a bit
different checks.

I have never used a separate coastline-file in mkgmap, so I do not know
if it would solve the problem of a sea tile that lacks any
natural=coastline polygon.

Best regards,

        Marko
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

osm-8
Hi
I use josm (validator) to find the bug in the coastline. I think this
method is quiet easy.
But I only search for errors, if mkgmap generates wrong water-textures.
Then I convert pbf-tile to osm-xml, change ="-1" to ="1" an load this
file into josm.
Afterwards filter natural=coastline an use validator with
coastline-check only. Then I load the original data to another layer and
fix any error.

If you will do this with a coastline-extract it should work to, but you
should fix the errors in local data and in osm-db.

Henning

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
In reply to this post by ligfietser
> Use --description in your splitter args, otherwise if there are no places
in that
> tile, the splitter takes the  description of the previous tile.

When I use --description=somename  then the --geoname-file option does not
work anymore. All tiles are named "somename" then.


>
> Did you use --no-trim in your splitter parameters? Maybe this helps to
cover
> some holes.

I did. Without this option there  are many more holes.


After viewing the tiles with Josm, I think I understand the problem now:

There are coastlines in most of the hole-tiles but only because I use
--overlap. The part of the tile that is actually used later does not touch
the coastline. Only the --overlap area does, and will be cut away later. So
a coastline in the overlap area seems not to be enough for --generate-sea to
recognize the tile as sea.


After trying lots of different --max-nodes settings I did not find one that
produces a tile pattern that is completely free of empty tiles. So I will
need to find a workaround.

Modifying areas.list manually could be one way. But I do not really know how
to. And it will be difficult because the bigger hole-tiles have many small
neighbor-tiles which are again somehow nested to each other. So it will be
necessary to change lots of tiles to make them match to a sea-tile made
bigger manually.

It might be easier to tell mkgmap which empty tiles are sea. How can I do
that?


https://wiki.openstreetmap.org/wiki/User:RheinSkipper


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
>When I use --description=somename  then the --geoname-file option does not
>work anymore. All tiles are named "somename" then.

You have to put first --geoname-file then --description=somename

>It might be easier to tell mkgmap which empty tiles are sea. How can I do
>that?

I'd like to know that too, sooner or later I will face that problem too with
my Europe map (until now it went ok).
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
> You have to put first --geoname-file then --description=somename
>

I will try it.

If this works and I can name just the empty tiles "INT-Offshore", it should
be possible to develop an option for mkgmap to overcome the sea hole
problem.

Just let mkgmap recognize all tiles with a defined description as sea.
 


https://wiki.openstreetmap.org/wiki/User:RheinSkipper


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
If you know where those empty tiles are located, maybe put a tiny almost invisible island there
created with josm and merge it with your coastlinefile.
And label it with place="created by mkgmap" so the empty tile will not cause booting problems?

>Just let mkgmap recognize all tiles with a defined description as sea.
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
> If you know where those empty tiles are located, maybe put a tiny almost
> invisible island there created with josm and merge it with your
coastlinefile.
> And label it with place="created by mkgmap" so the empty tile will not
cause
> booting problems?
>

This sounds easier than it is. I created an island with josm. But I cannot
merge it using osmosis. The XML formats are not compatible:
http://trac.openstreetmap.org/ticket/2620


_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
I have used osmconvert to merge a part of the N-African coast with the European coast:
osmconvert coastlines_europe-latest.osm.pbf coastlineafrica.osm -o=coastlines_europe.osm.pbf

As far of the islands, it's better to merge them with the osm extract instead of the coastline-file,
because place=sea & name=atlantic ocean (or name=Méditerranée) isnt't rendered otherwise.


RheinSkipper:
This sounds easier than it is. I created an island with josm. But I cannot
merge it using osmosis. The XML formats are not compatible:
http://trac.openstreetmap.org/ticket/2620
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

rheinskipper1000
> I have used osmconvert to merge a part of the N-African coast with the
> European coast:
> osmconvert coastlines_europe-latest.osm.pbf coastlineafrica.osm -
> o=coastlines_europe.osm.pbf
>
> As far of the islands, it's better to merge them with the osm extract instead
> of the coastline-file, because place=sea & name=atlantic ocean (or
> name=Méditerranée) isnt't rendered otherwise.
>

But it is not possible to read files created by Josm with osmosis. See http://trac.openstreetmap.org/ticket/2620 for further information.

I also tried to merge Josm output with "osmt". But like osmosis it cannot read Josm files.

_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Reply | Threaded
Open this post in threaded view
|

Re: Holes in the Sea

ligfietser
I dont use osmosis, where do you need it for?
_______________________________________________
mkgmap-dev mailing list
[hidden email]
http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
12