Coastline shapefiles

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Coastline shapefiles

Paul Norman
I have been experimenting with generating the coastline shapefiles locally
using http://svn.openstreetmap.org/applications/utils/coastcheck/ and ended
up with a few questions

1. Is this the same code that is currently used to generate the processed_p
files?

2. It took my server about one hour from the start of extracting the
coastline data to creating the shapefiles. My understanding was that this
process took about a day. I'm using a different route to extract coastline
data than osm2coast, could this account for the difference?

3. If I were to run the coastline generation daily and upload the files
somewhere, could someone then host a slippymap showing coastline errors?

4. Given that it only took an hour to generate, is there any way to get more
frequent updates to the coastline files?


_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Coastline shapefiles

David Groom
----- Original Message -----
From: "Paul Norman" <[hidden email]>
To: <[hidden email]>
Sent: Monday, January 30, 2012 5:09 AM
Subject: [OSM-dev] Coastline shapefiles


>I have been experimenting with generating the coastline shapefiles locally
> using http://svn.openstreetmap.org/applications/utils/coastcheck/ and
> ended
> up with a few questions
>
> 1. Is this the same code that is currently used to generate the
> processed_p
> files?
>
> 2. It took my server about one hour from the start of extracting the
> coastline data to creating the shapefiles. My understanding was that this
> process took about a day. I'm using a different route to extract coastline
> data than osm2coast, could this account for the difference?
>
> 3. If I were to run the coastline generation daily and upload the files
> somewhere, could someone then host a slippymap showing coastline errors?
>

Firstly , is a slippy map showing coastline errors really necessary?  If you
could upload the error point shapefile somewhere daily I'm sure you would
find these errors were corrected within a few hours.  Michal Migurski has
been generating coastline error files every 3 - 4 weeks
http://metro.teczno.com/#coastline, and recently I've been finding that by
the time I try and correct the errors someone has got there first.

Secondly , if my memory serves me correctly, the original coastline error
map showed both points where there was a disconnect in the coastline, and
also places where there were geometry issues (mainly the small artefacts
left over form the PGS import).  These errors were I believe displayed on
the slippy map by way of  transparent tiles, and obviously there was time
taken to generate these tiles, and then bandwidth / space requirements
involved in deploying them.

Recently ( last 3 - 4 months)  the number of disconnects has been reduced to
a low number, and it is now possible to deploy a slippy map much more
simply, see http://www.wightpaths.co.uk/coast/  and hosting requirements for
this are therefore minimal, if it was felt a coastline error map was useful.

The number of the second type of error (geometry issues) is still quite
high, and probably the only way to show these on a slippy map would be to go
down the transparent tile route, though I guess some form of clustering of
marker points would be possible.  I do wonder though what the benefit of
showing these on a slippy map would be.  Over the last few months I've been
going round clearing these errors up, and will continue to do so.  I think I
have reduced the number by about 2 / 3 rds so quantity wise it is not an
impossible task.

If at the end there are any remaining errors they are likely to be two
causes:

a) these are in Canada - Paul you are probably aware of my posting to the
talk-ca list asking if it was worth my effort in clearing these up due to
the possibility of wholesale replacement of the current coastline data with
Canvec data

b) clearing up the error is too complex.  I know this is may sound like an
excuse, but there was at least one area in the USA where a latter import of
data with ways tagged as natural = water over the top of PGS ways tagged as
natural = coastline, made a quick resolution of the problem difficult, and I
gave up and moved to another area.  Without wishing to sound arrogant, if
someone needs a slippy map to identify the errors, then they are probably
not the right person to go about sorting out the issue in this particular
circumstance. ( BTW that's not a comment aimed at you Paul -  if you're
running the coastline error files and can get GeoBase NHN into OSM then I
realise you are technically competent ; I'm just saying that crowdsourcing
coastline error issues may in fact lead to more errors being created than
are solved)

Anyway to summarise.  My belief is that producing a slippy map showing the
coastline errors is not particularly necessary

Daily production, and upload to somewhere accessible, of the error points
shapefilefile ,and to a lesser extent the processed_p shapefile, would be
very useful

David


> 4. Given that it only took an hour to generate, is there any way to get
> more
> frequent updates to the coastline files?
>
>
> _______________________________________________
> dev mailing list
> [hidden email]
> http://lists.openstreetmap.org/listinfo/dev
>



_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Coastline shapefiles

Lennard
In reply to this post by Paul Norman
On 30-1-2012 6:09, Paul Norman wrote:
> I have been experimenting with generating the coastline shapefiles locally
> using http://svn.openstreetmap.org/applications/utils/coastcheck/ and ended
> up with a few questions
>
> 1. Is this the same code that is currently used to generate the processed_p
> files?

Yes, except for the slight tweaking of some initial values. Also, two
sets of shapefiles are generated, each with slightly different values
for shape overlap. The first is processed_p, the second is another
shapefile from which shoreline_300 is generated. The latter one takes
more time to generate and then simplify.

> 2. It took my server about one hour from the start of extracting the
> coastline data to creating the shapefiles. My understanding was that this
> process took about a day. I'm using a different route to extract coastline
> data than osm2coast, could this account for the difference?

A full coastline shapefile run (ie. both processed_p and shoreline_300)
takes a few hours to generate, generally. I have no doubt that current
hardware (cpu, ssd) can have a drastic positive influence to get it to
your 'one hour'.

> 3. If I were to run the coastline generation daily and upload the files
> somewhere, could someone then host a slippymap showing coastline errors?

I'm sure there could be ways to get that done.

> 4. Given that it only took an hour to generate, is there any way to get more
> frequent updates to the coastline files?

Another limiting factor in this is that you would need the full planet
file, to be able to extract the coastlines. Applying diffs to update
that planet file adds a serious amount of time to a coastline run.


--
Lennard

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Coastline shapefiles

Paul Norman
> From: Lennard [mailto:[hidden email]]
> Sent: Monday, January 30, 2012 12:33 PM
> To: [hidden email]
> Subject: Re: [OSM-dev] Coastline shapefiles
>
> On 30-1-2012 6:09, Paul Norman wrote:
> > I have been experimenting with generating the coastline shapefiles
> > locally using
> > http://svn.openstreetmap.org/applications/utils/coastcheck/ and ended
> > up with a few questions
> >
> > 1. Is this the same code that is currently used to generate the
> > processed_p files?
>
> Yes, except for the slight tweaking of some initial values. Also, two
> sets of shapefiles are generated, each with slightly different values
> for shape overlap. The first is processed_p, the second is another
> shapefile from which shoreline_300 is generated. The latter one takes
> more time to generate and then simplify.

How are the shoreline_300 generated? Coastcheck doesn't have any
documentation, I had to figure out what doit did and how to compile the
programs.

>
> > 2. It took my server about one hour from the start of extracting the
> > coastline data to creating the shapefiles. My understanding was that
> > this process took about a day. I'm using a different route to extract
> > coastline data than osm2coast, could this account for the difference?
>
> A full coastline shapefile run (ie. both processed_p and shoreline_300)
> takes a few hours to generate, generally. I have no doubt that current
> hardware (cpu, ssd) can have a drastic positive influence to get it to
> your 'one hour'.

I'm querying my pgsnapshot database - I think this accounts for most of the
differences. My server runs an AMD Phenom II 1090T x6 with 6 7200 RPM drives
in RAID10


>
> > 3. If I were to run the coastline generation daily and upload the
> > files somewhere, could someone then host a slippymap showing coastline
> errors?
>
> I'm sure there could be ways to get that done.

If anyone is interested in doing something with these files, I can set up a
cron script that will run the generation and scp it somewhere.

>
> > 4. Given that it only took an hour to generate, is there any way to
> > get more frequent updates to the coastline files?
>
> Another limiting factor in this is that you would need the full planet
> file, to be able to extract the coastlines. Applying diffs to update
> that planet file adds a serious amount of time to a coastline run.

Since I keep a pgsnapshot database up to date, this isn't an issue for my
setup.


_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Coastline shapefiles

Cartinus
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The original coastline processing from way back ran on a virtual
server that ran on hardware from 2008 or before that was also doing a
lot of other things and that took about half a day to generate the
shapefiles (including the error files)

It's no wonder your much newer dedicated hardware is a lot faster.

On 01/31/2012 12:39 AM, Paul Norman wrote:

>>>>> 2. It took my server about one hour from the start of
>>>>> extracting the coastline data to creating the shapefiles.
>>>>> My understanding was that this process took about a day.
>>>>> I'm using a different route to extract coastline data than
>>>>> osm2coast, could this account for the difference?
>>>
>>> A full coastline shapefile run (ie. both processed_p and
>>> shoreline_300) takes a few hours to generate, generally. I have
>>> no doubt that current hardware (cpu, ssd) can have a drastic
>>> positive influence to get it to your 'one hour'.
> I'm querying my pgsnapshot database - I think this accounts for
> most of the differences. My server runs an AMD Phenom II 1090T x6
> with 6 7200 RPM drives in RAID10
>
>


- ---
m.v.g.,
Cartinus
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPJ1/QAAoJEInLUvwrL2kIEdkH/10GAUuicF8BgVs17aJ3HB0B
hj+hqcyt1hxae416/EpKdRpj2ZFXgm3R24Q8XaOZoeadPAL9Se8V/NBGiQUf9y+4
dDawTz0ZQgXJe4ooC8xdXUjaZz/weUH8ugXg8RIzOLqzdJvVoDHVxRa1Etfjv3b/
e8hM1paTh6z+6QzXwNXcpBtvF6xQUqOrJSnHneH6ZVZ8k3lSXJyTpzQybT/0RYe+
6QSU0B5/8UdwBTA835zDKTQLUFr8M+8RKvlhMM0ejO42KPV3ZPiVjGV9eemUFmuy
9k985fZZw/4Xr6nBhKKbNxjTTPFUeeJLBET/VAda8DJOxtBf6G5oQ3EU0LdEuXU=
=1Q5r
-----END PGP SIGNATURE-----

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev