Postcodes to Shapefile

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Postcodes to Shapefile

Kev js1982
I am currently trying to create a series of shapefiles from postcodes (using
OS Open Geo Data) using the code from "Random Junk" (
http://random.dev.openstreetmap.org/postcodes/#) running on Ubuntu 9.10 but
I can't get it working.

What I have done so far is to create a folder called /home/kev/osm

In this folder is "makeShapeColoured.py"

Prompted by http://www.mail-archive.com/talk at openstreetmap.org/msg20932.htmlI
have downloaded the pyshapelib from
http://ftp.intevation.de/users/bh/pyshapelib/pyshapelib-0.3.tar.gz (the URL
appears to have changed when compared with the mail archive post), which has
been extracted into the same folder and then renamed shapelib (i.e. there
exists a file called /home/kev/osm/shapelib/shapelib.py ).

If I then run

$ python makeShapeColoured.py

I get the following message

Traceback (most recent call last):
  File "makeShapeColoured.py", line 34, in <module>
    import shapelib.shapelib as shp, shapelib.dbflib as dbf
ImportError: No module named shapelib.shapelib

I'm guessing this is because the "python binding" needs installing - but
which binding?  Google seams to bring back results about the subversion,
gdal and not just a generic python bindings - has anyone any ideas what
needs to be done here?

Install is a standard (English/GB) installation of Ubuntu running inside
Virtual Box which has had the following command run
sudo aptitude install openssh-server python python-mapnik mapnik-utils
libmapnik-dev python-dev
and all the respective dependencies installed.


Hopefully converting it to running on the OS projection will be as simple as
changing the project line!

Regards,

Kev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstreetmap.org/pipermail/talk-gb/attachments/20100426/8441af8b/attachment.htm 

Reply | Threaded
Open this post in threaded view
|

Postcodes to Shapefile

Kev js1982
On Mon, Apr 26, 2010 at 12:00 PM, Kev js1982 <osm at kevswindells.eu> wrote:

> I am currently trying to create a series of shapefiles from postcodes
> (using OS Open Geo Data) using the code from "Random Junk" (
> http://random.dev.openstreetmap.org/postcodes/#) running on Ubuntu 9.10
> but I can't get it working.
>
> lots of blah blah about what I did...
>

Think i've sussed most of it...

I zapped my pyshapelib folder and downloaded both it and shapelib again

With the shapelib and pyshapelib tar gzs inside my osm folder I then issued
the following commands

tar -xvzf shapelib-1.2.10.tar.gz
mv shapelib-1.2.10 shapelib
tar -xvzf pyshapelib-0.3.tar.gz
mv pyshapelib-0.3 shapelib/pyshapelib/
cd shapelib
make
cd pyshapelib
python setup.py build
sudo python setup.py install
cd ../../
# The next line is really important if you want python to think this folder
has python scripts
touch shapelib/__init__.py
cp shapelib/pyshapelib/* shapelib/

This seamed to get over the original problem

Then you need to ensure you input file has no trailing lines

And now to work out why I'm getting

Traceback (most recent call last):
  File "makeShapeColoured.py", line 349, in <module>
    result = voronoi.computeVoronoiDiagram(pts)
  File "/home/kev/osm/voronoi.py", line 746, in computeVoronoiDiagram
    voronoi(siteList,context)
  File "/home/kev/osm/voronoi.py", line 206, in voronoi
    edge = Edge.bisect(bot,newsite)
  File "/home/kev/osm/voronoi.py", line 404, in bisect
    newedge.a = dx/dy
ZeroDivisionError: float division
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstreetmap.org/pipermail/talk-gb/attachments/20100426/3e1d6ded/attachment.htm 

Reply | Threaded
Open this post in threaded view
|

Postcodes to Shapefile

Chris Hill-6
In reply to this post by Kev js1982
Kev js1982 wrote:

> I am currently trying to create a series of shapefiles from postcodes
> (using OS Open Geo Data) using the code from "Random Junk"
> (http://random.dev.openstreetmap.org/postcodes/#) running on Ubuntu
> 9.10 but I can't get it working.
>
> What I have done so far is to create a folder called /home/kev/osm
>
> In this folder is "makeShapeColoured.py"
>
> Prompted by
> http://www.mail-archive.com/talk at openstreetmap.org/msg20932.html I
> have downloaded the pyshapelib from
> http://ftp.intevation.de/users/bh/pyshapelib/pyshapelib-0.3.tar.gz 
> (the URL appears to have changed when compared with the mail archive
> post), which has been extracted into the same folder and then renamed
> shapelib (i.e. there exists a file called
> /home/kev/osm/shapelib/shapelib.py ).
>
> If I then run
>
> $ python makeShapeColoured.py
>
> I get the following message
>
> Traceback (most recent call last):
>   File "makeShapeColoured.py", line 34, in <module>
>     import shapelib.shapelib as shp, shapelib.dbflib as dbf
> ImportError: No module named shapelib.shapelib
>
> I'm guessing this is because the "python binding" needs installing -
> but which binding?  Google seams to bring back results about the
> subversion, gdal and not just a generic python bindings - has anyone
> any ideas what needs to be done here?
>
> Install is a standard (English/GB) installation of Ubuntu running
> inside Virtual Box which has had the following command run
> sudo aptitude install openssh-server python python-mapnik mapnik-utils
> libmapnik-dev python-dev
> and all the respective dependencies installed.
>
>
> Hopefully converting it to running on the OS projection will be as
> simple as changing the project line!
I have been working on the OS boundary files for counties etc and the
coastline files. They all have a .prj which is incorrect. If you
re-project the shpfiles using the OS supplied .prj using, for example,
GDAL routines you will get polygons or lines which are substantially
inaccurate. Jerry Clough supplied me a corrected .prj file:

PROJCS["British National Grid (ORD SURV GB)", GEOGCS["unnamed",
DATUM["D_OSGB_1936", SPHEROID["Airy - 1848",6377563,299.319997677743],
TOWGS84[446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894]],
PRIMEM["Greenwich",0],UNIT["degree",0.0174532925199433]],
PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",49],  
PARAMETER["central_meridian",-2],
PARAMETER["scale_factor",0.9996012717],  
PARAMETER["false_easting",400000], PARAMETER["false_northing",-100000],
UNIT["METER",1]]

Replace the .prj file with the data above.


Cheers, Chris


Reply | Threaded
Open this post in threaded view
|

Postcodes to Shapefile

randomjunk
In reply to this post by Kev js1982
On Mon, Apr 26, 2010 at 1:24 PM, Kev js1982 <osm at kevswindells.eu> wrote:

> On Mon, Apr 26, 2010 at 12:00 PM, Kev js1982 <osm at kevswindells.eu> wrote:
>>
>> I am currently trying to create a series of shapefiles from postcodes
>> (using OS Open Geo Data) using the code from "Random Junk"
>> (http://random.dev.openstreetmap.org/postcodes/#) running on Ubuntu 9.10 but
>> I can't get it working.
>>
>> lots of blah blah about what I did...
>
> And now to work out why I'm getting
>
> Traceback (most recent call last):
> ? File "makeShapeColoured.py", line 349, in <module>
> ??? result = voronoi.computeVoronoiDiagram(pts)
> ? File "/home/kev/osm/voronoi.py", line 746, in computeVoronoiDiagram
> ??? voronoi(siteList,context)
> ? File "/home/kev/osm/voronoi.py", line 206, in voronoi
> ??? edge = Edge.bisect(bot,newsite)
> ? File "/home/kev/osm/voronoi.py", line 404, in bisect
> ??? newedge.a = dx/dy
> ZeroDivisionError: float division
>


These are caused by more than one postcode for the same point --
you'll need to preprocess the input files to remove any duplicates
coordinates. There's quite a lot of apparent PO boxes and other odd
postcodes in the OS data which result in duplicate points.

Also note that it'll use about 6GBs of RAM to run for the complete OS
dataset of 1.6 million points.

Dave


Reply | Threaded
Open this post in threaded view
|

Postcodes to Shapefile

Jerry Clough - OSM
In reply to this post by Kev js1982
GRASS seems to have some kind of Voronoi algorithm, but I find its interface very non-intuitive, so have not found it to try it out. This may be worth pursuing as an alternative route.

As an aside: I notice that virtually all postcode boundaries are obviously created in this way. For instance the Philips Street Atlas shows totally implausible boundaries along the River Trent in S. Notts. Given that postcode boundaries are ultimately determined by logistically sensible walks for postmen, in this case it's pretty safe to assume that the boundary is actually the river. What this means is that by applying a bit of local knowledge and the existing points it is possible to create better delineating zones in OSM than appear in current mapping. Whether this is a good thing to do, or not, I leave for others to decide.




________________________________
From: Kev js1982 <osm at kevswindells.eu>
To: OSM - Talk GB <talk-gb at openstreetmap.org>
Sent: Mon, 26 April, 2010 12:24:26
Subject: Re: [Talk-GB] Postcodes to Shapefile


On Mon, Apr 26, 2010 at 12:00 PM, Kev js1982 <osm at kevswindells.eu> wrote:

>I am currently trying to create a series of shapefiles from postcodes (using OS Open Geo Data) using the code from "Random Junk" (http://random.dev.openstreetmap.org/postcodes/#) running on Ubuntu 9.10 but I can't get it working.
>
>lots of blah blah about what I did...
>

Think i've sussed most of it...

I zapped my pyshapelib folder and downloaded both it and shapelib again

With the shapelib and pyshapelib tar gzs inside my osm folder I then
issued the following commands

tar -xvzf shapelib-1.2.10.tar.gz
mv shapelib-1.2.10 shapelib
tar -xvzf pyshapelib-0.3.tar.gz
mv pyshapelib-0.3 shapelib/pyshapelib/
cd shapelib
make
cd pyshapelib
python setup.py build
sudo python setup.py install
cd ../../
# The next line is really important if you want python to think this folder has python scripts
touch shapelib/__init__.py
cp shapelib/pyshapelib/* shapelib/

This seamed to get over the original problem

Then you need to ensure you input file has no trailing lines

And now to work out why I'm getting

Traceback (most recent call last):
  File "makeShapeColoured.py", line 349, in <module>
    result = voronoi.computeVoronoiDiagram(pts)
  File "/home/kev/osm/voronoi.py", line 746, in computeVoronoiDiagram
    voronoi(siteList,context)
  File "/home/kev/osm/voronoi.py", line 206, in voronoi
    edge = Edge.bisect(bot,newsite)
  File "/home/kev/osm/voronoi.py", line 404, in bisect
    newedge.a = dx/dy
ZeroDivisionError: float division


     
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstreetmap.org/pipermail/talk-gb/attachments/20100426/aa4df1b8/attachment.htm 

Reply | Threaded
Open this post in threaded view
|

Postcodes to Shapefile

Emilie Laffray
On 26 April 2010 15:13, Jerry Clough - OSM <sk53_osm at yahoo.co.uk> wrote:

> GRASS seems to have some kind of Voronoi algorithm, but I find its
> interface very non-intuitive, so have not found it to try it out. This may
> be worth pursuing as an alternative route.
>
> As an aside: I notice that virtually all postcode boundaries are obviously
> created in this way. For instance the Philips Street Atlas shows totally
> implausible boundaries along the River Trent in S. Notts. Given that
> postcode boundaries are ultimately determined by logistically sensible walks
> for postmen, in this case it's pretty safe to assume that the boundary is
> actually the river. What this means is that by applying a bit of local
> knowledge and the existing points it is possible to create better
> delineating zones in OSM than appear in current mapping. Whether this is a
> good thing to do, or not, I leave for others to decide.
>
>
I think that GEOS has now implemented both voronoi and Delaunay
triangulation algorithms. I am not quite though that they have a Python
Binding. Some part of GEOS is expose through the GDAL binding, but it is
incomplete.

Emilie Laffray
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openstreetmap.org/pipermail/talk-gb/attachments/20100426/4b72e980/attachment.htm 

Reply | Threaded
Open this post in threaded view
|

Re: Postcodes to Shapefile

Kev js1982
In reply to this post by randomjunk
Resurrecting an old thread I know but with the NSPD Open data also being available allowing Northern Ireland to be generated and having access to an otherwise idle 64bit server I've taken the opportunity to revisit this, and have successfully created the shapefiles (it only took the server 12 days to generate the blighters - it managed to generate Z16 tiles for the whole of Europe and Z18 for the British Isles in just 36 hours - meh!) but now have one more obstacle to overcome...

The Veroni thingy obviously generates the tiles so they butt up against one another which works perfectly here in the landlocked East Midlands, but goes somewhat wrong in coastal areas (Fig 1).

My thought here is that the "World Boundaries" shape file can be used to trim the coastal boundaries to be locked to land so that the map looks "nice" (i.e. postcodes don't end up in the sea save for a little overlap on beaches) - Indeed if you add the World Boundaries file to Quantum GIS and use the "Clip Tool" you end up with what visually looks correct ( Fig. 3) but if you then hide the World Boundaries file the problem becomes obvious (Fig 2.) 

What I want to know, is it possible to trim the postcode shapes so that nothing outside another set of shapes (i.e. the British Isles landmass) is included, but instead of leaving gaps the postcode shapes (e.g. FY3 1) are adjusted so that the line runs along the coast line?  i.e. I would be left with something visually the same as Fig. 3 but with the coastlines part of the NNXX-X shapefile layer, and more specifically the correct polygon (e.g. the FY3 1 polygon).

Kev


Fig 1 - Postcode areas in south west Lancashire and the north Wales coast (green = NNXX-X shapefile, blackline and dotted area uses the worldboundaries file)
http://kjs.me.uk/3rdparty/osm/SouthWestLancs-NNXX-X.png

Fig 2 - After using Quantum GIS's "Clip" tool - http://kjs.me.uk/3rdparty/osm/SouthWestLancs-NNXX-X_trimmed.png

Fig 3 - After adding the "World Boundaries" back on. - SouthWestLancs-NNXX-X_trimmed_withwb.png

Open Street Map data licenced under the Creative Commons Attribution-Share Alike 2.0 license by the OpenStreetMap project and its contributors. Maps contain Ordnance Survey OpenData © Crown copyright and database right 2010. Postcode data in Great Britain is provided by Code-Point Open which contains Royal Mail data © Royal Mail copyright and database right 2010. Postcode data in Northern Ireland is from the NSPD Open which contains National Statistics data © Crown copyright and database right 2010.

On Fri, Apr 30, 2010 at 12:30, Kev js1982 <[hidden email]> wrote:
Hi Dave,

Thanks for providing the shapefiles for download - they did the job nicely.

One thing I have noticed (which also afflicts random.dev.openstreetmap.org) is that a few postcode area/districts are missing - namely

FY2 - (North Shore) Blackpool, Lancs
PE11 - Spalding, Lincs
PL17 - Callington, Cornwall

Using the code point download (which I got via the MySociety mirror) shows that these postcodes do exist.

Also one of the AB ones (12 or 21 IIRC) for some reason includes parts of Éire, Spain, Portugal and atlantic; while HS includes Reykjavik

Just thought you'd like to know there is a possible error with the conversion process.

Nice work though - been after a postcode map for a while, and to go from an A5 diagram to full "google maps" goodness in one swoop is awesome!

Regards

Kev Swindells.


On Mon, Apr 26, 2010 at 5:29 PM, Dave Stubbs <[hidden email]> wrote:
On Mon, Apr 26, 2010 at 5:15 PM, Kev js1982 <[hidden email]> wrote:
> On Mon, Apr 26, 2010 at 5:00 PM, Kev js1982 <[hidden email]> wrote:
>>
>> Thanks for that Dave - really useful.
>>
>> One question though - which prj string/file do I need for these?
>>
>
> Answering my own question - looks to be Google Mercator.
>
> http://spatialreference.org/ref/sr-org/6627/
>
> Kev Swindells
>
>

Actually, for some obscure historical reason it's projected into
"+proj=merc" which is srs 3395.

Close to 900913, but not quite the same -- my mapnik stylesheet is
then set to reproject to google mercator for the tile generation.

Dave



_______________________________________________
Talk-GB mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/talk-gb