Victorian suburbs/localities relations possible bulk upload

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Victorian suburbs/localities relations possible bulk upload

Ewen Hill
Good evening,
  First time poster so be gentle. I have had the need to look at suburbs this week and noticed a number of issues with the osm data. 

I downloaded https://data.gov.au/dataset/vic-suburb-locality-boundaries-psma-administrative-boundaries and matched this to an overpass osm extract and then merged the data into this map (avail early May 2018 only). https://goo.gl/XCwLnj

The red lines are the PSMA whilst the dark green are the OSM. If you look at the map you can see
> New or osm localities not added or incorrectly adde
> Modified boundaries since the last upload
> Incorrectly attributed or missing boundaries
> Some minor river/creek alignment issues that don't need remedial work that much.

The issue is (apart from getting clearance although it is a CC By 4 copyright) is how would you reload the dataset given that existing boundaries like weather and fire districts use the existing ones and the changes are significant

My initial thoughts would be to upload these as none visible lines with the key details in notes fields so that over time localities can be updated but it would be a long term mess. An OSM note node would be added as a pointer but it is still a large process. 

Perhaps the best way is to add those missing localities, refine and then look at those areas that have a mismatch over 10%.

Another way would be to remove all Victorian relations that are admin_level=10 an then load the new data? 

Can anyone offer some guidance please. I have not performed a bulk upload but understand the mechanics and process required, it is more uploading a huge amount of data and how those in the future can quickly reload the entire state on another major suburb upheaval. 


TIA

_______________________________________________
Talk-au mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-au
Reply | Threaded
Open this post in threaded view
|

Re: Victorian suburbs/localities relations possible bulk upload

Andrew Harvey-3
Hi Ewen,

Nothing wrong with asking questions, hope my response doesn't come off too harshly! My thoughts below...

On 28 April 2018 at 19:30, Ewen Hill <[hidden email]> wrote:
I downloaded https://data.gov.au/dataset/vic-suburb-locality-boundaries-psma-administrative-boundaries and matched this to an overpass osm extract and then merged the data into this map (avail early May 2018 only). https://goo.gl/XCwLnj

The red lines are the PSMA whilst the dark green are the OSM. If you look at the map you can see
> New or osm localities not added or incorrectly adde
> Modified boundaries since the last upload
> Incorrectly attributed or missing boundaries

We should take caution here, just how do you know an admin boundary in OSM is incorrectly attributed? For example, I've seen a release of the PSMA Admin Boundaries where the suburb name didn't match signage on the ground, which shall prevail, so you can't always just overwrite what's in OSM currently.
 
> Some minor river/creek alignment issues that don't need remedial work that much.

I agree with this as I don't think OSM should strictly == the upstream admin boundaries. For example where the boundary is legally defined as a watercourse, it should be matching the watercourse we have in OSM which may be more accurate or more current than the upstream data.
  
...apart from getting clearance although it is a CC By 4 copyright...

We still need the OSMF CC BY 4.0 waiver for the reasons at https://blog.openstreetmap.org/2017/03/17/use-of-cc-by-data/ we don't yet have the waiver to include it in OSM, however just to let you know I have been trying in the last few months.

I sent off an email to the Department of the Prime Minister and Cabinet, who look after the publication of the PSMA Admin Boundaries on data.gov.au outlining why we ask for the waiver and why we'd like to be able to use it in OSM, on the 28th Feb, followed up on the 16th Mar to which they replied they are consulting on information before they provide a reply. I followed up again on the 13th Apr, and again just now.

So until we have that I don't believe we should be using this dataset at all in OpenStreetMap.

That said, if you're only interested in Victoria we do have the waiver https://wiki.openstreetmap.org/wiki/File:Vicmap_CCBYPermission_OSM_Final_Jan2018_Ltr.pdf to use the CC BY 4.0 licensed Vicmap Admin https://www2.delwp.vic.gov.au/maps/spatial-data/victorian-spatial-data#Vicmap_Admin-2. I'm open to a proposal investigating using it to update OpenStreetMap.
 
 
My initial thoughts would be to upload these as none visible lines with the key details in notes fields so that over time localities can be updated but it would be a long term mess. An OSM note node would be added as a pointer but it is still a large process. 

I think that would be too messy.
 
Perhaps the best way is to add those missing localities, refine and then look at those areas that have a mismatch over 10%.

From a high level, that sounds reasonable.
 
Another way would be to remove all Victorian relations that are admin_level=10 an then load the new data? 

I don't agree with that, we should respect folks who have contributed this data already and as much as possible keep the history. Given the admin boundaries are often glued to roads, rivers, coastlines, other relations, I don't think it will work to do a bulk load.
 
Can anyone offer some guidance please. I have not performed a bulk upload but understand the mechanics and process required, it is more uploading a huge amount of data and how those in the future can quickly reload the entire state on another major suburb upheaval. 

I think the only option is to:
1. Identify the major differences (as you've done and suggest)
2. Manually go over these differences and manually touch up the existing OSM data with the changes, being mindful of the local editors.


_______________________________________________
Talk-au mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-au
Reply | Threaded
Open this post in threaded view
|

Re: Victorian suburbs/localities relations possible bulk upload

cleary



I agree with most of what Andrew says. However I offer a couple of comments:

1. In regard to PSMA Admin Boundaries, it is my understanding that PSMA cannot grant a licence waiver as it is not the copyright holder.  PSMA has been licensed to use the data under a CC licence. However, as far as I am aware, any variation to permission must be obtained from the licence holders which are the state and terrritory authorities identified by PSMA.

2. In regard to administrative boundaries along waterways, it is my understanding that (in NSW at least) the boundary does not change even when the waterway does. Therefore over time, there can be significant diffferences between waterways and nearby boundaries. For this reason the admin boundary and the waterway should be mapped separately.  The most famous is the NSW/Vic state boundary which was originally aligned with the southern bank of the Murray but, even though the river has changed course in parts, the boundary remains the same. Someone may know if the situation is different in other states/territories.






On Sat, Apr 28, 2018, at 8:38 PM, Andrew Harvey wrote:
Hi Ewen,

Nothing wrong with asking questions, hope my response doesn't come off too harshly! My thoughts below...
On 28 April 2018 at 19:30, Ewen Hill <[hidden email]> wrote:
I downloaded https://data.gov.au/dataset/vic-suburb-locality-boundaries-psma-administrative-boundaries and matched this to an overpass osm extract and then merged the data into this map (avail early May 2018 only). https://goo.gl/XCwLnj

The red lines are the PSMA whilst the dark green are the OSM. If you look at the map you can see
> New or osm localities not added or incorrectly adde
> Modified boundaries since the last upload
> Incorrectly attributed or missing boundaries

We should take caution here, just how do you know an admin boundary in OSM is incorrectly attributed? For example, I've seen a release of the PSMA Admin Boundaries where the suburb name didn't match signage on the ground, which shall prevail, so you can't always just overwrite what's in OSM currently.
 
> Some minor river/creek alignment issues that don't need remedial work that much.

I agree with this as I don't think OSM should strictly == the upstream admin boundaries. For example where the boundary is legally defined as a watercourse, it should be matching the watercourse we have in OSM which may be more accurate or more current than the upstream data.
  
...apart from getting clearance although it is a CC By 4 copyright...

We still need the OSMF CC BY 4.0 waiver for the reasons at https://blog.openstreetmap.org/2017/03/17/use-of-cc-by-data/ we don't yet have the waiver to include it in OSM, however just to let you know I have been trying in the last few months.

I sent off an email to the Department of the Prime Minister and Cabinet, who look after the publication of the PSMA Admin Boundaries on data.gov.au outlining why we ask for the waiver and why we'd like to be able to use it in OSM, on the 28th Feb, followed up on the 16th Mar to which they replied they are consulting on information before they provide a reply. I followed up again on the 13th Apr, and again just now.

So until we have that I don't believe we should be using this dataset at all in OpenStreetMap.

That said, if you're only interested in Victoria we do have the waiver https://wiki.openstreetmap.org/wiki/File:Vicmap_CCBYPermission_OSM_Final_Jan2018_Ltr.pdf to use the CC BY 4.0 licensed Vicmap Admin https://www2.delwp.vic.gov.au/maps/spatial-data/victorian-spatial-data#Vicmap_Admin-2. I'm open to a proposal investigating using it to update OpenStreetMap.
 
 
My initial thoughts would be to upload these as none visible lines with the key details in notes fields so that over time localities can be updated but it would be a long term mess. An OSM note node would be added as a pointer but it is still a large process. 

I think that would be too messy.
 
Perhaps the best way is to add those missing localities, refine and then look at those areas that have a mismatch over 10%.

From a high level, that sounds reasonable.
 
Another way would be to remove all Victorian relations that are admin_level=10 an then load the new data? 

I don't agree with that, we should respect folks who have contributed this data already and as much as possible keep the history. Given the admin boundaries are often glued to roads, rivers, coastlines, other relations, I don't think it will work to do a bulk load.
 
Can anyone offer some guidance please. I have not performed a bulk upload but understand the mechanics and process required, it is more uploading a huge amount of data and how those in the future can quickly reload the entire state on another major suburb upheaval. 

I think the only option is to:
1. Identify the major differences (as you've done and suggest)
2. Manually go over these differences and manually touch up the existing OSM data with the changes, being mindful of the local editors.
_______________________________________________
Talk-au mailing list


_______________________________________________
Talk-au mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-au
Reply | Threaded
Open this post in threaded view
|

Re: Victorian suburbs/localities relations possible bulk upload

Andrew Harvey-3
On 28 April 2018 at 21:08, cleary <[hidden email]> wrote:
... PSMA has been licensed to use the data under a CC licence...

Are you sure? Honest question, because I don't know. I suspect this data is provided by the states to PSMA under a liberal license (less restrictive than CC BY) meaning they can do whatever they like, and in tern Department of Prime Minister and Cabinet has a liberal license from PSMA, If that's the case then PM&C can license how they see fit. Let's wait and see what PM&C say.
 
2. In regard to administrative boundaries along waterways, it is my understanding that (in NSW at least) the boundary does not change even when the waterway does. Therefore over time, there can be significant diffferences between waterways and nearby boundaries. For this reason the admin boundary and the waterway should be mapped separately.  The most famous is the NSW/Vic state boundary which was originally aligned with the southern bank of the Murray but, even though the river has changed course in parts, the boundary remains the same. Someone may know if the situation is different in other states/territories.

I wasn't sure what the laws actually were, so thanks for making the point!

_______________________________________________
Talk-au mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-au
Reply | Threaded
Open this post in threaded view
|

Re: Victorian suburbs/localities relations possible bulk upload

Ewen Hill
In reply to this post by cleary
Thank you Andrew and Cleary,
   As I mentioned, I wasn't concerned at this point in time with the
process, each element has been done time and time again from approval
sign-off to processing plan and backout. What I really wanted to discuss is
how to import such and I didn't really want to get into copyright tangent
just yet, that comes with time.

My single question is that if you have the PSMA available and the tools to
upload it all, in chunks, or nibbles, how would you do it so that the next
time PSMA data changed significantly, it would be a doddle for others to
update and it is not to burdensome on current editors?  

   With Central Melbourne, you could do this manually due to the minimal
changes however there are hours if not months of work to do this manually
out bush.

Thanks again.



--
Sent from: http://gis.19327.n8.nabble.com/Australia-f5416966.html

_______________________________________________
Talk-au mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-au
Reply | Threaded
Open this post in threaded view
|

Re: Victorian suburbs/localities relations possible bulk upload

cleary
In reply to this post by Andrew Harvey-3

I made contact with DPMC and posted some info last year - https://lists.openstreetmap.org/pipermail/talk-au/2017-March/011307.html    I did not receive any response to my further letter to DPMC and my information is based on an earlier informal communication with one of their staff. 

However I would be very happy if you can get a waiver signed.




On Sat, Apr 28, 2018, at 9:25 PM, Andrew Harvey wrote:
On 28 April 2018 at 21:08, cleary <[hidden email]> wrote:

... PSMA has been licensed to use the data under a CC licence...

Are you sure? Honest question, because I don't know. I suspect this data is provided by the states to PSMA under a liberal license (less restrictive than CC BY) meaning they can do whatever they like, and in tern Department of Prime Minister and Cabinet has a liberal license from PSMA, If that's the case then PM&C can license how they see fit. Let's wait and see what PM&C say.
 
2. In regard to administrative boundaries along waterways, it is my understanding that (in NSW at least) the boundary does not change even when the waterway does. Therefore over time, there can be significant diffferences between waterways and nearby boundaries. For this reason the admin boundary and the waterway should be mapped separately.  The most famous is the NSW/Vic state boundary which was originally aligned with the southern bank of the Murray but, even though the river has changed course in parts, the boundary remains the same. Someone may know if the situation is different in other states/territories.

I wasn't sure what the laws actually were, so thanks for making the point!
_______________________________________________
Talk-au mailing list


_______________________________________________
Talk-au mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-au