Address data for Miami Florida United States

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Address data for Miami Florida United States

mangokm40
This is my first post.  Be gentle, if I fail some etiquette test. :)

I was testing the "maps.me" application and noticed address search didn't work.  It turns out it was a data problem.  OSM doesn't have all the addresses for Miami (Dade county), Florida.  This surprised me because our county makes this information available to the public.

The main dataset I thought would be useful can be found here:
As a point feature class, I figure it could be lightweight and allow search for ANY address in the county.  The data can be downloaded in KML or SHP format.  The license link displays this:

"Miami-Dade County provides this data for use "as is". The areas depicted by this map/data are approximate, and are not accurate to surveying or engineering standards. The maps/data shown here are for illustration purposes only and are not suitable for site-specific decision making. Information found here should not be used for making financial or any other commitments. Miami-Dade County provides this information with the understanding that it is not guaranteed to be accurate, correct or complete and conclusions drawn from such information are the sole responsibility of the user. While every effort has been made to ensure the accuracy, correctness and timeliness of materials presented, Miami-Dade County assumes no responsibility for errors or omissions, even if Miami-Dade County is advised of the possibility of such damage."

Since we think of Miami as a "global" destination, I thought search address would be valuable to OSM.  Is this something that has
been explored?  Am I wasting my time?

If this is useful, I'll be happy to help.  If I posted to the wrong place, please point to the right place.


Mango, in rainy Miami    (_8'()



_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Rihards
On 2018.09.11. 19:13, mangokm40 wrote:

> This is my first post.  Be gentle, if I fail some etiquette test. :)
>
> I was testing the "maps.me" application and noticed address search
> didn't work.  It turns out it was a data problem.  OSM doesn't have all
> the addresses for Miami (Dade county), Florida.  This surprised me
> because our county makes this information available to the public.
>
> The main dataset I thought would be useful can be found here:
> https://gis-mdc.opendata.arcgis.com/datasets/address-with-condo
> As a point feature class, I figure it could be lightweight and allow
> search for ANY address in the county.  The data can be downloaded in KML
> or SHP format.  The license link displays this:
>
> "Miami-Dade County provides this data for use "as is". The areas
> depicted by this map/data are approximate, and are not accurate to
> surveying or engineering standards. The maps/data shown here are for
> illustration purposes only and are not suitable for site-specific
> decision making. Information found here should not be used for making
> financial or any other commitments. Miami-Dade County provides this
> information with the understanding that it is not guaranteed to be
> accurate, correct or complete and conclusions drawn from such
> information are the sole responsibility of the user. While every effort
> has been made to ensure the accuracy, correctness and timeliness of
> materials presented, Miami-Dade County assumes no responsibility for
> errors or omissions, even if Miami-Dade County is advised of the
> possibility of such damage."
>
> Since we think of Miami as a "global" destination, I thought search
> address would be valuable to OSM.  Is this something that has
> been explored?  Am I wasting my time?

Hi and thank you for such a thoughtful email - and for your attention to
detail.

It looks like what you have quoted is more like a liability clause than
a full licence. Do they have more on the licence? Or is it PD because of
higher regulation?

> If this is useful, I'll be happy to help.  If I posted to the wrong
> place, please point to the right place.
>
>
> Mango, in rainy Miami    (_8'()
--
 Rihards

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

mangokm40
OK, after speaking to some folks, I am here:

1.  Permission:
    There's an address on their website ([hidden email]). I asked
    them specifically about availability for openstreetmap.org. I got
    a response from the lady in charge (I know her) that said:

    "The data is freely available for download and use with the understanding that it is not guaranteed to be accurate, correct, or
     complete and conclusions drawn from such information are the sole responsibility of the user."

    I saved the email, in case it is needed.

2.  Which data:
    Originally, I thought a "point" layer for all addresses would be great
    because it would allow address search without the weight of a full polygon. I've been told OSM prefers address tags on buildings.
    Thus, it's 2 layers we would need:
    a) address with point representation:
    https://gis-mdc.opendata.arcgis.com/datasets/address-with-condo
    b) building footprint polygons:
    https://gis-mdc.opendata.arcgis.com/datasets/building-footprint-2d

3.  Things that need to be done, which I know how to do:
    The data needs to be "massaged".
    The address data includes "condo". Thus, it has an address for each unit in a building. That needs to be trimmed to just one street
    address per building.
    Address abbreviations need to be converted. (ie. 1234 NW 33rd Ct needs to become 1234 Northwest 33 Court).

4.  Things that need to be done, which I don't know how to do:
    ALMOST EVERYTHING. :)
    Some building footprints already exist and some of them have an address. There would need to be a way to determine duplication.
    I have AutoCad & ESRI experience, but know nothing of OSM data & tools.
    I don't know any local people that are doing this. Is there a way to
    find them?


Thanks to all who have already helped me.


(_8'()

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, 11 September 2018 12:22, Rihards [hidden email] wrote:


> On 2018.09.11. 19:13, mangokm40 wrote:
>
> > This is my first post.  Be gentle, if I fail some etiquette test. :)
> > I was testing the "maps.me" application and noticed address search
> > didn't work.  It turns out it was a data problem.  OSM doesn't have all
> > the addresses for Miami (Dade county), Florida.  This surprised me
> > because our county makes this information available to the public.
> > The main dataset I thought would be useful can be found here:
> > https://gis-mdc.opendata.arcgis.com/datasets/address-with-condo
> > As a point feature class, I figure it could be lightweight and allow
> > search for ANY address in the county.  The data can be downloaded in KML
> > or SHP format.  The license link displays this:
> > "Miami-Dade County provides this data for use "as is". The areas
> > depicted by this map/data are approximate, and are not accurate to
> > surveying or engineering standards. The maps/data shown here are for
> > illustration purposes only and are not suitable for site-specific
> > decision making. Information found here should not be used for making
> > financial or any other commitments. Miami-Dade County provides this
> > information with the understanding that it is not guaranteed to be
> > accurate, correct or complete and conclusions drawn from such
> > information are the sole responsibility of the user. While every effort
> > has been made to ensure the accuracy, correctness and timeliness of
> > materials presented, Miami-Dade County assumes no responsibility for
> > errors or omissions, even if Miami-Dade County is advised of the
> > possibility of such damage."
> > Since we think of Miami as a "global" destination, I thought search
> > address would be valuable to OSM.  Is this something that has
> > been explored?  Am I wasting my time?
>
> Hi and thank you for such a thoughtful email - and for your attention to
> detail.
> It looks like what you have quoted is more like a liability clause than
> a full licence. Do they have more on the licence? Or is it PD because of
> higher regulation?
>
> > If this is useful, I'll be happy to help.  If I posted to the wrong
> > place, please point to the right place.
> > Mango, in rainy Miami    (_8'()
>
> --
> Rihards
> Talk-us mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/talk-us

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Leif Rasmussen
In reply to this post by mangokm40
Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Levente Juhász
In reply to this post by mangokm40
Hey,

As far as licenses go, these datasets are public domain so no other action is needed. 

As for all the other things, we gave this project a try in 2016 and imported a large chunk of Large Buildings with address points back then. It was meant to be a community project where we aimed to engage local mappers in OSM activities, too. This other objective was not fully achieved, though.

Anyway, here are a few resources that you might find useful.

The Tasking Manager we used to coordinate follow ups and quality checks is currently offline (but I still have the database).

+1 An open access publication summarizing the finding, more from an academic point of view:
Juhász, Levente, and Hartwig H. Hochmair. "OSM Data Import as an Outreach Tool to Trigger Community Growth? A Case Study in Miami." ISPRS International Journal of Geo-Information 7.3 (2018): 113. http://www.mdpi.com/2220-9964/7/3/113 

Cheers,
Levente


On Wed, Sep 12, 2018 at 3:55 PM mangokm40 <[hidden email]> wrote:
OK, after speaking to some folks, I am here:

1.  Permission:
    There's an address on their website ([hidden email]). I asked
    them specifically about availability for openstreetmap.org. I got
    a response from the lady in charge (I know her) that said:

    "The data is freely available for download and use with the understanding that it is not guaranteed to be accurate, correct, or
     complete and conclusions drawn from such information are the sole responsibility of the user."

    I saved the email, in case it is needed.

2.  Which data:
    Originally, I thought a "point" layer for all addresses would be great
    because it would allow address search without the weight of a full polygon. I've been told OSM prefers address tags on buildings.
    Thus, it's 2 layers we would need:
    a) address with point representation:
    https://gis-mdc.opendata.arcgis.com/datasets/address-with-condo
    b) building footprint polygons:
    https://gis-mdc.opendata.arcgis.com/datasets/building-footprint-2d

3.  Things that need to be done, which I know how to do:
    The data needs to be "massaged".
    The address data includes "condo". Thus, it has an address for each unit in a building. That needs to be trimmed to just one street
    address per building.
    Address abbreviations need to be converted. (ie. 1234 NW 33rd Ct needs to become 1234 Northwest 33 Court).

4.  Things that need to be done, which I don't know how to do:
    ALMOST EVERYTHING. :)
    Some building footprints already exist and some of them have an address. There would need to be a way to determine duplication.
    I have AutoCad & ESRI experience, but know nothing of OSM data & tools.
    I don't know any local people that are doing this. Is there a way to
    find them?


Thanks to all who have already helped me.


(_8'()

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, 11 September 2018 12:22, Rihards [hidden email] wrote:


> On 2018.09.11. 19:13, mangokm40 wrote:
>
> > This is my first post.  Be gentle, if I fail some etiquette test. :)
> > I was testing the "maps.me" application and noticed address search
> > didn't work.  It turns out it was a data problem.  OSM doesn't have all
> > the addresses for Miami (Dade county), Florida.  This surprised me
> > because our county makes this information available to the public.
> > The main dataset I thought would be useful can be found here:
> > https://gis-mdc.opendata.arcgis.com/datasets/address-with-condo
> > As a point feature class, I figure it could be lightweight and allow
> > search for ANY address in the county.  The data can be downloaded in KML
> > or SHP format.  The license link displays this:
> > "Miami-Dade County provides this data for use "as is". The areas
> > depicted by this map/data are approximate, and are not accurate to
> > surveying or engineering standards. The maps/data shown here are for
> > illustration purposes only and are not suitable for site-specific
> > decision making. Information found here should not be used for making
> > financial or any other commitments. Miami-Dade County provides this
> > information with the understanding that it is not guaranteed to be
> > accurate, correct or complete and conclusions drawn from such
> > information are the sole responsibility of the user. While every effort
> > has been made to ensure the accuracy, correctness and timeliness of
> > materials presented, Miami-Dade County assumes no responsibility for
> > errors or omissions, even if Miami-Dade County is advised of the
> > possibility of such damage."
> > Since we think of Miami as a "global" destination, I thought search
> > address would be valuable to OSM.  Is this something that has
> > been explored?  Am I wasting my time?
>
> Hi and thank you for such a thoughtful email - and for your attention to
> detail.
> It looks like what you have quoted is more like a liability clause than
> a full licence. Do they have more on the licence? Or is it PD because of
> higher regulation?
>
> > If this is useful, I'll be happy to help.  If I posted to the wrong
> > place, please point to the right place.
> > Mango, in rainy Miami    (_8'()
>
> --
> Rihards
> Talk-us mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/talk-us

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
--

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Brian May
In reply to this post by Leif Rasmussen
I would like to help as well. I've been mapping in Florida for many moons, live in SE FL, and would like to get involved in imports. I think we should consider merging the addresses with the Microsoft buildings. As Leif mentioned, buildings from Miami-Dade county have been imported, but they are incomplete.

Some of the condo buildings have over 100 units. When searching for an address, IMHO, unit is not important when dealing with a tall building. My vote is to not consider units because it just clutters the database.

Data in FL is public domain, unless exempted by FL Statutes. Refer to Chapter 119 - www.leg.state.fl.us/Statutes/index.cfm?App_mode=Display_Statute&URL=0100-0199/0119/0119.html

I've also been thinking about how to get more POIs into OSM in bulk utilizing existing government data that "theoretically" should be pretty accurate. For eample, Florida requires every business submit a tangible tax return. What this means is every single business in FL has a record in the tangible tax file that includes a site address, NAICS code, owner name, etc. Some have parcel IDs as well.  At the very least, we could map NAICS codes to OSM feature types like amenity=restaurant. In addition, I have all parcels for the state of Florida which include land use types. This could be used to add land use info to buildings as well, if its residential or not a business listed in the tangible file. We could go even further by referencing other GIS layers available from Miami-Dade like schools, hospitals, fire stations, etc. Basically, look into utilizing more data sets and some overlay queries to add to the known information about each building. A first step may be to merge all the relevant various Miami-Dade point layers into one master POI data set.

So just thinking out loud here. I've been a manual mapper for years, but focused mostly on road improvements, water, and a few other basic things. But I'm a GIS guy and have been wanting to put the power of GIS and existing GIS data to use for a while to go beyond the "basic" import for POIs and buildings.

I know there's a lot of existing OSM data to contend with and this should be a two step process, e.g. don't blindly place data on top of data, but rather first pass is greenfield where no buildings exist and second pass is manually conflate to existing data.

Brian aka grouper

On 9/12/2018 4:17 PM, Leif Rasmussen wrote:
Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen


_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us


_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

mangokm40
In reply to this post by Leif Rasmussen
Mr. Rasmussen,

Thanks for the offer!  I definitely need help. :)

I looked, and failed to find, the layer without unit#.  That's what I thought would be preferred.  Thanks
for spotting it.  I don't see why we would search for a specific unit on a map.  When I navigate, I would
just want directions to "1234 NW 33rd Ct", not "1234 NW 33rd Ct Apt 6".  If you know
of a good reason for the unit #s, let me know.  It doesn't matter to me, since I don't need navigation in
Miami.  :D
If you think 600k points is big, imagine the building footprints. :)  It's available, if required.  Heck, they
even make a 3d building layer available.  But I don't think we'll enjoy the size.

I knew the license is not a problem.  However, I saved that email just in case it comes up.

I'm going to read the info Mr. Juhász provided.  Unfortunately, I'm way behind here.  
Also, I replied to all b/c I think that's what I'm supposed to do.  However, I don't want to 'bug' people on
the list.  Hopefully, someone will let me know if this needs to go off-list.


(_8'()


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 12, 2018 4:17 PM, Leif Rasmussen <[hidden email]> wrote:

Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen



(_8'()

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 12, 2018 4:17 PM, Leif Rasmussen <[hidden email]> wrote:

Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen


_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Leif Rasmussen
Update on the address data:
I tried transforming the small data file (550,000 addresses) to OSM format, but my computer ran out of memory.  I will try using a more powerful computer later.  The transformation worked perfectly on a smaller file of 8000 addresses in Miami Beach, however.  I uploaded that sample file to a google drive folder.  Simply download the file and drag and drop it into JOSM to view that addresses.  The address data source only had "addr:city" for some of the addresses, so that tag will have to be added manually based on postcode later.
Thanks,
Leif Rasmussen

Also, the addresses with numbered street names (4th Street) have not been expanded (to eg. "Fourth Street").  The roads in OSM currently have the numbered versions (4th Street), so I will just leave the addresses like they are now.

On Thu, Sep 13, 2018 at 10:42 AM mangokm40 <[hidden email]> wrote:
Mr. Rasmussen,

Thanks for the offer!  I definitely need help. :)

I looked, and failed to find, the layer without unit#.  That's what I thought would be preferred.  Thanks
for spotting it.  I don't see why we would search for a specific unit on a map.  When I navigate, I would
just want directions to "1234 NW 33rd Ct", not "1234 NW 33rd Ct Apt 6".  If you know
of a good reason for the unit #s, let me know.  It doesn't matter to me, since I don't need navigation in
Miami.  :D
If you think 600k points is big, imagine the building footprints. :)  It's available, if required.  Heck, they
even make a 3d building layer available.  But I don't think we'll enjoy the size.

I knew the license is not a problem.  However, I saved that email just in case it comes up.

I'm going to read the info Mr. Juhász provided.  Unfortunately, I'm way behind here.  
Also, I replied to all b/c I think that's what I'm supposed to do.  However, I don't want to 'bug' people on
the list.  Hopefully, someone will let me know if this needs to go off-list.


(_8'()


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 12, 2018 4:17 PM, Leif Rasmussen <[hidden email]> wrote:

Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen



(_8'()

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 12, 2018 4:17 PM, Leif Rasmussen <[hidden email]> wrote:

Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen


_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Levente Juhász
Great stuff, Leif! I will catch up with this project over the weekend and will provide some input.

I have access to a few workstations with a bunch of memory so I can help out with converting/processing the input data if needed. Just let me know. I don't think of the size of these datasets as an issue in terms of processing (not even the buildings). As you said, splitting them up into smaller chunks or using a totally different toolset would also work.

Finding enough people to review and put the data on the map will be a bigger issue in Miami, imho. You guys are planning to implement an entirely manual approach, correct?

Cheers,
Levente



On Fri, Sep 14, 2018, 4:06 PM Leif Rasmussen <[hidden email]> wrote:
Update on the address data:
I tried transforming the small data file (550,000 addresses) to OSM format, but my computer ran out of memory.  I will try using a more powerful computer later.  The transformation worked perfectly on a smaller file of 8000 addresses in Miami Beach, however.  I uploaded that sample file to a google drive folder.  Simply download the file and drag and drop it into JOSM to view that addresses.  The address data source only had "addr:city" for some of the addresses, so that tag will have to be added manually based on postcode later.
Thanks,
Leif Rasmussen

Also, the addresses with numbered street names (4th Street) have not been expanded (to eg. "Fourth Street").  The roads in OSM currently have the numbered versions (4th Street), so I will just leave the addresses like they are now.

On Thu, Sep 13, 2018 at 10:42 AM mangokm40 <[hidden email]> wrote:
Mr. Rasmussen,

Thanks for the offer!  I definitely need help. :)

I looked, and failed to find, the layer without unit#.  That's what I thought would be preferred.  Thanks
for spotting it.  I don't see why we would search for a specific unit on a map.  When I navigate, I would
just want directions to "1234 NW 33rd Ct", not "1234 NW 33rd Ct Apt 6".  If you know
of a good reason for the unit #s, let me know.  It doesn't matter to me, since I don't need navigation in
Miami.  :D
If you think 600k points is big, imagine the building footprints. :)  It's available, if required.  Heck, they
even make a 3d building layer available.  But I don't think we'll enjoy the size.

I knew the license is not a problem.  However, I saved that email just in case it comes up.

I'm going to read the info Mr. Juhász provided.  Unfortunately, I'm way behind here.  
Also, I replied to all b/c I think that's what I'm supposed to do.  However, I don't want to 'bug' people on
the list.  Hopefully, someone will let me know if this needs to go off-list.


(_8'()


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 12, 2018 4:17 PM, Leif Rasmussen <[hidden email]> wrote:

Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen



(_8'()

Sent with ProtonMail Secure Email.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, September 12, 2018 4:17 PM, Leif Rasmussen <[hidden email]> wrote:

Hi Mango,
I have quite a lot of experience with address imports, and would love to help with Miami.  I have visited Miami several times, and have grown a liking for it.  Adding addresses there would be a real pleasure.
There appears to be two address data sets - one with "addr:unit", and one without.  The one with "addr:unit" addresses has 1,166,445 points, and the one without has 586,171 points.  Both of these should be considered.  I would suggest importing the one with condos, or "addr:unit" features if the quality is good.  Otherwise, I think that the dataset without addr:unit should be imported.
Also, the license seems OK.  According to the Miami-Dade County Buildings Import, the license is public domain, which they claim is true of all government produced data in Florida.
The only issue I see with the data is the size.  My laptop took 5 minutes to open the address points (including addr:unit, so 1,166,445 nodes) and more than 20 minutes to edit a single key.  This could be worked around, though, by splitting up the data.
I created a wiki page for the import, which is a step of the Import Guidelines. Sending a proposal to the local community and imports mailing list will also be needed.
I hope that this import will end up working out!
Leif Rasmussen

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
--

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Leif Rasmussen
In reply to this post by mangokm40
Another update on the address data:
I managed to do the address data transformation by splitting the data up into five files.  The data is now in ready to upload OSM format (except for data errors).  All duplicates of existing addresses in the OSM database have automatically been removed, leaving only missing addresses in the dataset.  This means that the data could be uploaded to the OSM database without creating duplicate addresses.
The data is available here:
I split it up into five manageable files with roughly 100,000 points in each.
The data itself has some minor issues.
1) Many "addr:city" tags are missing.  These can be added in manually before upload by selecting all addresses with a certain postcode and adding the city to them.  In the US, postcodes usually only have one city associated with them, so adding the missing addr:city tags is much easier because of this.
2) Some other tags are missing from about 50 addresses.  They don't have "addr:street", "addr:postcode", or "addr:city".  Just "addr:housenumber" and "addr:state".  
Other than that, the data looks great!  
I will fix up the wiki page and contact the imports mailing list sometime this weekend if I can.

Levente Juhász,
Manually adding the addresses would take way too long.  Instead, a tasking manager project should be created for organizing address upload.  If the uploader wants, they can merge the data with existing features such as buildings, but this is not required.  An upload without merging would simply add all missing addresses around the existing ones.  I was thinking that the tasks be about 5,000 - 10,000 addresses each.  This should allow for quick and easy upload, even if Miami does not have very many contributors interested in helping.
Thanks,
Leif Rasmussen

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

idnwys
In reply to this post by mangokm40

I would just like to point out two things.

 

addr:unit – I get there are a lot of tall buildings where this isn’t useful in Miami.  It is useful for large footprint buildings to know what entrance to park by.  Sometimes around here (unsure of in Miami) one address will even serve multiple buildings.  For instance at a hospital there may be 3 buildings containing doctors offices with the same address.

 

zip codes – Zip codes are USPS routing codes.  They do not align to cities and may even cross over each other often.  USPS only uses the city field on mail as a backup to zip codes.  They fudge the cities intentionally to make it easier on their sorting machines.  I’d be cautious when getting city data from USPS based on zip codes.


_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us
Reply | Threaded
Open this post in threaded view
|

Re: Address data for Miami Florida United States

Mark Wagner
On Mon, 17 Sep 2018 02:13:46 -0500
Aaron Forsythe <[hidden email]> wrote:

> zip codes – Zip codes are USPS routing codes.  They do not align to
> cities and may even cross over each other often.  USPS only uses the
> city field on mail as a backup to zip codes.  They fudge the cities
> intentionally to make it easier on their sorting machines.  I’d be
> cautious when getting city data from USPS based on zip codes.

"City" on a USPS address isn't the city, and never has been.  It's the
name of the sorting office.  This usually corresponds to the name of
the city (and for places with only one post office, the name of the
post office), but not always.  For unincorporated areas, it's usually
the same as the name of the nearest incorporated area, but not always:
unincorporated areas can still get post offices.

Postal Zip codes are point clouds, not areas (and Zip+4 codes even more
so).  Census zip codes are areas, but they've only got a rough
correspondence to the postal codes.

--
Mark

_______________________________________________
Talk-us mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk-us