How to save Overpass query results to a GeoJSON file with Python?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading its documentation I came up with the following code:


import overpy

API = overpy.Overpass()

# Fetch highways within Granollers' city.

result = API.query("""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?



Carlos Cámara-Menoyo



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

Mateusz Konieczny-3



Mar 11, 2019, 11:18 AM by [hidden email]:
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading its documentation I came up with the following code:


import overpy

API = overpy.Overpass()

# Fetch highways within Granollers' city.

result = API.query("""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?

There are two parts here

(1) using overpass itself
(2) writing python script

That are separate (you can call Overpass API from any program, not just python)

"I have just started to use Python" - note that it means learning two separate languages at once:

(1) Python itself
(2) Overpass query syntax

PS In case that Overpass is new for you my page may be of some use:

https://mapsaregreat.com/geographic-data-mining-and-visualisation-for-beginners/overpass-turbo-tutorial.html

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
Thanks for your reply, Mateusz.

I know (and use) Overpass turbo. However, what I want to do is to store a query and execute it in order to update results and store them as a file for further manipulation, so I can easily update results without the need to accessing to the interface, generate the query, run it and then download the results. I would like to achieve same functionallity but just by executing a file (which, additionally, may have more than one query).

So far, I found overpy, which seems to work fine in terms of querying but has two problems for me:
  1. It seems to be unmaintained (last commit dates from April 2017)
  2. I didn't succeed in storing the result into a geospatial file (or a geopandas data frame) (visit this jupyter lab with my current achievements: https://github.com/ccamara/osm-python/blob/master/overpy-demo.ipynb)
I also have just found Overpass API wrapper, but I still have to explore it and understand how does it work and how can I tame it to my needs.

You propose a third way, which is using overpass and writing a script. It is not clear to me what do you mean with that, since I thought that that was precisely what I was attempting. Could you be more specific, please? Do you have other insight?

Regards,
Carlos


Carlos Cámara-Menoyo


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday 11 March de 2019 a les 14:15, Mateusz Konieczny <[hidden email]> wrote:




Mar 11, 2019, 11:18 AM by [hidden email]:
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading its documentation I came up with the following code:


import overpy

API = overpy.Overpass()

# Fetch highways within Granollers' city.

result = API.query("""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?

There are two parts here

(1) using overpass itself
(2) writing python script

That are separate (you can call Overpass API from any program, not just python)

"I have just started to use Python" - note that it means learning two separate languages at once:

(1) Python itself
(2) Overpass query syntax

PS In case that Overpass is new for you my page may be of some use:

https://mapsaregreat.com/geographic-data-mining-and-visualisation-for-beginners/overpass-turbo-tutorial.html


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

Bryce Jasmer
I use the overpass library (I think it's the one you mentioned). The resulting dictionary can be coerced to a string and written out and it should be valid geojson.
#!/usr/bin/python3
import overpass
api = overpass.API()
result = api.get('area[name="Granollers"][admin_level=8];(nwr["highway"](area);relation["highway"](area););')
with open('results.geojson', 'w') as fd:
    fd.write(str(result))

On Mon, Mar 11, 2019 at 8:04 AM Carlos Cámara Menoyo via talk <[hidden email]> wrote:
Thanks for your reply, Mateusz.

I know (and use) Overpass turbo. However, what I want to do is to store a query and execute it in order to update results and store them as a file for further manipulation, so I can easily update results without the need to accessing to the interface, generate the query, run it and then download the results. I would like to achieve same functionallity but just by executing a file (which, additionally, may have more than one query).

So far, I found overpy, which seems to work fine in terms of querying but has two problems for me:
  1. It seems to be unmaintained (last commit dates from April 2017)
  2. I didn't succeed in storing the result into a geospatial file (or a geopandas data frame) (visit this jupyter lab with my current achievements: https://github.com/ccamara/osm-python/blob/master/overpy-demo.ipynb)
I also have just found Overpass API wrapper, but I still have to explore it and understand how does it work and how can I tame it to my needs.

You propose a third way, which is using overpass and writing a script. It is not clear to me what do you mean with that, since I thought that that was precisely what I was attempting. Could you be more specific, please? Do you have other insight?

Regards,
Carlos


Carlos Cámara-Menoyo


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday 11 March de 2019 a les 14:15, Mateusz Konieczny <[hidden email]> wrote:




Mar 11, 2019, 11:18 AM by [hidden email]:
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading its documentation I came up with the following code:


import overpy

API = overpy.Overpass()

# Fetch highways within Granollers' city.

result = API.query("""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?

There are two parts here

(1) using overpass itself
(2) writing python script

That are separate (you can call Overpass API from any program, not just python)

"I have just started to use Python" - note that it means learning two separate languages at once:

(1) Python itself
(2) Overpass query syntax

PS In case that Overpass is new for you my page may be of some use:


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

Mateusz Konieczny-3
In reply to this post by General Discussion mailing list



Mar 11, 2019, 4:04 PM by [hidden email]:
Thanks for your reply, Mateusz.

I know (and use) Overpass turbo. However, what I want to do is to store a query and execute it in order to update results and store them as a file for further manipulation
OK, in that case you probably don;t need materials that I linked
So far, I found overpy, which seems to work fine in terms of querying but has two problems for me:
  1. It seems to be unmaintained (last commit dates from April 2017)
Unfortunately there is no really good library for Overpass access - I was looking for one
some time ago and found nothing that would work well (maintained, support for
queries longer than URL limit, support for filesystem caching).

You propose a third way, which is using overpass and writing a script. It is not clear to me what do you mean with that, since I thought that that was precisely what I was attempting.
Yes - I was not proposing any new way.


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
In reply to this post by General Discussion mailing list
Unsure, but would OSMtoGeojson be of use?
https://github.com/tyrasd/osmtogeojson

DaveF

On 11/03/2019 10:18, Carlos Cámara Menoyo via talk wrote:
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading [its documentation](https://python-overpy.readthedocs.io/en/latest/example.html) I came up with the following code:

import

overpy

API

=

overpy

.

Overpass

()

# Fetch highways within Granollers' city.

result

=

API

.

query

(

"""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """

)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?

Carlos Cámara-Menoyo
https://carloscamara.es


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
Thank you for the clarification, Matheus and for the link, Dave (I am afraid that's not what I was looking for, as it still requires to download data on osm format).

I tried Bryce's code, which uses overpass api python wrapper (https://github.com/mvexel/overpass-api-python-wrapper), and it just works fine!

Unfortunately, I have to admit that I don't wholly understand it (I have just landed in Python's world). Would you be so kind to explain the following?:
  1. Can I add the query in more than a single line? (the provided one is quite simple and still it is long) If so, how? (from what I am seeing Python is very strict dealing with new lines)
  2. what's going on after the result? I understand there's where the magic of creating a gjson file happens, but can't understand much about it.
Regards and thank you again to everyone!

Carlos Cámara-Menoyo


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday 11 March de 2019 a les 18:34, Dave F via talk <[hidden email]> wrote:

Unsure, but would OSMtoGeojson be of use?

DaveF


On 11/03/2019 10:18, Carlos Cámara Menoyo via talk wrote:
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading [its documentation](https://python-overpy.readthedocs.io/en/latest/example.html) I came up with the following code:

import

overpy

API

=

overpy

.

Overpass

()

# Fetch highways within Granollers' city.

result

=

API

.

query

(

"""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """

)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?

Carlos Cámara-Menoyo
https://carloscamara.es


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

Mateusz Konieczny-3



Mar 11, 2019, 6:54 PM by [hidden email]:
Can I add the query in more than a single line? (the provided one is quite simple and still it is long) If so, how? (from what I am seeing Python is very strict dealing with new lines)
This is not the best place for Python-specific questions.

Fortunately for beginners most simple questions are already asked and answered.

In this case important key words are
- multiline
- string
- Python

Searching for it, see for example
will find among other answers for example
with answer
https://stackoverflow.com/a/10660443/4130619

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

Martijn van Exel-3
In reply to this post by General Discussion mailing list
Carlos — I am the maintainer of that library Bryce provided the sample code for. I am happy to answer any specific questions about it. However, I’d advise you take some time to study python first. Without a basic understanding of the language you are programming in, you are unlikely to be successful at solving problems using that language. There are many excellent, free python learning resources available.

Martijn

On Mar 11, 2019, at 11:54 AM, Carlos Cámara Menoyo via talk <[hidden email]> wrote:

Thank you for the clarification, Matheus and for the link, Dave (I am afraid that's not what I was looking for, as it still requires to download data on osm format).

I tried Bryce's code, which uses overpass api python wrapper (https://github.com/mvexel/overpass-api-python-wrapper), and it just works fine!

Unfortunately, I have to admit that I don't wholly understand it (I have just landed in Python's world). Would you be so kind to explain the following?:
  1. Can I add the query in more than a single line? (the provided one is quite simple and still it is long) If so, how? (from what I am seeing Python is very strict dealing with new lines)
  2. what's going on after the result? I understand there's where the magic of creating a gjson file happens, but can't understand much about it.
Regards and thank you again to everyone!

Carlos Cámara-Menoyo


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday 11 March de 2019 a les 18:34, Dave F via talk <[hidden email]> wrote:

Unsure, but would OSMtoGeojson be of use?

DaveF


On 11/03/2019 10:18, Carlos Cámara Menoyo via talk wrote:
Dear colleagues,

I have just started to use Python and I would like to make a query to Overpass and store the results in a geospatial format (e.g. GeoJSON) so I can programatically update the data.

As far as I know, there is a library called overpy that should be what I am looking for. After reading [its documentation](https://python-overpy.readthedocs.io/en/latest/example.html) I came up with the following code:

import

overpy

API

=

overpy

.

Overpass

()

# Fetch highways within Granollers' city.

result

=

API

.

query

(

"""
    area[name="Granollers"][admin_level=8];
    // query part for: “highway=*”
    (way["highway"](area);
    relation["highway"](area);
    );
    // print results
    out body;
    """

)

The thing is that I am not familiar at all with Python and I don't know how to store result as a GeoJSON file (Honestly, I don't even know what kind of thing result is -I tried with type(result) and I get overpy.Result as an output, which doesn't mean much to my poor knowledge). I have been reading overpy documentation but I haven't been able to figure it out.

Can anyone give me a clue with this?

Carlos Cámara-Menoyo
https://carloscamara.es


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
In reply to this post by General Discussion mailing list


On 11/03/2019 17:54, Carlos Cámara Menoyo wrote:
> Thank you for the clarification, Matheus and for the link, Dave (I am afraid that's not what I was looking for, as it still requires to download data on osm format).

You can run the api from a command line or script (such as python).

Paste this into your browser & look at the file it downloads:

http://overpass-api.de/api/interpreter?data=area[name="Granollers"][admin_level=8];(way["highway"](area);relation["highway"](area););out
body;

This can be loaded from within python (urllib?)



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
Maybe adapt this answer:

https://gis.stackexchange.com/questions/115733/converting-json-to-geojson-or-csv/115736#115736



On 11/03/2019 20:00, Dave F via talk wrote:

>
>
> On 11/03/2019 17:54, Carlos Cámara Menoyo wrote:
>> Thank you for the clarification, Matheus and for the link, Dave (I am
>> afraid that's not what I was looking for, as it still requires to
>> download data on osm format).
>
> You can run the api from a command line or script (such as python).
>
> Paste this into your browser & look at the file it downloads:
>
> http://overpass-api.de/api/interpreter?data=area[name="Granollers"][admin_level=8];(way["highway"](area);relation["highway"](area););out
> body;
>
> This can be loaded from within python (urllib?)
>
>
>
> _______________________________________________
> talk mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/talk


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: How to save Overpass query results to a GeoJSON file with Python?

General Discussion mailing list
Thank you everyone for your help.

With Bryce's example + Martijn Library (thanks for developing it!) + some further research I am sure I will be able to achieve what I was looking for!

Best,


Carlos Cámara-Menoyo
https://carloscamara.es

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday 11 March de 2019 a les 21:45, Dave F <[hidden email]> wrote:

> Maybe adapt this answer:
>
> https://gis.stackexchange.com/questions/115733/converting-json-to-geojson-or-csv/115736#115736
>
> On 11/03/2019 20:00, Dave F via talk wrote:
>
> > On 11/03/2019 17:54, Carlos Cámara Menoyo wrote:
> >
> > > Thank you for the clarification, Matheus and for the link, Dave (I am
> > > afraid that's not what I was looking for, as it still requires to
> > > download data on osm format).
> >
> > You can run the api from a command line or script (such as python).
> > Paste this into your browser & look at the file it downloads:
> > http://overpass-api.de/api/interpreter?data=area[name="Granollers"][admin_level=8];(way"highway";relation"highway";);out
> > body;
> > This can be loaded from within python (urllib?)
> >
> > talk mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/talk



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk