Python XML Parser on Windows

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Python XML Parser on Windows

Spencer Gardner
I'm sort of resurrecting an old thread I started last year. I find Martijn's Overpass Python library to be very useful for small queries but I frequently run into usage limits. I'd like to query an XML download directly to save bandwidth and reduce consumption of Overpass resources, but I can't find any suitable Python libraries that are WIndows-ready. Is there something I'm missing? Has anyone else found a solution that works?

By way of example, Geoff Boeing's excellent osmnx package appears to use a home-grown OSM parser due to lack of a cross-platform option.

Thanks for any help

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Python XML Parser on Windows

Jochen123
Hi!

PyOsmium https://osmcode.org/pyosmium/ should work on Windows.

Jochen

On Tue, Aug 06, 2019 at 09:20:39PM -0700, Spencer Gardner wrote:

> Date: Tue, 6 Aug 2019 21:20:39 -0700
> From: Spencer Gardner <[hidden email]>
> To: [hidden email]
> Subject: [OSM-dev] Python XML Parser on Windows
>
> I'm sort of resurrecting an old thread I started last year. I find
> Martijn's Overpass Python library to be very useful for small queries but I
> frequently run into usage limits. I'd like to query an XML download
> directly to save bandwidth and reduce consumption of Overpass resources,
> but I can't find any suitable Python libraries that are WIndows-ready. Is
> there something I'm missing? Has anyone else found a solution that works?
>
> By way of example, Geoff Boeing's excellent osmnx package appears to use a
> home-grown OSM parser due to lack of a cross-platform option.
> https://github.com/gboeing/osmnx/blob/eebbca0ed1d59a6dfe07c90493d54c0beab45145/osmnx/utils.py#L1181
>
> Thanks for any help

> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/dev


--
Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Python XML Parser on Windows

Frank Steggink
In reply to this post by Spencer Gardner
Hi Spencer,

I'd definitely recommend lxml: https://lxml.de/. One of the benefits of
lxml is that it is cross platform.
You can find Windows binaries for lxml and many other Python libraries
here: https://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

Regards,

Frank

On 07-08-2019 06:20, Spencer Gardner wrote:

> I'm sort of resurrecting an old thread I started last year. I find
> Martijn's Overpass Python library to be very useful for small queries
> but I frequently run into usage limits. I'd like to query an XML
> download directly to save bandwidth and reduce consumption of Overpass
> resources, but I can't find any suitable Python libraries that are
> WIndows-ready. Is there something I'm missing? Has anyone else found a
> solution that works?
>
> By way of example, Geoff Boeing's excellent osmnx package appears to
> use a home-grown OSM parser due to lack of a cross-platform option.
> https://github.com/gboeing/osmnx/blob/eebbca0ed1d59a6dfe07c90493d54c0beab45145/osmnx/utils.py#L1181
>
> Thanks for any help
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/dev



_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Python XML Parser on Windows

Sarah Hoffmann
In reply to this post by Jochen123
On Wed, Aug 07, 2019 at 07:16:07AM +0200, Jochen Topf wrote:
> Hi!
>
> PyOsmium https://osmcode.org/pyosmium/ should work on Windows.

Just stay away from bz2 compressed xml. That's known to be
broken on Windows. Uncompressed xml should be fine though.

Sarah

> On Tue, Aug 06, 2019 at 09:20:39PM -0700, Spencer Gardner wrote:
> > Date: Tue, 6 Aug 2019 21:20:39 -0700
> > From: Spencer Gardner <[hidden email]>
> > To: [hidden email]
> > Subject: [OSM-dev] Python XML Parser on Windows
> >
> > I'm sort of resurrecting an old thread I started last year. I find
> > Martijn's Overpass Python library to be very useful for small queries but I
> > frequently run into usage limits. I'd like to query an XML download
> > directly to save bandwidth and reduce consumption of Overpass resources,
> > but I can't find any suitable Python libraries that are WIndows-ready. Is
> > there something I'm missing? Has anyone else found a solution that works?
> >
> > By way of example, Geoff Boeing's excellent osmnx package appears to use a
> > home-grown OSM parser due to lack of a cross-platform option.
> > https://github.com/gboeing/osmnx/blob/eebbca0ed1d59a6dfe07c90493d54c0beab45145/osmnx/utils.py#L1181
> >
> > Thanks for any help
>
> > _______________________________________________
> > dev mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/dev
>
>
> --
> Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Python XML Parser on Windows

Jochen123
On Wed, Aug 07, 2019 at 08:30:07AM +0100, Sarah Hoffmann wrote:
> On Wed, Aug 07, 2019 at 07:16:07AM +0200, Jochen Topf wrote:
> > Hi!
> >
> > PyOsmium https://osmcode.org/pyosmium/ should work on Windows.
>
> Just stay away from bz2 compressed xml. That's known to be
> broken on Windows. Uncompressed xml should be fine though.

Or, even better, use the PBF file format which should be much faster
than XML.

Jochen
--
Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Python XML Parser on Windows

Spencer Gardner
Thanks. I'm aware of PyOsmium, which appears to work on Windows but not without installing a number of additional packages related to the libosmium. I may go that route in the absence of easier alternatives, but installation of libosmium doesn't seem very straightforward on Windows. Or perhaps the problem is the documentation doesn't provide much information on how to install on Windows short of building from source, which is a nonstarter for most users. What I'm really hoping for is an isolated Python solution that can be installed via a simple pip command.

Have I misunderstood something about PyOsmium and libosmium?

On Wed, Aug 7, 2019 at 1:03 AM Jochen Topf <[hidden email]> wrote:
On Wed, Aug 07, 2019 at 08:30:07AM +0100, Sarah Hoffmann wrote:
> On Wed, Aug 07, 2019 at 07:16:07AM +0200, Jochen Topf wrote:
> > Hi!
> >
> > PyOsmium https://osmcode.org/pyosmium/ should work on Windows.
>
> Just stay away from bz2 compressed xml. That's known to be
> broken on Windows. Uncompressed xml should be fine though.

Or, even better, use the PBF file format which should be much faster
than XML.

Jochen
--
Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Python XML Parser on Windows

Sarah Hoffmann
On Wed, Aug 07, 2019 at 06:39:14AM -0700, Spencer Gardner wrote:
> Thanks. I'm aware of PyOsmium, which appears to work on Windows but not
> without installing a number of additional packages related to the
> libosmium. I may go that route in the absence of easier alternatives, but
> installation of libosmium doesn't seem very straightforward on Windows. Or
> perhaps the problem is the documentation doesn't provide much information
> on how to install on Windows short of building from source, which is a
> nonstarter for most users. What I'm really hoping for is an isolated Python
> solution that can be installed via a simple pip command.

We do provide precomiled binaries for Windows (64bit), so you
should be able to just 'pip install osmium' without building libosmium
yourself.

I admit that this is not well tested because we don't have a maintainer
with a Windows development platform (and experience with it). If you run
into trouble, feel free to file an issue and we see what we can do.

Sarah


>
> Have I misunderstood something about PyOsmium and libosmium?
>
> On Wed, Aug 7, 2019 at 1:03 AM Jochen Topf <[hidden email]> wrote:
>
> > On Wed, Aug 07, 2019 at 08:30:07AM +0100, Sarah Hoffmann wrote:
> > > On Wed, Aug 07, 2019 at 07:16:07AM +0200, Jochen Topf wrote:
> > > > Hi!
> > > >
> > > > PyOsmium https://osmcode.org/pyosmium/ should work on Windows.
> > >
> > > Just stay away from bz2 compressed xml. That's known to be
> > > broken on Windows. Uncompressed xml should be fine though.
> >
> > Or, even better, use the PBF file format which should be much faster
> > than XML.
> >
> > Jochen
> > --
> > Jochen Topf  [hidden email]  https://www.jochentopf.com/
> > +49-351-31778688
> >

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev