PBF output - prototype available for testing

[mmd]

Hi,

PBF is well known for its compactness and better efficiency. No wonder
it is also on the roadmap for the official API's map call (see [1].
That's reason enough to evaluate how PBF could also be integrated into
Overpass API.

Thanks to libosmium and Jochen's great support, I'm happy to present a
first (incomplete) prototype:

http://overpass-turbo.eu/s/jpK

Be sure to click on Export -> raw data directly from Overpass API to
avoid garbled output in overpass turbo.

When trying your own examples, be sure to always use `out meta;`, all
other modes are not implemented in this prototype.

I tested pbf output for areas of roughly 300km by 300km, yielding about
500MB in PBF file size. (in turbo use: Load -> Examples -> Map Call)

Along with PBF, libosmium also provides additional output formats like
OPL (a text based representation), which is useful for command line
tooling. We get this format basically for free:

http://overpass-turbo.eu/s/jpL

Note: This feature is available for testing on the following endpoint
only: http://dev.overpass-api.de/test753/


Best,
mmd


[1] http://www.asklater.com/matt/blog/2015/11/18/the-road-to-api-07/

[pierzen]

Great mmd

For Humanitarian OSM Responses, we often have to extract a lot of data either to validate, product extracts or build statistics. Compression of the data should greatly help, especially for teams in the field with limited internet access.

I tested extracting highways for south-west of Haiti. I then succeeded to import in QGIS.

One great option would be to have a file name field. It could be something like

[out:pbf="filename-xxx.pbf"].  It is not easy to indentify + determine extension of files named interpreter in directory file list.

Great thanks to the Overpass team for such constant improvements.

---

De : mmd <[hidden email]>
À : [hidden email]
Envoyé le : lundi 17 octobre 2016 12h59
Objet : [overpass] PBF output - prototype available for testing

Hi,

PBF is well known for its compactness and better efficiency. No wonder
it is also on the roadmap for the official API's map call (see [1].
That's reason enough to evaluate how PBF could also be integrated into
Overpass API.

Thanks to libosmium and Jochen's great support, I'm happy to present a
first (incomplete) prototype:

http://overpass-turbo.eu/s/jpK

Be sure to click on Export -> raw data directly from Overpass API to
avoid garbled output in overpass turbo.

When trying your own examples, be sure to always use `out meta;`, all
other modes are not implemented in this prototype.

I tested pbf output for areas of roughly 300km by 300km, yielding about
500MB in PBF file size. (in turbo use: Load -> Examples -> Map Call)

Along with PBF, libosmium also provides additional output formats like
OPL (a text based representation), which is useful for command line
tooling. We get this format basically for free:

http://overpass-turbo.eu/s/jpL

Note: This feature is available for testing on the following endpoint
only: http://dev.overpass-api.de/test753/


Best,
mmd


[1] http://www.asklater.com/matt/blog/2015/11/18/the-road-to-api-07/

[mmd]

Am 17.10.2016 um 20:51 schrieb Pierre Béland:

> For Humanitarian OSM Responses, we often have to extract a lot of data
> either to validate, product extracts or build statistics. Compression of
> the data should greatly help, especially for teams in the field with
> limited internet access.
>
> I tested extracting highways for south-west of Haiti. I then succeeded
> to import in QGIS.

Thank you for the positive feedback! I hope Roland won't object the use
of libosmium as an 'include-only' library in this case, given that we
can also leverage it to quickly set up a new database using a .pbf
planet. Like we did for lz4 compression, this could also be an optional
feature.

Anyway, the main reason for making this a fairly minimum prototype has
to do with ongoing restructuring of the code Roland is currently working
on. Once that task is completed, we will probably revisit this PBF topic.


> One great option would be to have a file name field. It could be
> something like
> [out:pbf="filename-xxx.pbf"].  It is not easy to indentify + determine
> extension of files named interpreter in directory file list.

I believe, we already have an issue for it
(https://github.com/drolbr/Overpass-API/issues/153), if you remember ;)

--

[pierzen]

 mmd <[hidden email]> wrote

> I believe, we already have an issue for it

> (https://github.com/drolbr/Overpass-API/issues/153), if you remember ;)

--