Multithreading in new version of osmosis

Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Multithreading in new version of osmosis

Sanjay B
Again, please let me know if there's a better place to ask osmosis
specific questions :-)

I have managed to successfully build and run the latest (v0.47) version
of osmosis on Github: https://github.com/openstreetmap/osmosis

I am trying to run a task that reads a PBF file and populates the
`apidb` schema. Earlier, when running the same command, I'd see osmosis
using all CPU cores on my machine, running multiple processes. However,
now I see only one `java` process running.

The command I'm running looks like:

`./osmosis --read-pbf file=~//pbf/monaco.pbf --write-apidb
database=openstreetmap host=127.0.0.1 user=osm password=pwd
validateSchemaVersion=no`

This works fine and data is imported, except that it seems to be running
only one thread, and I worry this would result in significantly slower
imports of larger data-sets.

I see this commit from April 2018 that might be (?) related to this:
https://github.com/openstreetmap/osmosis/commit/c438e9ddf64c4dff2dcc80191c95a453e3c067fc

Is there an option one can pass to force the older behaviour with
regards to multithreading? I was unable to find it in the docs - I tried
playing around with the `--buffer` parameter, but it did not seem to help.

Of course, I understand this version isn't properly "released" yet and
docs could be in flux, am super happy to play the role of tester here.
And of course, it's entirely possible I'm missing something obvious
(like my previous question here).

Thank you again for any help!
-Sanjay


_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Multithreading in new version of osmosis

Sanjay B
On Thursday 04 October 2018 04:29 PM, Sanjay B wrote:

>
> This works fine and data is imported, except that it seems to be
> running only one thread, and I worry this would result in
> significantly slower imports of larger data-sets.
>

I'm sorry - this was some other configuration I had misconfigured that
seems to have been affecting the speed of imports when testing the new
osmosis 0.47 - it seems to work fine with no slow-down from the previous
version(s) - I apologize for the hastily sent email without
double-checking the cause.

Thank you,
Sanjay

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Multithreading in new version of osmosis

Brett Henderson
Hi Sanjay, I'm glad you figured it out.  I must admit I'd forgotten to respond to your initial message ...

On Mon, 8 Oct 2018 at 20:19 Sanjay B <[hidden email]> wrote:
On Thursday 04 October 2018 04:29 PM, Sanjay B wrote:

>
> This works fine and data is imported, except that it seems to be
> running only one thread, and I worry this would result in
> significantly slower imports of larger data-sets.
>

I'm sorry - this was some other configuration I had misconfigured that
seems to have been affecting the speed of imports when testing the new
osmosis 0.47 - it seems to work fine with no slow-down from the previous
version(s) - I apologize for the hastily sent email without
double-checking the cause.

Thank you,
Sanjay

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev