[osmosis-dev] Error on a node modified and deleted in the same changeset

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[osmosis-dev] Error on a node modified and deleted in the same changeset

Jocelyn Jaubert
Hi,

I have just switched to Geofabrik's diffs to update my Switzerland .pbf file, and I'm hitting an issue with diffs. I'm not quite sure if the issue is with osmosis or not.

When downloading these two files:

http://download.geofabrik.de/europe/switzerland-updates/000/000/058.osc.gz
http://download.geofabrik.de/europe/switzerland-updates/000/000/059.osc.gz

and merging them through --read-replication-interval

You will see that node 559647562 is present twice, with the same changeset:

<?xml version='1.0' encoding='UTF-8'?>
<osmChange version="0.6" generator="Osmosis 0.41">
  <modify>
    <node id="559647562" version="2" timestamp="2013-04-29T12:55:06Z" uid="65044" user="hugi" changeset="15907651" lat="46.9473255" lon="7.4651306"/>
  </modify>
  <delete>
    <node id="559647562" version="2" timestamp="2013-05-01T00:33:27Z" uid="65044" user="hugi" changeset="15907651" lat="46.9473255" lon="7.4651306"/>
  </delete>
</osmChange>

So first question: should this be correctly handled by osmosis ? Currently, a --sort-change complains about a non-sorted file:

GRAVE: Thread for task 1-read-xml-change failed
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities are not sorted, previous entity type=Node, id=559647562, version=2 current entity type=Node, id=559647562, version=2.



According to this node history:

http://www.openstreetmap.org/browse/node/559647562/history

this node was in fact deleted in a following changeset

Could there be an issue with how the diffs are generated ?


Thanks,
Jocelyn

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

michael spreng-3
Hi

This problem comes from the diff, when an object is deleted, the Version
is not incremented. But for logical reasons, it should. This
incrementation is missing in osmosis as well as osmconvert, so at the
moment, custom diffs can't be merged while keeping them sorted.

Michael

On 02.05.2013 20:28, Jocelyn Jaubert wrote:

> Hi,
>
> I have just switched to Geofabrik's diffs to update my Switzerland .pbf file, and I'm hitting an issue with diffs. I'm not quite sure if the issue is with osmosis or not.
>
> When downloading these two files:
>
> http://download.geofabrik.de/europe/switzerland-updates/000/000/058.osc.gz
> http://download.geofabrik.de/europe/switzerland-updates/000/000/059.osc.gz
>
> and merging them through --read-replication-interval
>
> You will see that node 559647562 is present twice, with the same changeset:
>
> <?xml version='1.0' encoding='UTF-8'?>
> <osmChange version="0.6" generator="Osmosis 0.41">
>   <modify>
>     <node id="559647562" version="2" timestamp="2013-04-29T12:55:06Z" uid="65044" user="hugi" changeset="15907651" lat="46.9473255" lon="7.4651306"/>
>   </modify>
>   <delete>
>     <node id="559647562" version="2" timestamp="2013-05-01T00:33:27Z" uid="65044" user="hugi" changeset="15907651" lat="46.9473255" lon="7.4651306"/>
>   </delete>
> </osmChange>
>
> So first question: should this be correctly handled by osmosis ? Currently, a --sort-change complains about a non-sorted file:
>
> GRAVE: Thread for task 1-read-xml-change failed
> org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities are not sorted, previous entity type=Node, id=559647562, version=2 current entity type=Node, id=559647562, version=2.
>
>
>
> According to this node history:
>
> http://www.openstreetmap.org/browse/node/559647562/history
>
> this node was in fact deleted in a following changeset
>
> Could there be an issue with how the diffs are generated ?
>
>
> Thanks,
> Jocelyn
>
> _______________________________________________
> osmosis-dev mailing list
> [hidden email]
> http://lists.openstreetmap.org/listinfo/osmosis-dev


_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Jocelyn Jaubert
Hi,

Le 02/05/2013 22:30, Michael Spreng a écrit :
>
> This problem comes from the diff, when an object is deleted, the Version
> is not incremented. But for logical reasons, it should. This
> incrementation is missing in osmosis as well as osmconvert, so at the
> moment, custom diffs can't be merged while keeping them sorted.

Ok, that makes sense.

Do you mean that on diff generated on planet.osm.org, the version on
deleted objects is increased by one ?

If it is the recommended way, I can take a look at osmosis code source,
and see if this version increase is something easy to do.


Otherwise, does it mean that we have to apply diff from Geofabrik one
per one, and don't use osmosis --read-replication-interval with these
diffs ?


Thanks,
Jocelyn

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

michael spreng-3

On 03.05.2013 23:55, Jocelyn Jaubert wrote:
> Do you mean that on diff generated on planet.osm.org, the version on
> deleted objects is increased by one ?
Not on diff generation, actually, but on the api-db level already, when
someone deletes something.  I don't know exactly how diffs from the
api-db are generated, but the api-db certainly knows about the version
of deleted objects.
> If it is the recommended way, I can take a look at osmosis code source,
> and see if this version increase is something easy to do.
In my opinion yes, this is the only sane way to handle this. Just take
care that you won't break creation of diffs from the api-db.
> Otherwise, does it mean that we have to apply diff from Geofabrik one
> per one, and don't use osmosis --read-replication-interval with these
> diffs ?
Just ensure with the interval that it only consumes one diff at a time,
then osmosis won't have a problem with the diffs.

Michael


_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Jocelyn Jaubert
Le 04/05/2013 08:45, Michael Spreng a écrit :
>> If it is the recommended way, I can take a look at osmosis code source,
>> and see if this version increase is something easy to do.
> In my opinion yes, this is the only sane way to handle this. Just take
> care that you won't break creation of diffs from the api-db.

I'm including a patch for this, that only impacts the ChangeDeriver class.

I have tested it on a country, and the generated diff was correct - the
only difference with the current osmosis version is the version on
deleted elements.

I haven't tested on the api-db.


If you prefer that I send the patch to trac, or github, please ask me.


--
Jocelyn

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev

0001-Increment-version-in-ChangeDeriver-when-an-object-is.patch (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Peter Körner
In reply to this post by Jocelyn Jaubert
Am 03.05.2013 23:55, schrieb Jocelyn Jaubert:
> Otherwise, does it mean that we have to apply diff from Geofabrik one
> per one, and don't use osmosis --read-replication-interval with these
> diffs ?

Couldn't you use --simplify-change to fix the issue?

Peter


_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

michael spreng-3

On 08.05.2013 01:04, Peter Körner wrote:
> Am 03.05.2013 23:55, schrieb Jocelyn Jaubert:
>> Otherwise, does it mean that we have to apply diff from Geofabrik one
>> per one, and don't use osmosis --read-replication-interval with these
>> diffs ?
>
> Couldn't you use --simplify-change to fix the issue?
>
Simplify change will not work, because it requires sorted input. But
there will be a create/modify with the same version number as the
delete, which leads to the error reported in his first mail.
It works without --simplify-change though

Michael

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Martin Schafran
 
> Simplify change will not work, because it requires sorted input.

where do you know? it's not documented.

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

michael spreng-3
Just download the two diffs supplied in the first mail in this thread,
do a merge and a simplify change, and observe the error message.

Alternatively: read the source.


_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Frederik Ramm
In reply to this post by Jocelyn Jaubert
Hi,

On 07.05.2013 23:50, Jocelyn Jaubert wrote:
> I'm including a patch for this, that only impacts the ChangeDeriver class.
>
> I have tested it on a country, and the generated diff was correct - the
> only difference with the current osmosis version is the version on
> deleted elements.

I had not followed this thread, sorry. I didn't know the Geofabrik diffs
had this problem, but of course if one thinks about it, it does seem
logical - comparing two excerpts, when the newer one doesn't have an
object then how am I to know who deleted it when and in what version?

I could run Jocelyn's patch on the Geofabrik server but technically it
would create wrong osc files - if you were to compare a Geofabrik osc
with one loaded from osm.org you'd find that the Geofabrik diff reports
the wrong user name, wrong timestamp, and *possibly* also wrong version
on a deleted object.

No idea if that problem is worse than the one we have now!

Bye
Frederik

--
Frederik Ramm  ##  eMail [hidden email]  ##  N49°00'09" E008°23'33"

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Jocelyn Jaubert
Hi,

Le 09/07/2013 21:49, Frederik Ramm a écrit :
> I could run Jocelyn's patch on the Geofabrik server but technically it
> would create wrong osc files - if you were to compare a Geofabrik osc
> with one loaded from osm.org you'd find that the Geofabrik diff reports
> the wrong user name, wrong timestamp, and *possibly* also wrong version
> on a deleted object.

I understand that my patch is far from ideal, but it make it possible to
run osmosis to apply all diff at the same time. There must be some
better solution, but I didn't find any

I'm currently using a workaround, which is to set maxInterval in
configuration.txt to 0.5 day to make sure that osmosis is not trying to
merge two diffs. Then I had to add a loop to run osmosis until there are
no recent diffs.

Maybe the best way to fix this is to add a note on a website explaining
that the diffs generated by Geofabrik are not meant for "direct"
consumption by osmosis with a big value on maxInterval ?

--
Jocelyn

_______________________________________________
osmosis-dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Frederik Ramm
Hi,

   (re-heating this 4 month old thread)

On 07/13/2013 01:03 PM, Jocelyn Jaubert wrote:
> I'm currently using a workaround, which is to set maxInterval in
> configuration.txt to 0.5 day to make sure that osmosis is not trying to
> merge two diffs. Then I had to add a loop to run osmosis until there are
> no recent diffs.

I've just been bitten by the same problem when trying to keep an
Europe-only Nominatim instance up to date with Geofabrik diffs.

I wonder if simply dropping the --simplify-change from Nominatim's
osmosis invocation will solve things - might lose a little performance
for things modified on subsequent days but hey. (One could even do this:
try with --simplify-change by default but if this fails, re-run
without... Nominatim already re-runs osmosis if it fails so I'd just
have to add the "drop simplify-change" code.)

> Maybe the best way to fix this is to add a note on a website explaining
> that the diffs generated by Geofabrik are not meant for "direct"
> consumption by osmosis with a big value on maxInterval ?

I'll do that.

Bye
Frederik

--
Frederik Ramm  ##  eMail [hidden email]  ##  N49°00'09" E008°23'33"

_______________________________________________
osmosis-dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/osmosis-dev
Reply | Threaded
Open this post in threaded view
|

Re: [osmosis-dev] Error on a node modified and deleted in the same changeset

Sarah Hoffmann
In reply to this post by Jocelyn Jaubert
Hi,

[another try, this time to the list]

On Wed, Nov 06, 2013 at 10:10:03AM +0100, Frederik Ramm wrote:

> On 07/13/2013 01:03 PM, Jocelyn Jaubert wrote:
> > I'm currently using a workaround, which is to set maxInterval in
> > configuration.txt to 0.5 day to make sure that osmosis is not trying to
> > merge two diffs. Then I had to add a loop to run osmosis until there are
> > no recent diffs.
>
> I've just been bitten by the same problem when trying to keep an
> Europe-only Nominatim instance up to date with Geofabrik diffs.
>
> I wonder if simply dropping the --simplify-change from Nominatim's
> osmosis invocation will solve things - might lose a little performance
> for things modified on subsequent days but hey. (One could even do this:
> try with --simplify-change by default but if this fails, re-run
> without... Nominatim already re-runs osmosis if it fails so I'd just
> have to add the "drop simplify-change" code.)

That does not work either because osm2pgsql chokes on a osc diff
that was not simplified.

> > Maybe the best way to fix this is to add a note on a website explaining
> > that the diffs generated by Geofabrik are not meant for "direct"
> > consumption by osmosis with a big value on maxInterval ?
>
> I'll do that.

With Nominatim 2.1 and the settings as recommended at the Nominatim
installation website [1], the update process should be doing the
right thing and apply the daily diffs one after another.

Sarah

[1] http://wiki.openstreetmap.org/wiki/Nominatim/Installation#Setting_up_the_update_process

_______________________________________________
osmosis-dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/osmosis-dev