osmium check-refs failing on latest assembled PBF with hourly changes

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

osmium check-refs failing on latest assembled PBF with hourly changes

Developer Discussion mailing list

Hi, we have been applying new hourly changesets to create a daily PBF. This works very well and it is continuously stable. Recently, when the hourly changes were applied, I noticed a failure in osmium check-refs.


Hourly file changes applied are from these files

from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz


Updating sequence number from 71849 to 72051


Running a osmium check-refs * on the updated PBF results in several referential check failures for nodes. Nodes that belong to below OSM ways. When I looked into these OSM ways through JOSM, I could not spot any problems.


Ways

https://www.openstreetmap.org/way/185306812

https://www.openstreetmap.org/way/36960460

https://www.openstreetmap.org/way/326450137

https://www.openstreetmap.org/way/872325295

https://www.openstreetmap.org/way/878901824

https://www.openstreetmap.org/way/878968187

https://www.openstreetmap.org/way/878968195


Example nodes

https://www.openstreetmap.org/node/8176907685

https://www.openstreetmap.org/node/8176907689

https://www.openstreetmap.org/node/8176907690

https://www.openstreetmap.org/node/8176907686

https://www.openstreetmap.org/node/8176968661

https://www.openstreetmap.org/node/8176907653


Could someone help if they have observed any missing data in the hourly files or if osmium check-refs itself is not highlighting any problematic relation.

Any help is appreciated, thanks.


Manohar

https://www.openstreetmap.org/user/manoharuss


_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: osmium check-refs failing on latest assembled PBF with hourly changes

Jochen123
Hi!

There is a problem with the hourly diffs. From my analysis it looks like
minutely diff 4304922 is in hourly diff 72012 and minutely diff 430925
is in hourly 72013, but the two minutely diffs in between (4304923 and
4304924) are in neither. So you are definitely missing some data and
osmium check-refs found part of that.

Sysadmins are aware of the problem and maybe we going to get a fixed
hourly file. If not you should be able to fix this by applying those two
minutely diffs and then re-applying all the hourly diffs from that
point.

Jochen

On Tue, Dec 01, 2020 at 11:53:49AM -0500, Manohar Erikipati via dev wrote:

> Date: Tue, 1 Dec 2020 11:53:49 -0500
> From: Manohar Erikipati via dev <[hidden email]>
> To: [hidden email]
> Subject: [OSM-dev] osmium check-refs failing on latest assembled PBF with
>  hourly changes
>
> Hi, we have been applying new hourly changesets to create a daily PBF. This
> works very well and it is continuously stable. Recently, when the hourly
> changes were applied, I noticed a failure in osmium check-refs.
>
> Hourly file changes applied are from these files
>
> from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz
>
> Updating sequence number from 71849 to 72051
>
> Running a osmium check-refs * on the updated PBF results in several
> referential check failures for nodes. Nodes that belong to below OSM ways.
> When I looked into these OSM ways through JOSM, I could not spot any
> problems.
>
> Ways
>
> https://www.openstreetmap.org/way/185306812
>
> https://www.openstreetmap.org/way/36960460
>
> https://www.openstreetmap.org/way/326450137
>
> https://www.openstreetmap.org/way/872325295
>
> https://www.openstreetmap.org/way/878901824
>
> https://www.openstreetmap.org/way/878968187
>
> https://www.openstreetmap.org/way/878968195
>
> Example nodes
>
> https://www.openstreetmap.org/node/8176907685
>
> https://www.openstreetmap.org/node/8176907689
>
> https://www.openstreetmap.org/node/8176907690
>
> https://www.openstreetmap.org/node/8176907686
>
> https://www.openstreetmap.org/node/8176968661
>
> https://www.openstreetmap.org/node/8176907653
>
> Could someone help if they have observed any missing data in the hourly
> files or if osmium check-refs itself is not highlighting any problematic
> relation.
>
> Any help is appreciated, thanks.
>
> Manohar
>
> https://www.openstreetmap.org/user/manoharuss

> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/dev


--
Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: osmium check-refs failing on latest assembled PBF with hourly changes

Developer Discussion mailing list
Hi Jochen,

applying the 2 minutely files to the PBF worked nicely and passed osmium check-refs. Seems that is the only cause.

Thanks for looking into it and sharing the minutely files.
Manohar

On Wed, Dec 2, 2020 at 5:41 AM Jochen Topf <[hidden email]> wrote:
Hi!

There is a problem with the hourly diffs. From my analysis it looks like
minutely diff 4304922 is in hourly diff 72012 and minutely diff 430925
is in hourly 72013, but the two minutely diffs in between (4304923 and
4304924) are in neither. So you are definitely missing some data and
osmium check-refs found part of that.

Sysadmins are aware of the problem and maybe we going to get a fixed
hourly file. If not you should be able to fix this by applying those two
minutely diffs and then re-applying all the hourly diffs from that
point.

Jochen

On Tue, Dec 01, 2020 at 11:53:49AM -0500, Manohar Erikipati via dev wrote:
> Date: Tue, 1 Dec 2020 11:53:49 -0500
> From: Manohar Erikipati via dev <[hidden email]>
> To: [hidden email]
> Subject: [OSM-dev] osmium check-refs failing on latest assembled PBF with
>  hourly changes
>
> Hi, we have been applying new hourly changesets to create a daily PBF. This
> works very well and it is continuously stable. Recently, when the hourly
> changes were applied, I noticed a failure in osmium check-refs.
>
> Hourly file changes applied are from these files
>
> from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz
>
> Updating sequence number from 71849 to 72051
>
> Running a osmium check-refs * on the updated PBF results in several
> referential check failures for nodes. Nodes that belong to below OSM ways.
> When I looked into these OSM ways through JOSM, I could not spot any
> problems.
>
> Ways
>
> https://www.openstreetmap.org/way/185306812
>
> https://www.openstreetmap.org/way/36960460
>
> https://www.openstreetmap.org/way/326450137
>
> https://www.openstreetmap.org/way/872325295
>
> https://www.openstreetmap.org/way/878901824
>
> https://www.openstreetmap.org/way/878968187
>
> https://www.openstreetmap.org/way/878968195
>
> Example nodes
>
> https://www.openstreetmap.org/node/8176907685
>
> https://www.openstreetmap.org/node/8176907689
>
> https://www.openstreetmap.org/node/8176907690
>
> https://www.openstreetmap.org/node/8176907686
>
> https://www.openstreetmap.org/node/8176968661
>
> https://www.openstreetmap.org/node/8176907653
>
> Could someone help if they have observed any missing data in the hourly
> files or if osmium check-refs itself is not highlighting any problematic
> relation.
>
> Any help is appreciated, thanks.
>
> Manohar
>
> https://www.openstreetmap.org/user/manoharuss

> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/dev


--
Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Problem with hourly/daily diffs! was: Re: osmium check-refs failing on latest assembled PBF with hourly changes

Jochen123
In reply to this post by Jochen123
Hi!

There was a problem with hourly and daily diffs missing some changes on
Nov 11 2020. If you are digesting hourly or daily diffs and updating a
planet file or database regularly, you are affected by this and need to
take action or your planet file/database will be incomplete.

If you are using minutely diffs, you are okay. This doesn't apply there.

The diffs have been fixed by the sysadmins now (Thanks Tom!). This means
that everybody who is affected by this can fix their systems by going
back and re-applying diffs.

To fix your planet file/database, you have to go back to hourly diff
72011 and re-apply the hourly diffs from there. Or go back to daily diff
2999 and re-apply the daily diffs from there. It is not enough to just
apply the problematic minutely diffs or the one hourly/daily diff, you
have to reapply all diffs after that in the right order!

Jochen

On Wed, Dec 02, 2020 at 11:38:39AM +0100, Jochen Topf wrote:

> Date: Wed, 2 Dec 2020 11:38:39 +0100
> From: Jochen Topf <[hidden email]>
> To: Manohar Erikipati via dev <[hidden email]>
> Subject: Re: [OSM-dev] osmium check-refs failing on latest assembled PBF
>  with hourly changes
>
> Hi!
>
> There is a problem with the hourly diffs. From my analysis it looks like
> minutely diff 4304922 is in hourly diff 72012 and minutely diff 430925
> is in hourly 72013, but the two minutely diffs in between (4304923 and
> 4304924) are in neither. So you are definitely missing some data and
> osmium check-refs found part of that.
>
> Sysadmins are aware of the problem and maybe we going to get a fixed
> hourly file. If not you should be able to fix this by applying those two
> minutely diffs and then re-applying all the hourly diffs from that
> point.
>
> Jochen
>
> On Tue, Dec 01, 2020 at 11:53:49AM -0500, Manohar Erikipati via dev wrote:
> > Date: Tue, 1 Dec 2020 11:53:49 -0500
> > From: Manohar Erikipati via dev <[hidden email]>
> > To: [hidden email]
> > Subject: [OSM-dev] osmium check-refs failing on latest assembled PBF with
> >  hourly changes
> >
> > Hi, we have been applying new hourly changesets to create a daily PBF. This
> > works very well and it is continuously stable. Recently, when the hourly
> > changes were applied, I noticed a failure in osmium check-refs.
> >
> > Hourly file changes applied are from these files
> >
> > from hour/000/071/849.osc.gz to hour/000/072/051.osc.gz
> >
> > Updating sequence number from 71849 to 72051
> >
> > Running a osmium check-refs * on the updated PBF results in several
> > referential check failures for nodes. Nodes that belong to below OSM ways.
> > When I looked into these OSM ways through JOSM, I could not spot any
> > problems.
> >
> > Ways
> >
> > https://www.openstreetmap.org/way/185306812
> >
> > https://www.openstreetmap.org/way/36960460
> >
> > https://www.openstreetmap.org/way/326450137
> >
> > https://www.openstreetmap.org/way/872325295
> >
> > https://www.openstreetmap.org/way/878901824
> >
> > https://www.openstreetmap.org/way/878968187
> >
> > https://www.openstreetmap.org/way/878968195
> >
> > Example nodes
> >
> > https://www.openstreetmap.org/node/8176907685
> >
> > https://www.openstreetmap.org/node/8176907689
> >
> > https://www.openstreetmap.org/node/8176907690
> >
> > https://www.openstreetmap.org/node/8176907686
> >
> > https://www.openstreetmap.org/node/8176968661
> >
> > https://www.openstreetmap.org/node/8176907653
> >
> > Could someone help if they have observed any missing data in the hourly
> > files or if osmium check-refs itself is not highlighting any problematic
> > relation.
> >
> > Any help is appreciated, thanks.
> >
> > Manohar
> >
> > https://www.openstreetmap.org/user/manoharuss
>
> > _______________________________________________
> > dev mailing list
> > [hidden email]
> > https://lists.openstreetmap.org/listinfo/dev
>
>
> --
> Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688
>
> _______________________________________________
> dev mailing list
> [hidden email]
> https://lists.openstreetmap.org/listinfo/dev

--
Jochen Topf  [hidden email]  https://www.jochentopf.com/  +49-351-31778688

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Problem with hourly/daily diffs! was: Re: osmium check-refs failing on latest assembled PBF with hourly changes

Martin Raifer
Thanks for investigating this, Jochen! Just a small correction: when you wrote

> There was a problem with hourly and daily diffs missing some changes on Nov 11 2020.

I assume actually meant November 29, right?

_______________________________________________
dev mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/dev