Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

General Discussion mailing list
Dear all,

in the minute diff, in file
https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz
the way 40657824 uses node 7804408284, but that node is contained
neither in 694.osc.gz nor in any earlier minute diff.

In addition, it looks like there are large batches of nodes missing.
 From the create block:
    <node id="7804406584" version="1" timestamp="2020-08-11T16:36:55Z"
uid="2077889" user="zsolt d" changeset="89261325" lat="46.8903819"
lon="17.6638739">
       <tag k="denotation" v="landmark"/>
       <tag k="leaf_cycle" v="deciduous"/>
       <tag k="leaf_type" v="broadleaved"/>
       <tag k="natural" v="tree"/>
     </node>
     <node id="7804408683" version="1" timestamp="2020-08-11T16:32:40Z"
uid="11630388" user="xcpascual" changeset="89261826" lat="23.4076878"
lon="81.4237601"/>

There are multiple other referential integrity errors as well.

This has caused an emergency stop for the Overpass API updates, because
otherwise it would produce broken data. I'm sorry for that.

Could someone from the sysadmin team please have a look at the file?

Best regards,

Roland


_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

General Discussion mailing list
On 12/08/2020 13:56, Roland Olbricht via talk wrote:

> in the minute diff, in file
> https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz
> the way 40657824 uses node 7804408284, but that node is contained
> neither in 694.osc.gz nor in any earlier minute diff.

It's in 693 as far as I can see?

Tom

--
Tom Hughes ([hidden email])
http://compton.nu/

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
mmd
Reply | Threaded
Open this post in threaded view
|

Re: Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

mmd
Am 12.08.20 um 15:40 schrieb Tom Hughes via talk:

> On 12/08/2020 13:56, Roland Olbricht via talk wrote:
>
>> in the minute diff, in file
>> https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz
>> the way 40657824 uses node 7804408284, but that node is contained
>> neither in 694.osc.gz nor in any earlier minute diff.
>
> It's in 693 as far as I can see?
>
> Tom
>

693 has been generated twice by osmosis, and Overpass API picked up the
old version. This caused multiple nodes to be missing, which caused a
subsequent crash.

See my comment here: https://github.com/drolbr/Overpass-API/issues/591

--



_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

General Discussion mailing list
Hi all,

>>> in the minute diff, in file
>>> https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz
>>> the way 40657824 uses node 7804408284, but that node is contained
>>> neither in 694.osc.gz nor in any earlier minute diff.
>>
>> It's in 693 as far as I can see?
>>
>> Tom

I would like to thank mmd for the investigation and TomH for clarifying
the correct order of diff downloads, see
https://wiki.openstreetmap.org/wiki/Planet.osm/diffs#Fetching_diff_files

There is a new release of the Overpass API
https://dev.overpass-api.de/releases/osm-3s_v0.7.56.6.tar.gz
that adheres to these rules.

Best regards,
Roland

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk
Reply | Threaded
Open this post in threaded view
|

Re: Referential integrity of https://planet.openstreetmap.org/replication/minute//004/146/694.osc.gz

General Discussion mailing list
Hi all,

> There is a new release of the Overpass API
> https://dev.overpass-api.de/releases/osm-3s_v0.7.56.6.tar.gz
> that adheres to these rules.

Unfortunately, getting network communication right is hard. In that
version, there is a race condition. For that reason, there is another
new release 0.7.56.7.

You can run that release and keep the database, in particular if an
update based on 0.7.56.4 or 0.7.56.6 has ended with

     gzip: stdin: unexpected end of file
     Reading XML file ...Parse error at line 23105:
     unclosed token

This means that apply_osc_to_db.sh has tried to use an XML file before
fetch_osc.sh has completed to download it. It is harmless, in this case
the automatic emergency stop has applied early enough. Please do the
following (with EXEC_BASE_DIR, DB_DIR and DIFF_DIR appropriately replaced):

# Install new bash scripts over the exisiting broken ones
$ pushd "$EXEC_BASE_DIR"
$ cp ~/osm-3s_v0.7.56.7/bin/fetch_osc.sh bin/
$ cp ~/osm-3s_v0.7.56.7/bin/apply_osc_to_db.sh bin/

# Kill the frozen existing processes and remove the lock file
$ ps -ef | grep apply
roland   19676     1  0 Aug13 ?        00:00:08 bash
bin/apply_osc_to_db.sh /opt/osm3s/diff/ auto --meta=attic

$ ps -ef | grep 19676
roland   19676     1  0 Aug13 ?        00:00:08 bash
bin/apply_osc_to_db.sh /opt/osm3s/diff/ auto --meta=attic
roland   20652  4841  0 04:09 pts/0    00:00:00 grep --color=auto 19676
roland   21629 19676  0 Aug13 ?        00:00:00 ./update_from_dir
--osc-dir=/tmp/osm-3s_update_VUkjcE --version=2020-08-13T20\:33\:03Z
--keep-attic --flush-size=0

$ kill 19676 21629
$ rm /ssd/osm3s/db/osm_base_shadow.lock

$ ps -ef | grep fetch
roland   23675     1  0 Aug13 ?        00:00:06 bash bin/fetch_osc.sh
4146694 https://planet.openstreetmap.org/replication/minute/
/opt/osm3s/diff/
roland   27830  4841  0 04:11 pts/0    00:00:00 grep --color=auto fetch

$ kill 23675

# Restart the diff processes
$ nohup bin/fetch_osc.sh $(cat /ssd/osm3s/db/replicate_id)
'https://planet.openstreetmap.org/replication/minute/' "$DIFF_DIR"
 >>fetch_osc.out &
$ cat "$DB_DIR/replicate_id"
4150150
# Must exist and contain a meaningful diff id, this is a new lock file

$ nohup bin/apply_osc_to_db.sh /opt/osm3s/diff/ auto --meta=attic
 >>apply_osc_to_db.out &
$ sleep 30
$ tail apply_osc_to_db.out
/opt/osm3s/bin /opt/osm3s
Reading XML file ...
# Should show some messages after the "read unclosed token" error
# The first ones are the output of pushd to $DB_DIR/bin and the progress
line "Reading XML file ..." and so on

I'm sorry for the inconvenience and hope that this has fixed all race
conditions. Not yet adapted is "fetch_osc_and_apply.sh". If someone uses
this then please mail me. I will then watch with you whether an improved
version of that script is stable.

Best regards,
Roland

_______________________________________________
talk mailing list
[hidden email]
https://lists.openstreetmap.org/listinfo/talk