replicate id jump + db corruption

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

replicate id jump + db corruption

marc marc
Hello,

strange behavior :

2017-09-20 21:20:12
URL:http://planet.osm.org/replication/minute/002/631/278.state.txt 
[168/168] -> "/tmp/osm-3s_update_YRg34J/002631278.state.txt" [1]
2017-09-20 21:20:12
URL:http://planet.osm.org/replication/minute/002/631/278.osc.gz 
[28674/28674] -> "/tmp/osm-3s_update_YRg34J/002631278.osc.gz" [1]
2017-09-20 21:20:12: updating to 2631278
Reading XML file ... finished reading nodes. Flushing to database ......
done.
Reading XML file ... finished reading ways. Flushing to database ......
done.
Reading XML file ... finished reading relations. Flushing to database
....... done.
terminate called after throwing an instance of 'File_Error'
283.osc.gz [154664/154664] ->
"/tmp/osm-3s_update_iGmwJT/002631283.osc.gz" [1]

It try to access a file 283.osc.gz that was not yet downloaded.
It also try to use a gz file but the script gunzip all files
before running update_databse.

and the most annoying thing is that it thinks it has made the change
(next line have "updating from > 2631278", sorry I didn't keep the rest
of the file) and db was corrupt with in log file :
File_Blocks::read_block: Index inconsistent

Regards,
Marc
Reply | Threaded
Open this post in threaded view
|

Re: replicate id jump + db corruption

Roland Olbricht
Hello,

I'm sorry for the late answer. We moved from a flat to a house.

Could you please figure out which version is running? For the sake of
simplicity, the newest release 0.7.54.10 does show version numbers by

bin/osm3s_query --version

and in every output file in the first lines.

> Reading XML file ... finished reading relations. Flushing to database
> ....... done.
> terminate called after throwing an instance of 'File_Error'
> 283.osc.gz [154664/154664] ->
> "/tmp/osm-3s_update_iGmwJT/002631283.osc.gz" [1]

The last two lines look rather like a log from wget than to be related
to the error. If they were indeed from wget then either the version of
update_from_dir is so old that it still may produce return value zero on
error. Or changes in the fetch.. script conflict with return value
evaluation.

As newer version of both update_database and update_from_dir always
catch File_Error (the last few lines in the main(...)-function in the
files overpass_api/osm-backend/update_from_dir.cc catch File_Error), I
would like to ask you to double check which version is running.

Regards,

Roland
Reply | Threaded
Open this post in threaded view
|

Re: replicate id jump + db corruption

marc marc
Hello,

Le 29. 09. 17 à 06:51, Roland Olbricht a écrit :
> I'm sorry for the late answer. We moved from a flat to a house.

Happy moving :)

> Could you please figure out which version is running?

it should be 0.74.9. Binary was build on sept. 16 15:28

> the newest release 0.7.54.10 does show version numbers
> by bin/osm3s_query --version
> and in every output file in the first lines.

thank you for this addition which will facilitate debuging.

>> Reading XML file ... finished reading relations.
>> Flushing to database....... done.
>> terminate called after throwing an instance of 'File_Error'
>> 283.osc.gz [154664/154664] ->
>> "/tmp/osm-3s_update_iGmwJT/002631283.osc.gz" [1]
>
> The last two lines look rather like a log from wget than to be related
> to the error. If they were indeed from wget then either the version of
> update_from_dir is so old that it still may produce return value zero on
> error. Or changes in the fetch.. script conflict with return value
> evaluation.

I checked the small changes I made in the script,
nothing conflicts with return value.
I took the opportunity to add an error message
   apply_minute_diffs()
   {
     ./update_from_dir --osc-dir=$1 --version=$DATA_VERSION $META
--flush-size=0
     EXITCODE=$?
     while [[ $EXITCODE -ne 0 ]];
     do
     {
+     echo "`date -u '+%F %T'`: ERROR: update_from_dir return an
EXITCODE $EXITCODE. sleep 60 and try again."
       sleep 60
       ./update_from_dir --osc-dir=$1 --version=$DATA_VERSION $META
--flush-size=0
       EXITCODE=$?
     };
     done

what is the easiest way to transmit the changes?
I show them here and/or I post issue/commit/PR ?

Regards,
Marc