TAH tile status checking

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

TAH tile status checking

Andre Hinrichs
Hi folks,

for more than one year now I run the tile-check script which checks for
some circumstances (old tiles, empty coasts, tiles which has shrunk too
much, and empty tiles which had aciently content). Patrick (petschge) is
also running some kind of checker which checks for outdated tiles (don't
know exactly, what it is doing). Since today there is apparently a third
checker running which also checks for outdated tiles (it is not checking
the current state from the server but apparently relies on the weekly
status file).

Since it makes no sense to run all these (and maybe more in the future)
I would like to suggest to carry all ideas together and build a really
good checker which runs on a server continuously (mine is only running
when I'm at home).

Any opinions?


Andre



_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: TAH tile status checking

Patrick Kilian
Hi,

> Patrick (petschge) is also running some kind of checker which checks
> for outdated tiles (don't know exactly, what it is doing).
I had to check the source code to see which features are actually
implemented. *g*


> Since it makes no sense to run all these (and maybe more in the
> future) I would like to suggest to carry all ideas together and build
> a really good checker which runs on a server continuously (mine is
> only running when I'm at home).
>
> Any opinions?
Makes sense. I uploaded the source code of my checker to
http://www.petschge.de/osm/oldtiles.sh.txt and I'd be willing to host
the new checker unless it has massive dependencies which I can't satisfy
with my debian old-stable box.

The basic algorithm (besides all the logging and stuff) is the following:

A.) Select uniform random x/y tile numbers and do checks:
1.) Check the png2tileinfo DB from T@h if we are out in the middle of
    the ocean. If we are skip that tile.
2.) Ask the server for the metadata of the tile. If the server serves
    this tile from the oceanfiles.dat (because it is empty land) don't
    check any further.
3.) If the tile is newer then 6 weeks it's new enough for now.
4.) Request the tile to be rerendered

B.) If a tile has to be rerendered the 8 neighbours are checked as well.

C.) If at least have of the neighbours have to be rerended check the 16
    next-to-neighbours of the tile too

D.) If it is more then 10 minutes that we checked the load level of the
    T@h server, then check the lenght of the rendering queue and sleep
    if it is greater then 10000. Also sleep an hour if the server fails
    to respond to the load query (it sometime does due to RoR
    strangeness)

E.) Sleep at least a second and start over with step A.


Check #1 might fail for tiles containing new islands but reduces the
load on the server by almost 2/3.
Check #2 is not strictly necessary but is a lot older then check #1 and
apparently does no harm.
Check #3 is obviously available for fine tuning, but we basically need
12 to 15 weeks to rerender every (non-empty) tile so half of that seems
to be a reasonable number.

Steps B and C increase the probability to find stale tiles a lot plus
the necessary metadata are cache hot on the server.

Step D was introduced to keep the queue full enough to keep all clients
busy despite the fluctuations from the regular tileexpiry code. On the
other hand it tries to keep the queue short enough that request from
other sources egt processed in a timely manner.



HTH,
Patrick "Petschge" Kilian

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: TAH tile status checking

Matthias Julius
In reply to this post by Andre Hinrichs

On Wed, 29 Sep 2010 19:12:54 +0200, Andre Hinrichs <[hidden email]>
wrote:
> Since it makes no sense to run all these (and maybe more in the future)
> I would like to suggest to carry all ideas together and build a really
> good checker which runs on a server continuously (mine is only running
> when I'm at home).
>
> Any opinions?

I would vote for running at least the simple checks directly on the tah
server.  The check for old tiles could simply iterate over the tileset
files and look at their atime.

Matthias

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: TAH tile status checking

spaetz
On 2010-09-29, Matthias Julius wrote:
> I would vote for running at least the simple checks directly on the tah
> server.  The check for old tiles could simply iterate over the tileset
> files and look at their atime.

In general, I don't mind running such a script on the server, but as
I don't have enormous amounts of time to implement and monitor such a
script, I also wouldn't mind if e.g. petschge continues to run it. Less
of a central point of failure. The tile sizes and age is still weekly
being produced and available here:
http://tah.openstreetmap.org/media/filesizes.bz2

So the checks can run efficiently without having to query the server for
each tile.

If it is as simple as adding a script to a cron entry, I can do it on
the server. But as I said, I don't have plenty of time and the server
CPUs are busy stitching lowzoom tiles (and more continous disk access
from a tile checker slows down disk access for up/downloads of course).

Sebastian

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: TAH tile status checking

Matthias Julius

On Thu, 30 Sep 2010 11:21:57 +0200, Sebastian Spaeth
<[hidden email]>
wrote:

> On 2010-09-29, Matthias Julius wrote:
>> I would vote for running at least the simple checks directly on the tah
>> server.  The check for old tiles could simply iterate over the tileset
>> files and look at their atime.
>
> In general, I don't mind running such a script on the server, but as
> I don't have enormous amounts of time to implement and monitor such a
> script, I also wouldn't mind if e.g. petschge continues to run it. Less
> of a central point of failure. The tile sizes and age is still weekly
> being produced and available here:
> http://tah.openstreetmap.org/media/filesizes.bz2
>
> So the checks can run efficiently without having to query the server for
> each tile.

This is true.

Is there any way to get the "age" of empty tilesets?  I guess this
information could only be derived from the upload logs.

Matthias

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: TAH tile status checking

spaetz
On 2010-09-30, Matthias Julius wrote:
> > http://tah.openstreetmap.org/media/filesizes.bz2
> >
> > So the checks can run efficiently without having to query the server for
> > each tile.
>
> This is true.
>
> Is there any way to get the "age" of empty tilesets?  I guess this
> information could only be derived from the upload logs.

nahh, there is no way to find the age of empty tilesets easily. I'd have
to parse the apache logs indeed.

Sebastian

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: TAH tile status checking

Andre Hinrichs
In reply to this post by Andre Hinrichs
Hi!

I've now also put my script on my website (took a while to bring it into
a nice form).

You can look at it at http://www.andre-hinrichs.de/tile-checker/

The site (as the script) is still under development and may change
rapidly.


Cheers
Andre



_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome