t@h/osmarender being phased out (at least from my side)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

t@h/osmarender being phased out (at least from my side)

spaetz
Hi there,

you might have noticed that tiles@home was pretty quiet from my side
although the server is nicely humming along all the time. During the
last 6 month we had 416 user accounts uploading data (possibly using many
more boxes). 65GB of tiles were uploaded just yesterday.

I have to admit that I pretty much lost my motivation to continue
improving the server and the client code. deelkar has been saying that
he would keep the client going but is not likely to implement the next
cool feature either. Also the underlying osmarender implementation in
perl is kind of maintainerless.

I believe t@h has achieved a great deal of things. It is a great proof
of concept for implementing alternative renderers. And using XSLT
transformations to create .svg files out of .osm files. How cool is
that! All credits for that geekie hack go to Etienne (80n) who created
the original xslt templates that made osmarender. We were the first to
provide current maps when someone edited the data. And we were the first
system to have data validation layers.

It is also a great concept of a distributed renderer, which helped to
keep the world's maps up to date when mapnik could only be run on weekly
or even older snapshots. Furthermore, being separate from the OSMF
foundation infrastructure, we were an independent provider of map tiles
without usage restrictions to this day. No one could shut us off and we
were not limited by policies decided on by someone else.  

t@h has spawned a great deal of innovations surrounding the renderer
system. Due to our need of downloading huge amounts of data we triggered
the development of several read-only API servers that were able to
satisfy our needs. osm diffs every minute helped to keep the RO mirror
system up to date without burdening the API much. Custom layers helped
to identify data errors (maplint). Lots of great stuff there.

However, in its current form it passed its zenith. Developments such as
wikimedia's toolserver, mapcss clients, mapnik realtime updates, and 3rd
party improvements such as cloudmaps custom style editor achieved much
of what osmarender/t@h set out to do (at least in *my* vision)

There are also disadvantages, some of which could be fixed easily,
and some which are inherent in the current architecture:

- Being distributed we require huge amounts of data being transferred
  over the network. Each render client requested a new z12 tile
  full of data for each update.

- By proactively rendering the world on each change rather than
  rendering on demand, we uploaded lots of data much of which was never
  viewed. Generally, the server upload bandwidth is approximately equal
  to the server download bandwidth, so we were quite wasteful there.

- Mapnik has gotten so efficient that a single powerful box is able to
  keep the world up to date, while we were keeping lots of computers
  busy. During the last 6 months, we had 416 user accounts uploading
  data, and 65GB of data has been uploaded just yesterday.

- The osma client in perl is pretty much in abandoned state, and other
  clients seem to become more advanced as time goes by. It would easily
  be possible to switch the current tilesGen.pl client out for a
  different one, but no one seems to be interested in doing that
  development. And no one has volunteered during the previous years
  either.

- Using inkscape to render the tiles is obviously not optimal for the
  size of the tiles that we are rendering. We are still witnessing hangs
  and freezes. At least to corruption of the inkscape user config files
  seems to have been solved by now :-).  Admittedly, we probably force
  inkscape to perform feats it has not been designed to do.

- Similarly the stylesheets: I consider the openness in tweaking style
  sheets one of the killer features of the current system. Anyone with
  an OSM svn account can change the t@h style and improve it. However,
  there seemed little interest in really developing this
  further. Besides the tireless petschge whom I would like to thank for
  his efforts, and user "Mueck" who recently seems to have done stuff,
  not many people seem interested in tweaking and improving things.
  So easily modifyable style sheets don't seem to be needed by the
  community, at least not in the osmarender form :).

WHAT IS GOING TO HAPPEN

I will keep the server running as is, but I will not be making great
improvements to it. Deelkar is going to keep the tilesGen client running
but willnot be making great improvements to it. So these are the
possibilities:

- We keep things running as is until something (likely the server or our
  RO mirror network) breaks.

- Someone volunteers to take over client development (either the
  existing one or a replacement in whatever language you prefer). The
  server is pretty client agnostic, all it does is dish out requests and
  take back tileset files, so that could be done by whatever app.

- Some one volunteers to take over server development. This would likely
  involve getting a new server, as I will not be allowed to hand out
  access to the current one, unfortunately.

It has been a wild and fun ride over the years for me. I'd like the thank
Oliver White for conceiving t@h in the first place, petschge and deelkar
for everything they have done on the client side, server side and
otherwise. Frederik Ramm for helping with code (the perl implementation
of osma), advise and moral support. Bobkare for helping out with the
rendering code, and xslt stuff. Of course all those bug reporters and
testers helped a lot, you know who you are (yes, Alfons, for example
you!)
And last but not least, all those tireless renderers that keep
the world up to date. Keep them running!

Let me know what you think and how we should be proceeding. If you
volunteer to take over something or want to be involved in an effort to
reorganize things speak up.

spaetz

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome

attachment0 (203 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: t@h/osmarender being phased out (at least from my side)

Dirk-Lüder Kreie
Am 08.02.2011 16:49, schrieb Sebastian Spaeth:
> Let me know what you think and how we should be proceeding. If you
> volunteer to take over something or want to be involved in an effort to
> reorganize things speak up.

You don't need to stop your renderers, it's still a tileserver with
current tiles for Applications that need to download large quantities of
tiles without putting that load on the osm.org tileserver.

It's just that Spaetz, and to some extent I, have other stuff that we
do, and Osmarender/Tiles@Home does run well without much work from our side.

That means, that the Tiles@Home project will phase out itself, when we
finally don't have any time for it anymore and something major breaks.

That is, unless someone from the community steps up and takes things over.

The problems with that would be A) the server was granted to spaetz, and
not the Tiles@Home Project, so whoever takes over server maintenance has
to find another server.
B) the client software is quite complex, and anyone taking over that
would have a steep learning curve, but I am willing to help with that,
as I am the one dev that kept at it from nearly the beginning until now,
so I'm in a good position to explain why things are done the way they are.

So, we will not actively start to shut everything down, but let it run
until something comes up that requires too much effort to fix.
I would assume the remaining lifetime of Tiles@Home can be measured in
years rather than weeks.

Happy rendering.
--

Dirk-Lüder "Deelkar" Kreie
Bremen - 53.0901°N 8.7868°E


_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome

signature.asc (268 bytes) Download Attachment
ben
Reply | Threaded
Open this post in threaded view
|

Re: t@h/osmarender being phased out (at least from my side)

ben
In reply to this post by spaetz
hi,

oh no! I hope I will hit the Top100 before everything breaks up! ;)

Despite the personal things and disadvantages, do you (and all other readers) see any
reasons for t@h to be alive? I mean, is the effort worth to be done by possible "volunteers".
if not, then nobody will miss t@h and all are good to go to other projects.
which leads me to the question: what server specs would be needed? how many time should
be spent to "keep the thing running", etc.?

would love to hear any thoughts!

ben

On Tue, Feb 8, 2011 at 4:49 PM, Sebastian Spaeth <[hidden email]> wrote:
Hi there,

you might have noticed that tiles@home was pretty quiet from my side
although the server is nicely humming along all the time. During the
last 6 month we had 416 user accounts uploading data (possibly using many
more boxes). 65GB of tiles were uploaded just yesterday.

I have to admit that I pretty much lost my motivation to continue
improving the server and the client code. deelkar has been saying that
he would keep the client going but is not likely to implement the next
cool feature either. Also the underlying osmarender implementation in
perl is kind of maintainerless.

I believe t@h has achieved a great deal of things. It is a great proof
of concept for implementing alternative renderers. And using XSLT
transformations to create .svg files out of .osm files. How cool is
that! All credits for that geekie hack go to Etienne (80n) who created
the original xslt templates that made osmarender. We were the first to
provide current maps when someone edited the data. And we were the first
system to have data validation layers.

It is also a great concept of a distributed renderer, which helped to
keep the world's maps up to date when mapnik could only be run on weekly
or even older snapshots. Furthermore, being separate from the OSMF
foundation infrastructure, we were an independent provider of map tiles
without usage restrictions to this day. No one could shut us off and we
were not limited by policies decided on by someone else.

t@h has spawned a great deal of innovations surrounding the renderer
system. Due to our need of downloading huge amounts of data we triggered
the development of several read-only API servers that were able to
satisfy our needs. osm diffs every minute helped to keep the RO mirror
system up to date without burdening the API much. Custom layers helped
to identify data errors (maplint). Lots of great stuff there.

However, in its current form it passed its zenith. Developments such as
wikimedia's toolserver, mapcss clients, mapnik realtime updates, and 3rd
party improvements such as cloudmaps custom style editor achieved much
of what osmarender/t@h set out to do (at least in *my* vision)

There are also disadvantages, some of which could be fixed easily,
and some which are inherent in the current architecture:

- Being distributed we require huge amounts of data being transferred
 over the network. Each render client requested a new z12 tile
 full of data for each update.

- By proactively rendering the world on each change rather than
 rendering on demand, we uploaded lots of data much of which was never
 viewed. Generally, the server upload bandwidth is approximately equal
 to the server download bandwidth, so we were quite wasteful there.

- Mapnik has gotten so efficient that a single powerful box is able to
 keep the world up to date, while we were keeping lots of computers
 busy. During the last 6 months, we had 416 user accounts uploading
 data, and 65GB of data has been uploaded just yesterday.

- The osma client in perl is pretty much in abandoned state, and other
 clients seem to become more advanced as time goes by. It would easily
 be possible to switch the current tilesGen.pl client out for a
 different one, but no one seems to be interested in doing that
 development. And no one has volunteered during the previous years
 either.

- Using inkscape to render the tiles is obviously not optimal for the
 size of the tiles that we are rendering. We are still witnessing hangs
 and freezes. At least to corruption of the inkscape user config files
 seems to have been solved by now :-).  Admittedly, we probably force
 inkscape to perform feats it has not been designed to do.

- Similarly the stylesheets: I consider the openness in tweaking style
 sheets one of the killer features of the current system. Anyone with
 an OSM svn account can change the t@h style and improve it. However,
 there seemed little interest in really developing this
 further. Besides the tireless petschge whom I would like to thank for
 his efforts, and user "Mueck" who recently seems to have done stuff,
 not many people seem interested in tweaking and improving things.
 So easily modifyable style sheets don't seem to be needed by the
 community, at least not in the osmarender form :).

WHAT IS GOING TO HAPPEN

I will keep the server running as is, but I will not be making great
improvements to it. Deelkar is going to keep the tilesGen client running
but willnot be making great improvements to it. So these are the
possibilities:

- We keep things running as is until something (likely the server or our
 RO mirror network) breaks.

- Someone volunteers to take over client development (either the
 existing one or a replacement in whatever language you prefer). The
 server is pretty client agnostic, all it does is dish out requests and
 take back tileset files, so that could be done by whatever app.

- Some one volunteers to take over server development. This would likely
 involve getting a new server, as I will not be allowed to hand out
 access to the current one, unfortunately.

It has been a wild and fun ride over the years for me. I'd like the thank
Oliver White for conceiving t@h in the first place, petschge and deelkar
for everything they have done on the client side, server side and
otherwise. Frederik Ramm for helping with code (the perl implementation
of osma), advise and moral support. Bobkare for helping out with the
rendering code, and xslt stuff. Of course all those bug reporters and
testers helped a lot, you know who you are (yes, Alfons, for example
you!)
And last but not least, all those tireless renderers that keep
the world up to date. Keep them running!

Let me know what you think and how we should be proceeding. If you
volunteer to take over something or want to be involved in an effort to
reorganize things speak up.

spaetz

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome



_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome
Reply | Threaded
Open this post in threaded view
|

Re: t@h/osmarender being phased out (at least from my side)

spaetz
On Mon, 21 Feb 2011 22:14:56 +0100, ben <[hidden email]> wrote:
> oh no! I hope I will hit the Top100 before everything breaks up! ;)

Good luck with that :).
 
> Despite the personal things and disadvantages, do you (and all other
> readers) see any
> reasons for t@h to be alive? I mean, is the effort worth to be done by
> possible "volunteers".

Well, it essentially has been running by volunteers all the time :-).

> if not, then nobody will miss t@h and all are good to go to other projects.
> which leads me to the question: what server specs would be needed? how many
> time should
> be spent to "keep the thing running", etc.?

Well, besides a few GB of fast disk (we are on a SAN array with 1.5TB of
which we use something like 600GB or so, but that changes depending on
how many layers one wants), the server is pretty modest (4GB of RAM,
dual core Intel). It is kept (CPU) busy by stitching the low zoom tiles
continuously, otherwise it wouldn't even be very CPU intensive. It needs
a healthy up and download bandwidth. It has been a long time since I
measured bandwidth, but it has maxed out the 10MBit/s connection up and
down pretty much all the time (or was it 100MBit, well in the ballbark
of 8TB up and 8TB down per Month).

So disk and bandwidth mainly. Also the Read-only mirrors (TRAPIs) that
are required by the clients would need some reinforcement, judging from
the latest mails to this list.
 
> would love to hear any thoughts!

I would be happy to see t@h survive, it is a fun system and as
independent from OSMF as it gets.

Sebastian

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome

attachment0 (203 bytes) Download Attachment
ben
Reply | Threaded
Open this post in threaded view
|

Re: t@h/osmarender being phased out (at least from my side)

ben


On Mon, Feb 28, 2011 at 5:03 PM, Sebastian Spaeth <[hidden email]> wrote:
On Mon, 21 Feb 2011 22:14:56 +0100, ben <[hidden email]> wrote:
> oh no! I hope I will hit the Top100 before everything breaks up! ;)

Good luck with that :).

> Despite the personal things and disadvantages, do you (and all other
> readers) see any
> reasons for t@h to be alive? I mean, is the effort worth to be done by
> possible "volunteers".

Well, it essentially has been running by volunteers all the time :-).

yeah, i know that. maybe I'm not clear enough about this.
i asked this, because potential following project "admins" need time to get in the project and have to learn many things to do the work you do until now.
so if there are no reasons for t@h to exist in the future (and hopefully there are some! any thoughts?), this would be wasted time, if you know what i mean.

ben

_______________________________________________
Tilesathome mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/tilesathome