Re: FW: FW: Recent improvements to the editing applet

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Etienne Cherdlu
Lars Aronsson wrote:
> Currently the tiles (both in viewing and editing) are 256 x 128 pixels.  How
> would performance be different if the tiles were, say, 512 x 256 pixels
> instead?  That would cause fewer tiles and fewer calls to the server, but
> every tile would take a little longer to process.

Initially it appears to fetch 30 tiles (6 x 5) even though 12 (4 x 3)
would be sufficient to fill the viewing area.  At least it could fetch
the important 12 first.  And it could even try to fetch all the nodes
and segments first *before* swamping the tile server with 30 requests
for tiles.

When panning it would seem to also be fetching quite a few more tiles
than are needed.  Starting from the left edge of the new area (most of
which will be off the screen).  It would be better if it started at
the edge that was closest to the tiles that have already been
displayed.

The overhead of so many round trips is not going to help performance.
If the network latency is high then it will be pretty bad.

I'd guess there is some overhead in activating the tile server process
to generate a tile so this will also slow things down.

So, now the server is having to generate 983,040 pixels (256*128*30)
using 30 requests instead of 350,000 pixels (700*500) using 1 request.
 At a minimum it is doing 2.85 times more work than before.  Not good
if I have to wait until its all done before I can do anything.  Even
worse if the server is slow.

Etienne

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Mikel Maron

Good recommendations -- it's worth getting some solid metrics, and determine an optimal tile size. And looking at the load order of tiles, vectors.

I believe the editor/viewer is requesting 5 x 4 tiles, since the bounding box does not overlap the pre-tiling precisely.

----- Original Message ----
From: Etienne Cherdlu <[hidden email]>
To: [hidden email]
Sent: Friday, December 30, 2005 8:21:23 PM
Subject: Re: FW: FW: [Openstreetmap] Recent improvements to the editing applet

Lars Aronsson wrote:
> Currently the tiles (both in viewing and editing) are 256 x 128 pixels.  How
> would performance be different if the tiles were, say, 512 x 256 pixels
> instead?  That would cause fewer tiles and fewer calls to the server, but
> every tile would take a little longer to process.

Initially it appears to fetch 30 tiles (6 x 5) even though 12 (4 x 3)
would be sufficient to fill the viewing area.  At least it could fetch
the important 12 first.  And it could even try to fetch all the nodes
and segments first *before* swamping the tile server with 30 requests
for tiles.

When panning it would seem to also be fetching quite a few more tiles
than are needed.  Starting from the left edge of the new area (most of
which will be off the screen).  It would be better if it started at
the edge that was closest to the tiles that have already been
displayed.

The overhead of so many round trips is not going to help performance.
If the network latency is high then it will be pretty bad.

I'd guess there is some overhead in activating the tile server process
to generate a tile so this will also slow things down.

So, now the server is having to generate 983,040 pixels (256*128*30)
using 30 requests instead of 350,000 pixels (700*500) using 1 request.
 At a minimum it is doing 2.85 times more work than before.  Not good
if I have to wait until its all done before I can do anything.  Even
worse if the server is slow.

Etienne

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap




_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Steve Coast
* @ 31/12/05 11:21:17 AM [hidden email] wrote:
>
> Good recommendations -- it's worth getting some solid metrics, and
> determine an optimal tile size. And looking at the load order of
> tiles, vectors.
>
> I believe the editor/viewer is requesting 5 x 4 tiles, since the
> bounding box does not overlap the pre-tiling precisely.

That's what I remember from coding it, and what I get (20) from testing
it now.

> I'd guess there is some overhead in activating the tile server process
> to generate a tile so this will also slow things down.
>
> So, now the server is having to generate 983,040 pixels (256*128*30)
> using 30 requests instead of 350,000 pixels (700*500) using 1 request.
>  At a minimum it is doing 2.85 times more work than before.  Not good

That's not the case: In the old model Mr. Applet would ask Mr. Server
for an arbitrary image that would most likely never get used again.
Tiles are cached so to some extent will save server load.

have fun,

SteveC [hidden email] http://www.asklater.com/steve/

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

David Sheldon-5
On Sun, Jan 01, 2006 at 10:15:27PM +0000, SteveC wrote:
> That's not the case: In the old model Mr. Applet would ask Mr. Server
> for an arbitrary image that would most likely never get used again.
> Tiles are cached so to some extent will save server load.

Unfortunately this means that when you upload a new route you cannot
edit it/add roads until the tiles have timed out and the yellow dots
appear. My process is to upload a track, then draw on roads. This is now
not as easy as before as you don't necessarily get the track on the
editor straight away.

David
--
Ask me a silly question under threat of a large fine and I will give the
most accurate and unhelpful answer I can manage. J Jones, ox.talk

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Robert Scott-2
On Monday 02 Jan 2006 02:52, David Sheldon wrote:
> Unfortunately this means that when you upload a new route you cannot
> edit it/add roads until the tiles have timed out and the yellow dots
> appear. My process is to upload a track, then draw on roads. This is now
> not as easy as before as you don't necessarily get the track on the
> editor straight away.
>
> David

Indeed; I would be able to edit a lot faster if I didn't have to stop and wait
a few hours for another clump of my breadcrumbs to appear. Most of the time
breadcrumb trails disappear in the halfway down a road (one tile is cached,
another isn't) and I just can't go any further.

It's also particularly difficult if you're trying to do an area you don't know
too well. If you're able to work on it while the trip is still fresh in your
memory, it really helps. Especially with road names.

But of course I understand how it's a tradeoff with performance (which seems
to have stepped up in the last ~24 hours).


robert.

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Matt Amos-2
On Monday 02 January 2006 03:29, Robert Scott wrote:

> On Monday 02 Jan 2006 02:52, David Sheldon wrote:
> > Unfortunately this means that when you upload a new route you
> > cannot edit it/add roads until the tiles have timed out and the
> > yellow dots appear. My process is to upload a track, then draw on
> > roads. This is now not as easy as before as you don't necessarily
> > get the track on the editor straight away.
> >
> > David
>
> Indeed; I would be able to edit a lot faster if I didn't have to
> stop and wait a few hours for another clump of my breadcrumbs to
> appear. Most of the time breadcrumb trails disappear in the halfway
> down a road (one tile is cached, another isn't) and I just can't go
> any further.
would it be better to request three tiles, e.g:
1) the opaque "background" tile from landsat
2) the breadcrumbs on a transparent PNG
3) the roads on a transparent PNG

that way the client could deliberately request (2) and (3) with a
special form of URL (maybe appending ?cache=no or something) which
would have the side effect of invalidating the server cache for just
those tiles?

an alternative idea is that the client could re-render the tiles
itself (and possibly upload them) so you'd always have the latest
information on your screen.

anybody fancy implementing a REST-ful API for the tiles? something
like http://www.openstreetmap.org/api/0.2/tile/4/2,3 gives the tile
x=2, y=3 for zoom level 4. then you could PUT the rendered tiles
there and have the server backend figure out how to invalidate the
cache...

> It's also particularly difficult if you're trying to do an area you
> don't know too well. If you're able to work on it while the trip is
> still fresh in your memory, it really helps. Especially with road
> names.
>
> But of course I understand how it's a tradeoff with performance
> (which seems to have stepped up in the last ~24 hours).

this is especially difficult when there is a bandwidth/cpu trade-off,
as it is sometimes not clear how server performance is scaling with
I/O or compute.

cya,

matt

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap

attachment0 (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Erik Johansson-2
On 1/4/06, Matt Amos <[hidden email]> wrote:

> On Monday 02 January 2006 03:29, Robert Scott wrote:
> > On Monday 02 Jan 2006 02:52, David Sheldon wrote:
> > > Unfortunately this means that when you upload a new route you
> > > cannot edit it/add roads until the tiles have timed out and the
> > > yellow dots appear. My process is to upload a track, then draw on
> > > roads. This is now not as easy as before as you don't necessarily
> > > get the track on the editor straight away.
> > >
> > > David
> >
> > Indeed; I would be able to edit a lot faster if I didn't have to
> > stop and wait a few hours for another clump of my breadcrumbs to
> > appear. Most of the time breadcrumb trails disappear in the halfway
> > down a road (one tile is cached, another isn't) and I just can't go
> > any further.
>
> would it be better to request three tiles, e.g:
> 1) the opaque "background" tile from landsat
> 2) the breadcrumbs on a transparent PNG
> 3) the roads on a transparent PNG

This was the method used for the static version of the map before.
Your browser would fetch the landsat from some nasa server and then
fetch the roads as a seperate layer from OSM. I don't know why they
aren't fetched seperatly now days.

I don't think there is much speed up to be gained from this method,
and the bandwidth will be doubled. But you are right it's alot cleaner
way fo doing things.

--
/Erik

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Thomas Walraet
Erik Johansson wrote:
>
> I don't think there is much speed up to be gained from this method,
> and the bandwidth will be doubled. But you are right it's alot cleaner
> way fo doing things.

The bandwith overhead is not that simple to evaluate.

Landsat picture will stay longueur in browser and proxy caches. Road
picture will be real small (few color, large uniform color surfaces)

And it allow to easily balance the load by putting landsat and roads on
different servers.


Another idea : by using separate pictures, it could be easier to code a
system to re-use landsat images from one zoom level to another. (It was
suggested somewhere on the wiki)

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Robert Scott-2
In reply to this post by Erik Johansson-2
On Wednesday 04 Jan 2006 18:36, Erik Johansson wrote:
> I don't think there is much speed up to be gained from this method,
> and the bandwidth will be doubled. But you are right it's alot cleaner
> way fo doing things.

I'm quite sure it wouldn't be doubled. The breadcrumb layer would probably
compress out to almost nothing with PNG and a limited palette. (Who says we
need full alpha blending? Could use a paletted image with orange and a
transparent colour. Effectively a 1bpp image.)


-robert.

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Tom Carden
In reply to this post by Matt Amos-2
On 1/4/06, Matt Amos <[hidden email]> wrote:
>
> would it be better to request three tiles, e.g:
> 1) the opaque "background" tile from landsat
> 2) the breadcrumbs on a transparent PNG
> 3) the roads on a transparent PNG
>

I suggested a move back to this method for the viewer a while ago (I
think there's a ticket for it too!
http://www.openstreetmap.org/trac/ticket/42).  So then the viewer and
the applet can share a cache - the viewer drawing its own editable
vectors

It would be a really nice thing to be able to cache landsat
indefinitely - sometimes the NASA server is really slow.

Also - as others have mentioned - this frees up the possibility of
distributing the load, and of compressing each layer more efficiently
(it certainly wouldn't be double the bandwidth).  What hasn't been
mentioned is that it also gives the client the ability to choose which
things are shown - a generalised layering interface would be a really
useful thing.  I for one would like to show the GPS tracks in the
viewer (so I know where needs editing), and I know others would like
to display things from external WMS servers.

Steve - what were the disadvantages of client-side compositing?  Did
it just get dropped because of the move to the (originally civicmaps)
tile script?

> that way the client could deliberately request (2) and (3) with a
> special form of URL (maybe appending ?cache=no or something) which
> would have the side effect of invalidating the server cache for just
> those tiles?
>
> an alternative idea is that the client could re-render the tiles
> itself (and possibly upload them) so you'd always have the latest
> information on your screen.
>

You can force refresh with your browser (ctrl-refresh or alt-refresh I
think, in Firefox at least) - I assume that's an HTTP header for a GET
request... the applet could issue those in the background for any
tiles where vectors have been edited, and (perhaps only for logged in
users) I'm pretty sure we could offer a 'refresh visible tiles' button
on the viewer.  (Thought - can an HTTP HEAD request refresh a cached
image without downloading it?)

> anybody fancy implementing a REST-ful API for the tiles? something
> like http://www.openstreetmap.org/api/0.2/tile/4/2,3 gives the tile
> x=2, y=3 for zoom level 4. then you could PUT the rendered tiles
> there and have the server backend figure out how to invalidate the
> cache...
>

That would be neat, but do we really want clients in charge of rendering?

T.

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Erik Johansson-2
The nice thing about the separation of the layers is that there seems
to be other landsatservers around, and some seems to be fast. Choose,
landsat in the options bellow the map:

http://wms1.ccgis.de/mapbender2/frames/index.php?&gui_id=mapbender


On 1/5/06, Tom Carden <[hidden email]> wrote:
> Also - as others have mentioned - this frees up the possibility of
> distributing the load, and of compressing each layer more efficiently
> (it certainly wouldn't be double the bandwidth).

Pride really is a bad thing, but I should learn to motivate flametory
statements. Well sometimes worse, as someone said it doesn't need to
be RGBA, that's probably why it's so large.

#!/bin/bash
url='http://tile.openstreetmap.org/cgi-bin/steve/mapserv?map=/usr/lib/cgi-bin/steve/wms.map&service=WMS&WMTVER=1.0.0&REQUEST=map&STYLES=&TRANSPARENT=TRUE&WIDTH=256&HEIGHT=128&BBOX'
bbx=11.25,61.60639637138628,22.5,64.16810689799155
l='LAYERS'
layer=landsat
curl  -s "$url=$bbx&$l=$layer" |wc -c
layer=streets
curl -s "$url=$bbx&$l=$layer" |wc -c
bbx='-2.8125,52.05249047600099,-1.40625,52.482780222078226'
layer=landsat
curl -s "$url=$bbx&$l=$layer" |wc -c
layer=streets
curl -s "$url=$bbx&$l=$layer" |wc -c




--
/Erik

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Matt Amos-2
In reply to this post by Robert Scott-2
On Wednesday 04 January 2006 20:44, Robert Scott wrote:

> On Wednesday 04 Jan 2006 18:36, Erik Johansson wrote:
> > I don't think there is much speed up to be gained from this
> > method, and the bandwidth will be doubled. But you are right it's
> > alot cleaner way fo doing things.
>
> I'm quite sure it wouldn't be doubled. The breadcrumb layer would
> probably compress out to almost nothing with PNG and a limited
> palette. (Who says we need full alpha blending? Could use a
> paletted image with orange and a transparent colour. Effectively a
> 1bpp image.)
this is a very good idea! the landsat has to be RGB, obviously, but
the breadcrumbs could be just an A channel (allowing the user to
choose the opaque colour!) and the roads can be a 256 colour image
which blends from:

0 => transparent black
.5 => opaque black
1 => opaque white

since we always have a black border between the white road and the
transparent non-road. this would also allow people to choose hue for
the roads if they wanted...

of course, this doesn't work if we want motorways to be blue, etc...
but i guess you could put an extra H channel in there.

so we have one RGB image, an A image and an AH image, which is twice
as much information, landsat compresses well with JPEG and the A, AH
images will do well with PNG, due to their large transparent regions.

but we can't say for definite without anl implementation ;-)

cya,

matt

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap

attachment0 (197 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: FW: FW: Recent improvements to the editing applet

Ben Gimpert
In reply to this post by Tom Carden
On Wed, Jan 04, 2006 at 11:19:20PM +0000, Tom Carden wrote:
> > anybody fancy implementing a REST-ful API for the tiles? something
> > like http://www.openstreetmap.org/api/0.2/tile/4/2,3 gives the tile
> > x=2, y=3 for zoom level 4. then you could PUT the rendered tiles
> > there and have the server backend figure out how to invalidate the
> > cache...
> >
>
> That would be neat, but do we really want clients in charge of rendering?

I'm not sure if this is pure evil or not: Coders implement an
alternative client atop the OSM REST API. Their client renders in a
different (snazzy or strange) way, and then incrementally stomps the
rendering of whatever parts of the world are viewed via that client.
These different renderings [cw]ould be versioned, the Wiki Way(TM).
This might be a feature.

                Ben


_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap