Wiki down?

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Wiki down?

frank mohr
tried to add some text to the wiki, but got realy bad response
times first (about 1/2 h ago), now it's unreachable

frank

       

       
               
___________________________________________________________
Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de


_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Tom Carden
On 1/6/06, Frank Mohr <[hidden email]> wrote:
> tried to add some text to the wiki, but got realy bad response
> times first (about 1/2 h ago), now it's unreachable

Yeah, it's very patchy for me too at the moment.

It runs on the same server as my site, which recently hit the front
page of digg and was linked from spiegel.de - it could have something
to do with that, since my site is pretty much down now too :(

I'm looking into moving my hosting as soon as possible, at least for
big applets and the like, partly as a favour to openstreetmap and
partly so it will be somewhere that can cope with big bursts of
activity, which bat really can't.

Best,

Tom.

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Jo Walsh
On Fri, Jan 06, 2006 at 02:16:38PM +0000, Tom Carden wrote:
> It runs on the same server as my site, which recently hit the front
> page of digg and was linked from spiegel.de - it could have something
> to do with that, since my site is pretty much down now too :(
>
> I'm looking into moving my hosting as soon as possible, at least for
> big applets and the like, partly as a favour to openstreetmap and
> partly so it will be somewhere that can cope with big bursts of
> activity, which bat really can't.

There is a server that Saul and I purchased with media arts dosh
which is being used for freemap.in and not much else right now - we
set it up during Wsfii, then it had a hardware failure, then we had a
change control failure. It has decent bandwidth. I would like to set
up an OSM mirror/instance on it anyway. This is an open offer...


-jo

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

frank mohr
In reply to this post by Tom Carden
Tom Carden wrote:
> On 1/6/06, Frank Mohr <[hidden email]> wrote:
>
>>tried to add some text to the wiki, but got realy bad response
>>times first (about 1/2 h ago), now it's unreachable
>
>
> Yeah, it's very patchy for me too at the moment.

it's better now .. to good that i've edited my text in vi first

> It runs on the same server as my site, which recently hit the front
> page of digg and was linked from spiegel.de - it could have something
> to do with that, since my site is pretty much down now too :(

spiegel.de hit OSM or a different site on the same server?
OSM would be the right time as i'm currently adding
"WikiProject Germany"

Frank

       

       
               
___________________________________________________________
Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de


_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Tom Carden
> On 1/6/06, Frank Mohr <[hidden email]> wrote:
>
> spiegel.de hit OSM or a different site on the same server?
> OSM would be the right time as i'm currently adding
> "WikiProject Germany"
>

Unfortunately not OSM.  My interactive London tube map -
http://www.spiegel.de/netzwelt/netzkultur/0,1518,393787,00.html

Best,

Tom.


_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Steve Coast
In reply to this post by Jo Walsh
* @ 06/01/06 02:33:14 PM [hidden email] wrote:

> On Fri, Jan 06, 2006 at 02:16:38PM +0000, Tom Carden wrote:
> > It runs on the same server as my site, which recently hit the front
> > page of digg and was linked from spiegel.de - it could have something
> > to do with that, since my site is pretty much down now too :(
> >
> > I'm looking into moving my hosting as soon as possible, at least for
> > big applets and the like, partly as a favour to openstreetmap and
> > partly so it will be somewhere that can cope with big bursts of
> > activity, which bat really can't.
>
> There is a server that Saul and I purchased with media arts dosh
> which is being used for freemap.in and not much else right now - we
> set it up during Wsfii, then it had a hardware failure, then we had a
> change control failure. It has decent bandwidth. I would like to set
> up an OSM mirror/instance on it anyway. This is an open offer...

This would be ++useful if we can put the box with the others. It's of
limited use if it's far away from the database machine etc. That
possible?

have fun,

SteveC [hidden email] http://www.asklater.com/steve/

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Jo Walsh
On Fri, Jan 06, 2006 at 04:03:07PM +0000, SteveC wrote:
> > There is a server that Saul and I purchased with media arts dosh
> > which is being used for freemap.in and not much else right now - we
> > set it up during Wsfii, then it had a hardware failure, then we had a
> > change control failure. It has decent bandwidth. I would like to set
> > up an OSM mirror/instance on it anyway. This is an open offer...
>
> This would be ++useful if we can put the box with the others. It's of
> limited use if it's far away from the database machine etc. That
> possible?

Right now it lives in the rack in fremont.ca.us that holds geocoder.us and
my/Schuyler's personal box; this is a good situation for it because it
is physically near Rich if it needs to be kicked or cosseted.

I know jack all about database replication. ISTR mysql is easy to do a
master-slave type configuration but not peer-peer ... this may have changed.

Have you considered something like "sharding", like the MMORPGs have
to do when they get overfull? And synchronise the shards every N
hours. I wonder how complex this could wind up. A hard one, how much
complexity is necessary complexity? :)

Does any of the OGC prior art even address this problem, i wonder?


-jo




_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

David Sheldon-5
On Fri, Jan 06, 2006 at 08:27:43AM -0800, Jo Walsh wrote:
> Have you considered something like "sharding", like the MMORPGs have
> to do when they get overfull? And synchronise the shards every N
> hours. I wonder how complex this could wind up. A hard one, how much
> complexity is necessary complexity? :)

Just a thought, and this might be what you are refering to as sharding,
but could we redirect based on the lattitude/longditude, and
serve/store all the American (continent) points/map segments in the US,
and the European points/map segments in the UK.

Is it possible to pick lines down the middle of the Atlantic and
pacific such that no road will cross them?

I guess this wouldn't help if all of the new points are in Europe as all
the load would still be on the same machine.

I haven't really thought about the complexity either but it does sound
interesting. Would they even need syncing, or would that be just for
backups and stats?

David
--
"We can't possibly do that. Who'd clear up the mess?" - Rimmer gone nuts.

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Lars Aronsson
David Sheldon wrote:

> On Fri, Jan 06, 2006 at 08:27:43AM -0800, Jo Walsh wrote:
> > Have you considered something like "sharding", like the MMORPGs have
> > to do when they get overfull? And synchronise the shards every N
> > hours. I wonder how complex this could wind up. A hard one, how much
> > complexity is necessary complexity? :)
>
> Just a thought, and this might be what you are refering to as sharding,
> but could we redirect based on the lattitude/longditude, and
> serve/store all the American (continent) points/map segments in the US,
> and the European points/map segments in the UK.

It is way too early in the project for anything like this.  
Slicing the application over multiple servers will increase
complexity tremendously, and we don't want to spend our time
designing complex replication.  Instead we need to keep complexity
down, so the project can grow at constant cost, both in the amount
of data and in the number of users (editors and viewers).

My guess is that OSM currently never sees more than 10
simultaneous map editors.  In fact I believe that I'm often alone
with the server when I'm editing maps, and it's still slow. To get
useful maps, we need a lot of map data, and that will require many
times more contributors than we have now.  I would say the
successful project needs a system that can handle a few thousand
simultaneous editors.  Think Wikipedia scale.

If OSM currently experiences an increase from 10 to 20 users, and
if this would require the slicing over 2 servers, then the
increase from 1000 to 2000 users would require 200 servers, which
simply is unrealistic.  Who would pay for that?  Not me!

If HTTP requests (for map tiles or whatever) arrive at the server
at a rate of X per second, then the server had better spend less
than 1/X second on each request, so that it is free to receive and
process the next request as it arrives.  The response time must be
pressed down.  This should be our focus.  Trim away the fat.

I think that the green Landsat images should be served separately
from the white lines or yellow dots.  The green images can then be
cached indefinitely or served from another server or turned off.
White lines and yellow dots should be cached as little as
possible, so that updates are reflected with minimum delay. White
lines can be cached (two days or so) for non-logged in users, but
active (logged in) map editors need to see updates immediately.

I also guess that things would become a little faster with larger
tiles, but this could be marginal.

But most of the slowness right now probably comes from some
trivial inefficiency (string buffer copying? XML libraries?  ODBC
bandwidth?) that can become 10 or 100 times faster with the right
analysis.

The trick now is to set a target for the HTTP response time, say
0.1 seconds, and let the server software issue a log message
whenever a request takes much longer, say .3 seconds.  Zero
tolerance on slow requests is the way to go.  Not only do slow
requests annoy the user, but they also block the server (which we,
for all practical purposes, can assume is single-threaded), so it
cannot handle other requests.


--
  Lars Aronsson ([hidden email])
  Aronsson Datateknik - http://aronsson.se

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Gabriel Ebner
On Fri, Jan 06, 2006 at 08:15:27PM +0100, Lars Aronsson wrote:
> My guess is that OSM currently never sees more than 10
> simultaneous map editors.  In fact I believe that I'm often alone
> with the server when I'm editing maps, and it's still slow. To get
> useful maps, we need a lot of map data, and that will require many
> times more contributors than we have now.  I would say the
> successful project needs a system that can handle a few thousand
> simultaneous editors.  Think Wikipedia scale.

I doubt you're the only one who is currently editing maps.  E.g. I have been
trying to upload some 13,000 nodes+segments for the last few days (with some
pauses, after connections timed out and while editing the maps offline).
Considering that there are still around 11,103 left. which makes about 7 days
at 55 sec/PUT, so you surely won't be alone for yet another week at least
if recent access times will last.

BTW, as far as I understand the API, one can only put 1 segment or node per
PUT; would it make any sense to allow multiple nodes or segments to be
transferred at a time?

> If OSM currently experiences an increase from 10 to 20 users, and
> if this would require the slicing over 2 servers, then the
> increase from 1000 to 2000 users would require 200 servers, which
> simply is unrealistic.  Who would pay for that?  Not me!

It would be interesting to see where the load comes from.  I can't imagine
that 10 users accessing a database via HTTP would result in so much load.  Is
it the viewer?  Landsat tiles? (they are relatively fast, taking just a few
secs)

  Gabriel.

P.S.: After having read the sources, it seems to be possible to insert either
multiple nodes or multiple segments by just having multiple children inside
the <osm/>, resulting in all their assigned IDs being returned in order.  Is
this official?  If so, should it be documented in the wiki?

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Wiki down?

Gabriel Ebner
On Fri, Jan 06, 2006 at 09:52:53PM +0000, Gabriel Ebner wrote:
> P.S.: After having read the sources, it seems to be possible to insert either
> multiple nodes or multiple segments by just having multiple children inside
> the <osm/>, resulting in all their assigned IDs being returned in order.  Is
> this official?  If so, should it be documented in the wiki?

Sending multiple nodes per request seems to work, I'm seeing 10- to 20-fold
speed increases (1x 55s vs. 10x 55s).  However the limit of nodes per request
seems to lie somewhere between 20 and 30 nodes, anything above that currently
lets the connection time out.

  Gabriel.


_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap

attachment0 (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Wiki down?

Nick Whitelegg
?
>
> Sending multiple nodes per request seems to work, I'm seeing 10- to 20-fold
> speed increases (1x 55s vs. 10x 55s).  However the limit of nodes per
> request seems to lie somewhere between 20 and 30 nodes, anything above that
> currently lets the connection time out.

You can indeed send multiple nodes per request, osmeditor does this when you
upload a batch of new data for the first time (useful for if you want to edit
offline and upload stuff in one go)

I've had varying success with it. Sometimes I have uploaded well over 20-30
nodes successfully, other times (when server load has been heavy?) I've had
no response even from just a few nodes.

Nick

_______________________________________________
Openstreetmap mailing list
[hidden email]
http://bat.vr.ucl.ac.uk/cgi-bin/mailman/listinfo/openstreetmap