Any tile statistics (unique visitors)

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Any tile statistics (unique visitors)

Stefan de Konink-3
Where could we find some unique visitor statistics for the tileserver?


Stefan

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Tom Hughes-3
Stefan de Konink wrote:
> Where could we find some unique visitor statistics for the tileserver?

Nowhere.

As I'm sure you're aware the only way to produce such a number (well a
rough approximation to such a number anyway) is to issue cookies, and
the tile server doesn't issue any.

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Stefan de Konink-3
Tom Hughes wrote:
> As I'm sure you're aware the only way to produce such a number (well a
> rough approximation to such a number anyway) is to issue cookies, and
> the tile server doesn't issue any.

The best way is to create a log for one of the icons of the map that
everyone visits, and just issue awstats over it.


Stefan

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Tom Hughes-3
Stefan de Konink wrote:
> Tom Hughes wrote:
>> As I'm sure you're aware the only way to produce such a number (well a
>> rough approximation to such a number anyway) is to issue cookies, and
>> the tile server doesn't issue any.
>
> The best way is to create a log for one of the icons of the map that
> everyone visits, and just issue awstats over it.

What is an "icon of the map that everybody visits" exactly? We serve
tiles, and there is no one tile that every person will fetch. Plus most
people will fetch the same tile multiple times.

Why do you want this information anyway?

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Stefan de Konink-3
Tom Hughes wrote:
> What is an "icon of the map that everybody visits" exactly? We serve
> tiles, and there is no one tile that every person will fetch. Plus most
> people will fetch the same tile multiple times.

http://www.openstreetmap.org/openlayers/img/north-mini.png

Since unique visitors per day is a think awstats can do for us, there
are less things to worry about :)


> Why do you want this information anyway?

Because the person/organisation that gave OSM-NL last year around 10K in
euro, and now gave 28K to OpenStreetPhoto would like to know the unique
visitors, and I think that is a reasonable question...


Stefan

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Tom Hughes-3
Stefan de Konink wrote:
> Tom Hughes wrote:
>> What is an "icon of the map that everybody visits" exactly? We serve
>> tiles, and there is no one tile that every person will fetch. Plus
>> most people will fetch the same tile multiple times.
>
> http://www.openstreetmap.org/openlayers/img/north-mini.png
>
> Since unique visitors per day is a think awstats can do for us, there
> are less things to worry about :)

Well if you want to understand what it means then you need to know what
the calculation is behind what awstat is doing! In this case I assume
that it is telling you the number of unique IP addresses if there is no
tracking cookie but that is a very poor approximation for the a number
of visitors.

Plus you asked about unique visitors to the tile server but now you want
to know about unique visitors to our slippy map, which is an entirely
different question as not everybody that uses our tile server uses our
slippy map.

>> Why do you want this information anyway?
>
> Because the person/organisation that gave OSM-NL last year around 10K in
> euro, and now gave 28K to OpenStreetPhoto would like to know the unique
> visitors, and I think that is a reasonable question...

Every question seems reasonable to the person asking it, but as always
the question is where to draw the line - do I have to conjure up
whatever statistics people ask for no matter how much work it needs?

Anyway, if you can get the appropriate people to authorise it, and you
can clarify exactly what you want, then we might be able to do something
when we get a minute.

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Stefan de Konink-3
Tom Hughes wrote:
> Plus you asked about unique visitors to the tile server but now you want
> to know about unique visitors to our slippy map, which is an entirely
> different question as not everybody that uses our tile server uses our
> slippy map.

So do you agree with me that the minimal amount of unique visiters
exceeds the amount of users on the slippy map? Could you provide how
many users are on the slippy map?

>>> Why do you want this information anyway?
>>
>> Because the person/organisation that gave OSM-NL last year around 10K
>> in euro, and now gave 28K to OpenStreetPhoto would like to know the
>> unique visitors, and I think that is a reasonable question...
>
> Every question seems reasonable to the person asking it, but as always
> the question is where to draw the line - do I have to conjure up
> whatever statistics people ask for no matter how much work it needs?

You are always talking about you are the only person capable of doing
something. I sincerely wonder why that is so?


> Anyway, if you can get the appropriate people to authorise it, and you
> can clarify exactly what you want, then we might be able to do something
> when we get a minute.

And who might that be? Since when should I find authorisation for questions?



Stefan

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Tom Hughes-3
Stefan de Konink wrote:
> Tom Hughes wrote:
>> Plus you asked about unique visitors to the tile server but now you
>> want to know about unique visitors to our slippy map, which is an
>> entirely different question as not everybody that uses our tile server
>> uses our slippy map.
>
> So do you agree with me that the minimal amount of unique visiters
> exceeds the amount of users on the slippy map? Could you provide how
> many users are on the slippy map?

Well yes, but I still think that whatever awstat does to get this count
will be a very poor approximation to unique visitors.

>>>> Why do you want this information anyway?
>>>
>>> Because the person/organisation that gave OSM-NL last year around 10K
>>> in euro, and now gave 28K to OpenStreetPhoto would like to know the
>>> unique visitors, and I think that is a reasonable question...
>>
>> Every question seems reasonable to the person asking it, but as always
>> the question is where to draw the line - do I have to conjure up
>> whatever statistics people ask for no matter how much work it needs?
>
> You are always talking about you are the only person capable of doing
> something. I sincerely wonder why that is so?

I'm not the only one, but of the people with access to the relevant
server I'm the one most likely to do it if we do it from the slippy map
logs. If we do it from the tile server logs then Jon would be more
likely (though we both could do it from either machine).

If what you're asking is why we don't just give anybody who asks root
access to the web server then I hope that would answer itself.

>> Anyway, if you can get the appropriate people to authorise it, and you
>> can clarify exactly what you want, then we might be able to do
>> something when we get a minute.
>
> And who might that be? Since when should I find authorisation for
> questions?

I'm not sure to be honest - the foundation I guess. I don't believe we
have a policy on the release of statistical information like this at the
moment.

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Stefan de Konink-3
Tom Hughes wrote:
> If what you're asking is why we don't just give anybody who asks root
> access to the web server then I hope that would answer itself.

Read-rights on my logs are available to anyone :)

>>> Anyway, if you can get the appropriate people to authorise it, and
>>> you can clarify exactly what you want, then we might be able to do
>>> something when we get a minute.
>>
>> And who might that be? Since when should I find authorisation for
>> questions?
>
> I'm not sure to be honest - the foundation I guess. I don't believe we
> have a policy on the release of statistical information like this at the
> moment.

Who is 'we'? And in what respect is that group 'limited' in doing something?


Stefan

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Ian Dees
In reply to this post by Tom Hughes-3
Tom,

It would be pretty neat to see a map of the number of times a certain tile is served. While it certainly wouldn't be any measure of visitors, it would be a pretty nifty visualization of the "busy" areas of the planet.

On Sat, Dec 6, 2008 at 5:51 PM, Tom Hughes <[hidden email]> wrote:
Stefan de Konink wrote:
> Where could we find some unique visitor statistics for the tileserver?

Nowhere.

As I'm sure you're aware the only way to produce such a number (well a
rough approximation to such a number anyway) is to issue cookies, and
the tile server doesn't issue any.


_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Stefan de Konink-3
Ian Dees wrote:
> It would be pretty neat to see a map of the number of times a certain
> tile is served. While it certainly wouldn't be any measure of visitors,
> it would be a pretty nifty visualization of the "busy" areas of the planet.

Wouldn't that be able to generated from anything that is served using
things as mod_tile or tilecache by uri?


Stefan


_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Jochen123
In reply to this post by Stefan de Konink-3
On Sun, Dec 07, 2008 at 01:36:05AM +0100, Stefan de Konink wrote:
> Tom Hughes wrote:
> > If what you're asking is why we don't just give anybody who asks root
> > access to the web server then I hope that would answer itself.
>
> Read-rights on my logs are available to anyone :)

I hope you have properly anonymized them beforehand! Even anonymized,
the logs of a tile server are problematic in a privacy sense. You can
probably identify some individuals from the session data.

Jochen
--
Jochen Topf  [hidden email]  http://www.remote.org/jochen/  +49-721-388298


_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Tom Hughes-3
In reply to this post by Ian Dees
Ian Dees wrote:

> It would be pretty neat to see a map of the number of times a certain
> tile is served. While it certainly wouldn't be any measure of visitors,
> it would be a pretty nifty visualization of the "busy" areas of the planet.

Sure it would be. I've even thought of doing it but so far my supply of
round tuits has been insufficient.

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Tom Hughes-3
In reply to this post by Stefan de Konink-3
Stefan de Konink wrote:
> Tom Hughes wrote:
>> If what you're asking is why we don't just give anybody who asks root
>> access to the web server then I hope that would answer itself.
>
> Read-rights on my logs are available to anyone :)

Well good for you. In this country we take protecting people's personal
data a bit more seriously that you apparently do.

Tom

--
Tom Hughes ([hidden email])
http://www.compton.nu/

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

spaetz
In reply to this post by Stefan de Konink-3
Stefan de Konink wrote:
> http://www.openstreetmap.org/openlayers/img/north-mini.png
>
> Since unique visitors per day is a think awstats can do for us, there
> are less things to worry about :)

That assumes all unique visitors actually use a webbrowser, and from t@h
stats I can tell that much of the traffic comes from bots and scripts
rather than just web browsers.

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Frederik Ramm
In reply to this post by Ian Dees
Hi,

Ian Dees wrote:
> It would be pretty neat to see a map of the number of times a certain
> tile is served. While it certainly wouldn't be any measure of visitors,
> it would be a pretty nifty visualization of the "busy" areas of the planet.

I believe Andy Allan showed such pretty pictures made from Cycle Map
access statistics at this year's SOTM.

Bye
Frederik

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Frederik Ramm
In reply to this post by spaetz
Hi,

Sebastian Spaeth wrote:
> That assumes all unique visitors actually use a webbrowser, and from t@h
> stats I can tell that much of the traffic comes from bots and scripts
> rather than just web browsers.

I think the Foundation has recently discussed ceasing tile-serving for
non-openstreetmap.org originating requests, and during that discussion
it was found that only about 10% of tile traffic comes from
non-openstreetmap.org sites, which to me sounds like 90% of tile traffic
comes from our own slippy map (unless we have other openstreetmap.org
hosted services that consume tiles?). Unique users need not necessarily
be proportional to tile volume, but I tend to think that a figure like
this would make the bots/scripts you talk about negligible. I'm not
entirely sure about the 10% and what exactly it related to but it should
be findable in the Foundation board meeting minutes.

Bye
Frederik

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Frederik Ramm
In reply to this post by Tom Hughes-3
Hi,

Tom Hughes wrote:
> Well good for you. In this country we take protecting people's personal
> data a bit more seriously that you apparently do.

Which is good, given the enormous amount of people's personal data that
is collected in your country in the first place. May I suggest that you
could just go with tradition and mislay an external hard drive full of
log file data somewhere on a train ;-)

But seriously, people demanding - or politely asking for - Apache log
files is a topic that has been with us for as long as I can think. Steve
  once made a log file with API requests available in an anonymized way:

http://wiki.openstreetmap.org/index.php/Database#Data_statistics

And he also put a script there that does the anonymising. Maybe this can
be repeated once every few months, and maybe it can also be run on tile
access stats? I don't see why the creation of nifty access intensity
images should be restricted to people with root@osm - that's what we
have a crowd for. Also, speaking as someone remotely involved in the
setup of a national tile server architecture, such numbers would give
people in various countries an idea of how much traffic to plan for once
they set up their own servers.

If one can manage to ignore the rather demanding and not exactly polite
way in which Stefan was asking for information... the availability of
anonymized log files, not even permanently, just a snapshot every now
and then, would be a good for us all and I cannot think of a valid
reason not to provide them, except of course the "I am the only one who
has access and I have no time" argument, which, if used, should lead to
granting access to additional trustworthy people who do have the time.

Bye
Frederik



_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

randomjunk
In reply to this post by Frederik Ramm
2008/12/7 Frederik Ramm <[hidden email]>:

> Hi,
>
> Ian Dees wrote:
>> It would be pretty neat to see a map of the number of times a certain
>> tile is served. While it certainly wouldn't be any measure of visitors,
>> it would be a pretty nifty visualization of the "busy" areas of the planet.
>
> I believe Andy Allan showed such pretty pictures made from Cycle Map
> access statistics at this year's SOTM.
>

Yes, I made a (ridiculously inefficient) slippy heat map based on
cycle map apache logs for a given day/period. It was primarily done to
find areas people were looking for tiles but we didn't have them (ie:
the 404 responses), so we could add the areas to our render list. We
don't need that now because we cover everywhere :-)

Dave

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
Reply | Threaded
Open this post in threaded view
|

Re: Any tile statistics (unique visitors)

Joerg Ostertag (OSM Tettnang/Germany)
In reply to this post by Stefan de Konink-3
> > Anyway, if you can get the appropriate people to authorise it, and you
> > can clarify exactly what you want, then we might be able to do something
> > when we get a minute.
>
> And who might that be? Since when should I find authorisation for
> questions?

I can't give you the numbers for osm.org, but i can give you some
numbers for map.openstreetmap.de. Openstreetmap.de is on of the websites
using the tiles.openstreetmap.org.

I simply did something like:
        cat access.log | grep map-access | sed ...ip-only | sort -u | wc -l
on each weekly logfile.
So the reult are all all unique ip-addresses in the given Time range.
IPs is the value of all unique IP Addresses on the whole website.
IPs on map are those unique IP Addresse who where accessing the any page with the map.


I hope these numbers will give you


Joerg


07/Dec/2008:06:36:46 +0100      07/Dec/2008:11:04:08 +0100      IPs: 179        IPs on map: 149
30/Nov/2008:06:31:45 +0100      07/Dec/2008:06:21:57 +0100      IPs: 8282       IPs on map: 7527
23/Nov/2008:06:31:22 +0100      30/Nov/2008:06:21:16 +0100      IPs: 10091      IPs on map: 9345
16/Nov/2008:06:36:39 +0100      23/Nov/2008:06:30:44 +0100      IPs: 10997      IPs on map: 10319
09/Nov/2008:06:30:58 +0100      16/Nov/2008:06:21:49 +0100      IPs: 12519      IPs on map: 11836
02/Nov/2008:06:35:10 +0100      09/Nov/2008:06:29:45 +0100      IPs: 11892      IPs on map: 11157
26/Oct/2008:06:33:52 +0100      02/Nov/2008:06:28:05 +0100      IPs: 12648      IPs on map: 11925
19/Oct/2008:06:31:34 +0200      26/Oct/2008:06:29:13 +0100      IPs: 18186      IPs on map: 17423
12/Oct/2008:06:36:29 +0200      19/Oct/2008:06:23:58 +0200      IPs: 8259       IPs on map: 7617
05/Oct/2008:06:35:29 +0200      12/Oct/2008:06:29:10 +0200      IPs: 9029       IPs on map: 8394
28/Sep/2008:06:32:12 +0200      05/Oct/2008:06:31:28 +0200      IPs: 11439      IPs on map: 10821
21/Sep/2008:06:32:03 +0200      28/Sep/2008:06:27:17 +0200      IPs: 8710       IPs on map: 8072
14/Sep/2008:06:31:42 +0200      21/Sep/2008:06:29:48 +0200      IPs: 8912       IPs on map: 8214
07/Sep/2008:06:31:10 +0200      14/Sep/2008:06:25:17 +0200      IPs: 9591       IPs on map: 8924
31/Aug/2008:06:34:10 +0200      07/Sep/2008:06:29:12 +0200      IPs: 10140      IPs on map: 9491
24/Aug/2008:06:30:40 +0200      31/Aug/2008:06:27:59 +0200      IPs: 12287      IPs on map: 11642
17/Aug/2008:06:30:22 +0200      24/Aug/2008:06:23:51 +0200      IPs: 13001      IPs on map: 12371
10/Aug/2008:06:29:02 +0200      17/Aug/2008:06:28:01 +0200      IPs: 9177       IPs on map: 8617
03/Aug/2008:06:29:33 +0200      10/Aug/2008:06:28:31 +0200      IPs: 10358      IPs on map: 9810
27/Jul/2008:06:31:31 +0200      03/Aug/2008:06:28:13 +0200      IPs: 8241       IPs on map: 7655
20/Jul/2008:06:38:12 +0200      27/Jul/2008:06:26:44 +0200      IPs: 7510       IPs on map: 6850
13/Jul/2008:06:35:19 +0200      20/Jul/2008:06:26:55 +0200      IPs: 7834       IPs on map: 6955
06/Jul/2008:06:31:32 +0200      13/Jul/2008:06:28:58 +0200      IPs: 8484       IPs on map: 7764
29/Jun/2008:06:28:36 +0200      06/Jul/2008:06:25:24 +0200      IPs: 10965      IPs on map: 10134
22/Jun/2008:06:33:28 +0200      29/Jun/2008:06:28:21 +0200      IPs: 7354       IPs on map: 6658
15/Jun/2008:06:29:22 +0200      22/Jun/2008:06:27:46 +0200      IPs: 7264       IPs on map: 6762
08/Jun/2008:06:31:08 +0200      15/Jun/2008:06:26:05 +0200      IPs: 10218      IPs on map: 9663
01/Jun/2008:06:33:55 +0200      08/Jun/2008:06:28:40 +0200      IPs: 16685      IPs on map: 15794
25/May/2008:06:29:23 +0200      01/Jun/2008:06:24:23 +0200      IPs: 13326      IPs on map: 12624
18/May/2008:06:36:31 +0200      25/May/2008:06:26:19 +0200      IPs: 11946      IPs on map: 11289
11/May/2008:07:11:22 +0200      18/May/2008:06:25:19 +0200      IPs: 1798       IPs on map: 1574
04/May/2008:06:29:41 +0200      11/May/2008:06:18:24 +0200      IPs: 1767       IPs on map: 1536
06/Jan/2008:06:54:22 +0100      04/May/2008:06:23:20 +0200      IPs: 23334      IPs on map: 19723

_______________________________________________
dev mailing list
[hidden email]
http://lists.openstreetmap.org/listinfo/dev
12