Oversized message in log file

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Oversized message in log file

rougeusered
Hello,
I'm new to overpass. I've run into some issues and would appreciate any
pointers that might help resolve them.
I have set up a local overpass server.

1. For a particular query I have I see the following line in the
transactions.log
    'Oversized: runtime x seconds, size 541110848 bytes in line 1, statement
query

    From reading some of the docs and other user's experiences I was looking
to try the maxsize parameter with the query.

    However, when I run the same query -still without maxsize parameter-
against another instance of overpass, there is no such message and the server
responds to the query.

Both instances, have 8G RAM.
The differences between the two instances are:

a. Server with error in transactions log is dispatcher version 0.7.55.6
    Server without error in transactions log is dispatcher version 0.7.55.7

b. Server with error in transactions log is the planet database
    Server without error is a local region.
    The query is identical to both servers.

-->Any insight into why the one machine errors out?

2. The available free RAM is just above 7G at the time of the query resulting
in an error. I was running minute updates. I disabled minute updates but the
error didn't resolve. There is no other activity on the machine.  I started
the server using the default 12G maxsize.

-->Why doesn't/didn't overpass allocate the available RAM

Thank you
Reply | Threaded
Open this post in threaded view
|

Re: Oversized message in log file

Roland Olbricht
Hello,

I'm sorry for the longtime silence. I'm the maintainer of Overpass API,
and apparently the question looked like so technically involved that
noby else has attempted to answer.

> 1. For a particular query I have I see the following line in the
> transactions.log
>      'Oversized: runtime x seconds, size 541110848 bytes in line 1, statement
> query

I would like to suggest to turn off size limits on the local instance.
While there is a good reason to have size limits on the public instances
(because there really are clients with runaway scripts and often enough
so) to protect the sane from the insane users there is no benefit in the
limits system if you are the only user.

You can call
dispatcher --space=1000000000000
or with any other very high number (one US-trillion (=10^12) intended
here) and then use huge maxsize values on each query.

>      From reading some of the docs and other user's experiences I was looking
> to try the maxsize parameter with the query.
>
>      However, when I run the same query -still without maxsize parameter-
> against another instance of overpass, there is no such message and the server
> responds to the query.
>
> Both instances, have 8G RAM.
> The differences between the two instances are:
>

> b. Server with error in transactions log is the planet database
>      Server without error is a local region.
>      The query is identical to both servers.

Yes, the database sizes do make a difference. This can for some queries
happen even if the final outcome is identical.

The process counts at each step which number of objects it is expecting.
If there are too many objects anticipated then the process is aborted.
If there is more data in the database then the process might expect to
read more data.

To be more precise: the scheduler for queries gives reasonably small
regions (bounding boxes, around, areas) precedence over other criteria.
If you are searching with a small bounding box within the loaded
regional area then there should be no difference. The tipping point for
small is about one square degree.

On the other extreme, if your search criteria is something like a
globally present tag then the estimation is for a good reason higher.
The scheduler only takes into account on (the most prospective)
criterion because combination effects between multiple criteria are
beyond reasonable effort for an estimation.

> a. Server with error in transactions log is dispatcher version 0.7.55.6
>      Server without error in transactions log is dispatcher version 0.7.55.7

No, the two release only differ in that error messages have quotation
marks escaped.

> -->Why doesn't/didn't overpass allocate the available RAM

Please see above: the purpose of the limit system is to avoid harming
sane by insane users on the public instances. The estimations for that
purposes are rather geared to determine orders of magnitude, not precise
loads.

Best regards,

Roland
Reply | Threaded
Open this post in threaded view
|

Re: Oversized message in log file

rougeusered
Thank you Mr. Olbricht for your response. I was beginning to wonder if I had
posted to the wrong place.

Prior to your response, I bumped the query maxsize per query but not the space
argument on start up. This got through quite a bit of processing. But new
issues have cropped up. I'll try increasing the space argument.

I will post new questions, so as to isolate as much as possible the topics,
although they might be related.

The point is not to pester you but to try to resolve some of my
misunderstandings about overpass.