Systemd

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Systemd

Roland Olbricht
Hallo,

there has been recently demand to run the Overpass API via systemd.
After a thourough analysis I have written a blog post:
https://dev.overpass-api.de/blog/systemd.html

The executive summary is: Don't do it. I'm now happy to figure out with
the affected people what the use cases behind the intent to use systemd
are and to pave an alternative path.

Best regards,

Roland
Reply | Threaded
Open this post in threaded view
|

Re: Systemd

marc marc
Hello,

Le 27. 10. 17 à 07:00, Roland Olbricht a écrit :
> The executive summary is: Don't do it. I'm now happy to figure out with
> the affected people what the use cases behind the intent to use systemd
> are and to pave an alternative path.

the main reason for the osm-fr server is that it's the way to do on
recents systems (debian 9, RHEL/Centos).
the secondary reasons:
- easier to restart services in case of a crash, to manage dependencies
- centralized log management in syslog (although it is also possible to
do this by modifying the init.d script).

in a previous email, I was wondering what was the preferred method for
presenting changes to scripts.
talk about it in this thread ? make an issue + PR on github ?

Regards,
Marc
Reply | Threaded
Open this post in threaded view
|

Re: Systemd

Igor Brejc
Hi,

In my case I used systemd for two reasons:
  1. it was documented on the Overpass API page
  2. I'm coming from the Windows world, where Windows services (similar to systemd) are a standard way of ensuring a background service is started/stopped cleanly when the system starts/shuts down, so it was an obvious choice for me to use systemd (and I agree with marc's comments for his reasons).
As for my use case, I fall more into the second case (running on a notebook for private use), although my "notebook" is a full-blown server PC that needs to be able to return data for the whole world, and I use it only occasionally, pm-suspending it when not in use. 

Given that I my current systemd setup seems to be working OK now (and I spent too much time setting it up, debugging and documenting everything), I will leave it as it is. 

I see Overpass API as a very valuable tool that helps people to use OSM data even when the dataset is huge and ever-increasing. Right now the barrier of entry for users setting their own private instances is quite high, since you need to have at least a moderate level of Linux knowledge to be able to set it up. Those three steps you mention in the blog post are unfortunately not enough to replicate the whole behavior of the Overpass API public instance - there are still several ways how to import the data, then there are diffs, web server etc. and these can get quite messy, especially if you're doing it for the first time. I tried to document how I did it in my own specific case, from a Linux newby perspective (1), but I think a partially automated installation procedure would be better for the end-user.

Cheers,
Igor



On Fri, Oct 27, 2017 at 9:35 AM, marc marc <[hidden email]> wrote:
Hello,

Le 27. 10. 17 à 07:00, Roland Olbricht a écrit :
> The executive summary is: Don't do it. I'm now happy to figure out with
> the affected people what the use cases behind the intent to use systemd
> are and to pave an alternative path.

the main reason for the osm-fr server is that it's the way to do on
recents systems (debian 9, RHEL/Centos).
the secondary reasons:
- easier to restart services in case of a crash, to manage dependencies
- centralized log management in syslog (although it is also possible to
do this by modifying the init.d script).

in a previous email, I was wondering what was the preferred method for
presenting changes to scripts.
talk about it in this thread ? make an issue + PR on github ?

Regards,
Marc

Reply | Threaded
Open this post in threaded view
|

Re: Systemd

marc marc
Le 27. 10. 17 à 12:03, Igor Brejc a écrit :
> a partially automated installation
> procedure would be better for the end-user.

Rodolph @ osm-fr do nearly all the job for having a working ansible role
we are just working on a few bug and of course it should be good to add
it to the wiki when it reach a "stable" state
with this, on linux, installing a overpass api server take a few step
- install ansible and git (one command)
- edit conf (scope and path) if needed (one command)
- run ansible (one command)
it's done :)

Regards,
Marc
mmd
Reply | Threaded
Open this post in threaded view
|

Re: Systemd

mmd
In reply to this post by Igor Brejc


Am 27.10.2017 um 12:03 schrieb Igor Brejc:
 I did it in my own specific case, from a Linux
> newby perspective (1), but I think a partially automated installation
> procedure would be better for the end-user.
>


There's a number of docker images out there with varying scope and
quality. Usually they install all required packages, compile Overpass
binaries from source code, install them including web server, download
clones / import databases.

It would be really good to have a supported and endorsed version in the
main branch, which is following best practices.

Som examples:

https://github.com/drolbr/Overpass-API/pull/374
https://github.com/Frankkkkk/docker-overpass-api
https://github.com/mmd-osm/docker-overpass-api/
(and at least 5 others I forgot right now)

--




Reply | Threaded
Open this post in threaded view
|

Re: Systemd

Roland Olbricht
> Am 27.10.2017 um 12:03 schrieb Igor Brejc:
>   I did it in my own specific case, from a Linux
>> newby perspective (1), but I think a partially automated installation
>> procedure would be better for the end-user.

Thank you for the feedback. I fear that there is a certain cultural gap.
For example, I conside configure-make as essentially automatic. I'm
optimisitic that a chapter about the used parts in front of the
installation instructions could help.

I'm grateful for Igors instructions because they help to find the points
that cause confusion and must be adressed.

> It would be really good to have a supported and endorsed version in the
> main branch, which is following best practices.
It is always a good idea to give as much help in adavance as possible.
There are still some problems to solve for docker. They are essentially
a measure to assure that we do not run into a support avalanche:

- There should be a form of testing that checks whether the docker image
does what it is supposed to do such that we can test it on any available
configuration without effort. While tests on the software level are
relatively simple, I have no idea how to properly test software
installation: A typical example that such a test should catch are
improper file permissions or components running under the wrong user.

- Are the used interface components guaranteed to be long-term stable?
One of the major disadvantages of systemd against POSIX is that what
works today not necessarily works tomorrow. The RemoteIPC setting is an
example. This can or can not be better with docker.

- Do we understand the mechanics of docker good enough to find for any
weird error message the underlying source of the problem? Please note
that the two preceeding points are special cases of that variant. But
things like typos or missing arguments may come on top.

- It should be clear that we do not require expierenced users of
POSIX-based systems to do the extra detour of docker. This is mostly a
documentation problem.

Best regards,

Roland
mmd
Reply | Threaded
Open this post in threaded view
|

Re: Systemd

mmd
In reply to this post by Roland Olbricht
Am 27.10.2017 um 07:00 schrieb Roland Olbricht:

> Hallo,
>
> there has been recently demand to run the Overpass API via systemd.
> After a thourough analysis I have written a blog post:
> https://dev.overpass-api.de/blog/systemd.html
>
> The executive summary is: Don't do it. I'm now happy to figure out with
> the affected people what the use cases behind the intent to use systemd
> are and to pave an alternative path.
>

The systemd discussion made it into Weekly 380, and there's already some
feedback on the blog post:
http://blog.openstreetmap.de/blog/2017/11/wochennotiz-nr-380/#comment-226304
(in German).

--



mmd
Reply | Threaded
Open this post in threaded view
|

Re: Systemd

mmd
In reply to this post by Roland Olbricht
Am 29.10.2017 um 22:54 schrieb Roland Olbricht:

>
>> It would be really good to have a supported and endorsed version in the
>> main branch, which is following best practices.
> It is always a good idea to give as much help in advance as possible.
> There are still some problems to solve for docker. They are essentially
> a measure to assure that we do not run into a support avalanche:
>
> - There should be a form of testing that checks whether the docker image
> does what it is supposed to do such that we can test it on any available
> configuration without effort. While tests on the software level are
> relatively simple, I have no idea how to properly test software
> installation: A typical example that such a test should catch are
> improper file permissions or components running under the wrong user.

Basically you can create a new docker image by processing all steps
outlined in a so called Dockerfile. This file describes the whole
automated installation process, including the base image, all users and
groups to be created, all software packages to install, the exact steps
to compile and install all binaries as needed, etc.

Components running under a wrong user will not happen, unless there's a
wrong instruction in the Dockerfile. Also, if any of those steps fail,
the docker image creation process may as well fail.

>
> - Are the used interface components guaranteed to be long-term stable?
> One of the major disadvantages of systemd against POSIX is that what
> works today not necessarily works tomorrow. The RemoteIPC setting is an
> example. This can or can not be better with docker.

I think there's a misunderstanding here. Initially I referred to Docker
as an easy way to provide a ready-made software image for users, and
avoid all the manual steps Igor had to go through.

It really has nothing to do with the systemd discussion, I should have
opened a separate thread for it. You're free to use or ignore systemd
inside your Docker image depending on the best practices of your base image.

Here's an example for Postgres:
https://github.com/docker-library/postgres/tree/master/9.6


> - It should be clear that we do not require experienced users of
> POSIX-based systems to do the extra detour of docker. This is mostly a
> documentation problem.

I see Docker clearly as a way to provide an easy to use Overpass API
image with (almost) zero hassle. If you don't use Docker, there's no
detour in any way.