This site is not optimized for Internet Explorer 9 and lower. Please choose another browser or upgrade your existing browser in order get the best experience of this website.

systemd ONgDB service on Ubuntu

March 04, 2016

systemd ships as the default process manager on Ubuntu 15; get the ongdb systemd service file to install and learn about the configuration. systemd has a lot of great features and I encourage you to check it out as it’s a big improvement IMO from SystemV and Upstart. If you’ve started working with systemd to manage your ongdb process, then you

might have stumble through the new setup. Here are a couple things I found helpful and some config that might get you moving along faster. I won’t go into detail on how to install services with systemd but it basically breaks down into these steps:

  1. Save service file  /lib/systemd/system/ongdb.services 
  2. Reload config:  sudo systemctl daemon-reload 
  3. Enable on startup:  sudo systemctl enable ongdb.service 

Service file for systemd and ongdb

For those simply looking for a working ongdb.service file one is shown below. The rest of the writeup goes into detail how to install this service file and why some of the properties are set the way they are. This configuration assumes you have linked the ongdb script into /etc/init.d/ongdb and that your ongdb installation is at /opt/ongdb e.g. ONGDB_HOME=/opt/ongdb. The file below should be located at:

/lib/systemd/system/ongdb.services

 

Backwards compatibility of systemd service for ongdb

From all my readings (and experience) systemd is backwards compatible with SystemV. This means that running services such as service ongdb start should continue to work if you have linked the ongdb process into /etc/init.d/ongdb and run the update-rc.d ongdb defaults commands to setup /etc/rc[1-6].d. I have found this works well for the scripts that ship with ongdb in version 2.1-3.0. I found that calling service ongdb start-no-wait should not be used anymore after you get systemd configured. I had some issues with it and it has been removed in 3.0.0-M04 so it’s best not to rely on it.

Configuring systemd file limits for ongdb

It wasn’t first obvious to me but systemd does not respect the global security limits at /etc/security/limits.conf. If you are tuning ongdb using the linux performance guide then all you need to include LimitNOFILE=60000 under the [Service] section of your ongdb.service file. When you configure the limits in systemd, you will see the warning message from ongdb startup disappear. If you are seeing this message below, adding this config is how you fix it:

 WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the ONgDB manual. 

Configuring systemd start/stop timeouts for ongdb

As of 2.3, the default start (was 120s) and stop (was 600s) timeouts on ongdb have been removed. This was done to avoid scenarios where large graphs take a long time to start and under heavy transaction load could experience moments where the transaction log flushing prevents shutdown from occurring cleanly. By default, systemd has timeouts in place for start and stop set to 90s. This is not good for large graph deployments.  The solution is to override the default by setting TimeoutSec=600 under the [Service] section.  I recommend this because most conditions we have experienced where the graph hangs it’s necessary to do a kill regardless because something is running that isn’t shutting down properly (often times we see this with extension code gone wrong). So after 10 minutes it’s likely time to kill the process. I have experienced that ongdb does a good job of recovering transactions in situations of forced shutdown. If you want to execute a kill using different times then separately configure start and stop by setting TimeoutStartSec=120 and TimeoutStopSec=600. You may need to adjust based on the size of your graph.

 

Managing the ongdb daemon with systemd

Systemd supports processes that spawn sub processes (forking). The default ongdb startup script spawns such a process. In order to get systemd to manage the process that the ongdb startup script spawns, it’s necessary to configure the PIDFile to point to the pid file that ongdb manages. By default this is at $ONGDB_HOME/data/ongdb-service.pid. The big gotcha with the PIDFile setting is the inability to use environment variables.  The solution is to set an absolute path for PIDFile to work properly.  For systems that would prefer to have systemd load the location of your $ONGDB_HOME from an environment file it’s real bummer.

Auto restart ongdb on failure with systemd

ONgDB has run very reliably for us but it’s very nice to auto restart ongdb for events that cause the process to fail unexpectedly. The way to achieve this is with Restart=on-failure which causes the watchdog to monitor the PID we supplied via the PIDFile and if that process fails with an exit code that is non-normal e.g. anything except 0 then it will start the ongdb process again. It is possible to configure the exit codes but any exit code other than 0 on the ongdb process is a candidate for auto-restart.

Summary of configuring systemd and ongdb

In summary, systemd has been easy to use and works well with ongdb. I hope these few tips have been helpful. Please message with questions or comments.