This site is not optimized for Internet Explorer 9 and lower. Please choose another browser or upgrade your existing browser in order get the best experience of this website.

systemd Neo4j service on Ubuntu

March 04, 2016

GraphGrid systemd neo4j ubuntusystemd ships as the default process manager on Ubuntu 15; get the neo4j systemd service file to install and learn about the configuration. systemd has a lot of great features and I encourage you to check it out as it’s a big improvement IMO from SystemV and Upstart. If you’ve started working with systemd to manage your neo4j process, then you

might have stumble through the new setup. Here are a couple things I found helpful and some config that might get you moving along faster. I won’t go into detail on how to install services with systemd but it basically breaks down into these steps:

  1. Save service file  /lib/systemd/system/neo4j.services 
  2. Reload config:  sudo systemctl daemon-reload 
  3. Enable on startup:  sudo systemctl enable neo4j.service 

Service file for systemd and neo4j

For those simply looking for a working neo4j.service file one is shown below. The rest of the writeup goes into detail how to install this service file and why some of the properties are set the way they are. This configuration assumes you have linked the neo4j script into /etc/init.d/neo4j and that your neo4j installation is at /opt/neo4j e.g. NEO4J_HOME=/opt/neo4j. The file below should be located at:

/lib/systemd/system/neo4j.services

 

Backwards compatibility of systemd service for neo4j

From all my readings (and experience) systemd is backwards compatible with SystemV. This means that running services such as service neo4j start should continue to work if you have linked the neo4j process into /etc/init.d/neo4j and run the update-rc.d neo4j defaults commands to setup /etc/rc[1-6].d. I have found this works well for the scripts that ship with neo4j in version 2.1-3.0. I found that calling service neo4j start-no-wait should not be used anymore after you get systemd configured. I had some issues with it and it has been removed in 3.0.0-M04 so it’s best not to rely on it.

Configuring systemd file limits for neo4j

It wasn’t first obvious to me but systemd does not respect the global security limits at /etc/security/limits.conf. If you are tuning neo4j using the linux performance guide then all you need to include LimitNOFILE=60000 under the [Service] section of your neo4j.service file. When you configure the limits in systemd, you will see the warning message from neo4j startup disappear. If you are seeing this message below, adding this config is how you fix it:

 WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual. 

Configuring systemd start/stop timeouts for neo4j

As of 2.3, the default start (was 120s) and stop (was 600s) timeouts on neo4j have been removed. This was done to avoid scenarios where large graphs take a long time to start and under heavy transaction load could experience moments where the transaction log flushing prevents shutdown from occurring cleanly. By default, systemd has timeouts in place for start and stop set to 90s. This is not good for large graph deployments.  The solution is to override the default by setting TimeoutSec=600 under the [Service] section.  I recommend this because most conditions we have experienced where the graph hangs it’s necessary to do a kill regardless because something is running that isn’t shutting down properly (often times we see this with extension code gone wrong). So after 10 minutes it’s likely time to kill the process. I have experienced that neo4j does a good job of recovering transactions in situations of forced shutdown. If you want to execute a kill using different times then separately configure start and stop by setting TimeoutStartSec=120 and TimeoutStopSec=600. You may need to adjust based on the size of your graph.

 

Managing the neo4j daemon with systemd

Systemd supports processes that spawn sub processes (forking). The default neo4j startup script spawns such a process. In order to get systemd to manage the process that the neo4j startup script spawns, it’s necessary to configure the PIDFile to point to the pid file that neo4j manages. By default this is at $NEO4J_HOME/data/neo4j-service.pid. The big gotcha with the PIDFile setting is the inability to use environment variables.  The solution is to set an absolute path for PIDFile to work properly.  For systems that would prefer to have systemd load the location of your $NEO4J_HOME from an environment file it’s real bummer.

Auto restart neo4j on failure with systemd

Neo4j has run very reliably for us but it’s very nice to auto restart neo4j for events that cause the process to fail unexpectedly. The way to achieve this is with Restart=on-failure which causes the watchdog to monitor the PID we supplied via the PIDFile and if that process fails with an exit code that is non-normal e.g. anything except 0 then it will start the neo4j process again. It is possible to configure the exit codes but any exit code other than 0 on the neo4j process is a candidate for auto-restart.

Summary of configuring systemd and neo4j

In summary, systemd has been easy to use and works well with neo4j. I hope these few tips have been helpful. Please message with questions or comments.