Back to blog home

How to write a server density plugin

An important part of developing and maintaining applications is the ability to monitor the environment they run on. Inviqa’s 24/7 support services depend on a number of tools to ensure that we get all the information required to monitor the health of our client's applications, and for server monitoring we use Server Density.

One of the reasons that we use Server Density is that it has a very easy-to-use plugin system. In this article, I’m going to look at the structure and installation of a Server Density plugin by writing a plugin to check for the presence of running processes.

 

How do server density plugins work?

The Server Density agent is a Python application that gathers various system data from your server and sends it off to a centralised system for processing. A large number of useful checks are baked into the core agent application, but there are always cases for specific monitoring tasks and that is where plugins come in.

Server Density allows you to configure a plugins directory. Once configured, the agent will scan the plugins directory for plugin scripts to execute on every data gathering run. The output of each plugin is included in the JSON payload that is sent out. Instructions for configuring the plugin directory can be found in the root of the Server Density plugins repository.

A plugin must be a Python file containing a class. The file and the class must be named the same, and the file must have a .py extension. The class must define a constructor that takes three arguments:

  • Agent configuration - a dictionary of configuration data that has been loaded and processed by the agent application
  • Logger - an instance of the Python logging object
  • Raw configuration - a dictionary of raw, unprocessed config data from the config file

 

The class must also contain a method called run, which returns a dictionary of data that will be sent back to the central system as part of the monitoring payload.

A skeleton plugin file would look like this:

MyPlugin.py
class MyPlugin:
    def __init__(self, agentConfig, logger, rawConfig):
        self.agentConfig = agentConfig
        self.logger = logger
        self.rawConfig = rawConfig

    def run(self):
        data = {'the answer': 42}
        return data

 

Creating our own plugin

Let's say that we want to check whether or not a given process is running on our server. We can make a plugin that checks for the existence of that process and returns a binary value that our monitoring can use to alert us if a critical application has stopped.

First, let's create a new plugin file called RunningProcesses.py, and drop in the class skeleton used above:

class RunningProcesses:
    def __init__(self, agentConfig, logger, rawConfig):
        self.agentConfig = agentConfig
        self.logger = logger
        self.rawConfig = rawConfig

    def run(self):
        data = {}
        return data

 

As you can see, the constructor is identical, and this doesn't need to change for the majority of plugins. All that's changed so far is the class name, which matches the filename, and there is currently no data being returned. For the rest of this tutorial, changes to code will continue to be marked in bold.

To start with, let's check for a single process, and code the name into our class (don't worry, we'll tidy up later). To do that we'll use a subprocess to execute a shell command, so at the top of the file we need to import subprocess:

import subprocess

class RunningProcesses:

 

In our run method we will now run the pgrep command in the subprocess, and capture any errors issued as well as the command's return code:

def run(self):

        p = subprocess.Popen(
            ['pgrep', 'nginx'],
            stderr=subprocess.PIPE
        )
        errorOutput = p.communicate()[1]
        returnCode = p.returncode

        data = {}

 

Note: Server Density ships with support for NGINX - this is just a simple example.

It is also possible to capture the command's standard output, but that's not necessary for this plugin because the exit codes from pgrep have enough meaning.

0: One or more processes matched the criteria.

1:  processes matched.

2: Syntax error in the command line.

3: Fatal error: out of memory etc.

By using these codes we can determine whether the process is running or not, and whether there is a need to log error information about the query itself. So let's do that:

returnCode = p.returncode

        if (returnCode > 1):
            self.logger.error(
                'Error detecting status of nginx: {0}'.format(errorOutput))

        data = {'nginx': returnCode == 0}

        return data

 

We now have a working, but rather basic, plugin. Let's see it in action before we go any further. In the Server Density dashboard, you can view a page that shows the JSON output of plugins for a single device (server).

 

This is good - we are receiving data. It's reporting that the process is not running, but that's to be expected because NGINX is not installed on this server yet. Once it is installed, we should see a change.

 

Bingo. We can view this in the form of a graph. All plugins can be added to the standard set of graphs in Server Density using the "Plugin" graph type.

 

We can also set up an alert to ensure that we know when the process fails.

 

So far, so good

This is already useful, but there are definitely improvements we can make. Having the name of the process baked into the script is not very nice because it makes the script less reusable, so let's look at how we can move that to configuration.

Configuration for the Server Density agent and its plugins is stored in /etc/sd-agent/config.cfg (which you will have already modified to enable the plugin directory). Insert a new section at the end of the config file to manage settings for the plugin, and restart the agent.

# logging_level: debug

[Running Processes]
process_list: nginx

 

Now we can use this new configuration element to make our script a bit more portable:

def run(self):
        processName = self.rawConfig['Running Processes']['process_list']

        p = subprocess.Popen(
            ['pgrep', processName],
            stderr=subprocess.PIPE
        )
        errorOutput = p.communicate()[1]
        returnCode = p.returncode

        if (returnCode > 1):
            self.logger.error(
                'Error detecting status of {0}: {1}'.format(
                    processName, errorOutput))

        data = {processName: returnCode == 0}

        return data

 

Nothing has changed in the output of the plugin, but the code is now much easier to reuse, with only a small amount of configuration required.

Just one last thing...

The last improvement to be made as part of this article is to allow the plugin to monitor multiple processes. For this, we will need to have a list of process names to monitor in the configuration file, and a more complex way of querying the system for the status of the processes.

For simplicity, let's just make the list of processes a comma separated string:

[Running Processes]
process_list: nginx,php-fpm

 

And now let's update the plugin to use this configuration value as a list of processes to monitor:

def run(self):
        data = {}

        processes = self.rawConfig['Running Processes']['process_list'].split(',')

        for processName in processes:
            p = subprocess.Popen(
                ['pgrep', processName],
                stderr=subprocess.PIPE
            )
            errorOutput = p.communicate()[1]
            returnCode = p.returncode

            if (returnCode > 1):
                self.logger.error(
                    'Error detecting status of {0}: {1}'.format(
                        processName, errorOutput))

            data[processName] = returnCode == 0

        return data

 

So now if we restart the agent (and install php-fpm) we can see that the plugin starts to transmit the status of both processes:

 

And we can see both processes on the graph:

 

Conclusion

In this article we have seen how to create a plugin for the Server Density monitoring system. While this was obviously a simple example to illustrate the basic mechanism of a Server Density plugin, the potential is huge. Using a powerful and lightweight language like Python, with a robust method of accessing the underlying shell, means that plugins can be created for almost any application with relative ease. Best of all, Server Density's plugins are open source, so browse the existing plugins and start contributing!

The published version of this plugin, with forked processing for multiple calls to pgrep and more substantial error handling, can be seen in the plugin repository.

 


 

About the author

Shane has been working in web application development since 2004, primarily with PHP, but also with large portions of ActionScript and front-end technologies. Shane is particularly interested in open source software, and when not spending time with his family or his archery club he contributes to several open source projects. Find him on Twitter @shanethehat or his Github.