Wednesday, May 16, 2012

Python subprocess daemon manager: Pylot

One of the things I usually need is to execute subprocess daemons from python scripts to run task or even to offer some service that I need to be controlled by python code.
Executing subprocesses from python is quite easy with the Popen function from the subprocess module. You can exec subprocesses using an "exec" call, using a shell and you have a lot of parameters. This is quite cool, but sometimes you need to manage many daemons and you need them to be killed if the parent process dies and relaunched if they die. And there is other problem, if you send a SIGKILL signal to the parent process, the child processes will keep running and uncontrolled by any other program.
That is the reason I've implemented a module called Pylot. This module has two clases, Pylot and PylotManager. The first one is a thread that will execute the subprocess you want and other subprocess that acts as a watcher: if the main process dies, all your subprocesses will be killed by their watchers. Why to use watchers instead of capturing the SIGKILL signal??? Well, that is because... you can't capture the SIGKILL signal!
The PylotManager class will be used as an interface to handle the Pylots. You will be able to create, delete, check status and access the Pylots from PylotManager. PylotManager will check for the status of the Pylots and will relaunch them if they die, so it keeps all the daemons up and running.
Let's see a little example of how it works:

# Import PylotManager
from Pylot import PylotManager

# Create a new PylotManager and start it
pm = PylotManager()
pm.start()

# Once started, we can add a pylot for our daemon: htop
# NOTE: we set flush_output to False so we can later access
#   stderr and stdout. By default, flush_output is True so
#   you do not have to read the process output and you save
#   memory.
pm.addPylot('myfirstPylot','htop', flush_output=False)

# Now we access to our pylot
p = pm.getPylots()['myfirstPylot']

print p.pid
print p.stderr.read()
print p.stdout.read()

# Lets stop our pylot
pm.stopPylot('myfirstPylot')

# Now lets stop PylotManager. All remaining pylots will also
# be stopped
pm.stop()
As easy as it looks like. All the code is commented so you can see all the functions and parameters with the python help() command.
You can get the code from  https://github.com/diego-XA/dgtool/tree/master/pylot and you are free to use it and change it. If you find some bug or have a feature request, just comment.

Bye for now!


9 comments:

  1. Have you consider making this LGPL instead of GPL? This way it could be used as a library.

    ReplyDelete
  2. Hi, in fact it is GPL: https://github.com/diego-XA/dgtool/blob/master/pylot/LICENCE

    You can use it as library, that is why I published the code. Of course it is free both as in free beer and free speech. :)

    ReplyDelete
    Replies
    1. Anything deriving from GPL inherits the GPL license, so if you are mixing license and use a GPL module or library you taint the code base using the GPL module. I suggest you change it to LGPL, maybe this was your intent and you mistakenly only put GPL in the LICENCE file. The c library is LGPL that is why when you build your custom app it does not inherit the LGPL license.

      Delete
    2. I really like what you did, but would like to use it for create a custom process manager that we use on a product we build. If I use your module and integrate it into our code, I will taint our code base.

      Delete
    3. I didn't made a mistake with the licence, I wanted at first to be GPLv2. But the truth is that in this case, using GPL can be closing doors, so I'll change the licence to LGPL, what means that you can freely use it :).

      Of course any constructive opinions about the library are really welcome as well as bug reports (at github).

      Thanks for your comments and good luck with your product!

      Delete
    4. Thanks, I appreciate you changing the license so we can leverage your module. If I find anything or change anything in the module I will be sure to submit to github.

      Delete
  3. It appears that PyLot is setup to handle only processes that block and that do not daemonize (fork). Is this correct?

    ReplyDelete
  4. Yes, you are correct. Pylot uses subproces.Popen to exec each task and Popen has that limitation. What happens is that when Popen execs process p1 with pid 25 it handles only that process. When the executed process is a daemon, Popen execs p1 with pid 25 which forks and creates p1' with pid 26, so p1 ends and Popen detects the executed process as dead. This also have the problem that p1' with pid 26 will keep executing without any handler: an orphan process.

    ReplyDelete