Friday, November 23, 2012

Watch realtime HTTP requests per second

I've recently wanted to know how many requests per second where I having on a server. The server is using nginx as server and this is the way I found:

watch -n 1 'a="$(date +%d/%b/%Y:%H:%M:$(($(date +%S)-1)))";grep -c "$a" access.log'
I decrease the seconds in 1 unit as the current second could not be finished when the check is done (yes, it will fail on second 0 but... we have 59 seconds more!). You'll have to change the path to your access log file. You'll probably have to change the format of the date for other formats of access.log. Anybody knows a better way?

Thursday, June 7, 2012

ErlyVideo connector

I've recenctly worked with ErlyVideo servers. They allow you to manage RTMP streams so you can serve them in a web page using a flash player. They work pretty well but they have a big fail: then cannot be concatenated. What do I mean by concatenate? I mean that you cannot use an ErlyVideo server as source of other ErlyVideo. At least, in my tests, the result was a corrupted video output.
As there is a big lack of documentation regarding ErlyVideo configuration and deploy arquitecture, I decided to code a little application that would receive the rtmp streams from an ErlyVideo server and send them to other one. Using ffmpeg you can accomplish this job as it supports rtmp*, but you need an application to manage all the ffmpegs and the streams and, once you are working on, it's a good idea to code some user interface. And this is the result: erlyconnect. It is a very simple application but it has been very useful.



As always, you can download it from the github repository at: https://github.com/diego-XA/dgtool/tree/master/erlyconnect/trunk
There is a README file to see how it works. Please, let me know if you find some bug or just have some doubts about how it works. Comments are always welcome.

_____

* ffmpeg supports RTMP if it has bee compiled to support it. I used the medibuntu and debian-multimedia repositories as the standard ffmpeg of ubuntu and debian do not suport RTMP.

Wednesday, May 16, 2012

Python subprocess daemon manager: Pylot

One of the things I usually need is to execute subprocess daemons from python scripts to run task or even to offer some service that I need to be controlled by python code.
Executing subprocesses from python is quite easy with the Popen function from the subprocess module. You can exec subprocesses using an "exec" call, using a shell and you have a lot of parameters. This is quite cool, but sometimes you need to manage many daemons and you need them to be killed if the parent process dies and relaunched if they die. And there is other problem, if you send a SIGKILL signal to the parent process, the child processes will keep running and uncontrolled by any other program.
That is the reason I've implemented a module called Pylot. This module has two clases, Pylot and PylotManager. The first one is a thread that will execute the subprocess you want and other subprocess that acts as a watcher: if the main process dies, all your subprocesses will be killed by their watchers. Why to use watchers instead of capturing the SIGKILL signal??? Well, that is because... you can't capture the SIGKILL signal!
The PylotManager class will be used as an interface to handle the Pylots. You will be able to create, delete, check status and access the Pylots from PylotManager. PylotManager will check for the status of the Pylots and will relaunch them if they die, so it keeps all the daemons up and running.
Let's see a little example of how it works:

# Import PylotManager
from Pylot import PylotManager

# Create a new PylotManager and start it
pm = PylotManager()
pm.start()

# Once started, we can add a pylot for our daemon: htop
# NOTE: we set flush_output to False so we can later access
#   stderr and stdout. By default, flush_output is True so
#   you do not have to read the process output and you save
#   memory.
pm.addPylot('myfirstPylot','htop', flush_output=False)

# Now we access to our pylot
p = pm.getPylots()['myfirstPylot']

print p.pid
print p.stderr.read()
print p.stdout.read()

# Lets stop our pylot
pm.stopPylot('myfirstPylot')

# Now lets stop PylotManager. All remaining pylots will also
# be stopped
pm.stop()
As easy as it looks like. All the code is commented so you can see all the functions and parameters with the python help() command.
You can get the code from  https://github.com/diego-XA/dgtool/tree/master/pylot and you are free to use it and change it. If you find some bug or have a feature request, just comment.

Bye for now!