I’ve been playing around with Phantomjs as a solution for headless testing for quite some time now. It’s a great tool for testing!

I don’t want to get into details now, but there are two starting points in case you are interested:

• http://fcargoet.evolix.net/2012/01/use-phantomjs-to-take-screenshots-of-you-webapp-for-you/
  wrote a great article about how phantomjs can help you take screenshots of your webpage, thanks!
• CasperJS http://n1k0.github.com/casperjs/
  Framework with a few convenient function to do content parsing etc.

What did I miss?

A friend of mine asked me today how this works, so I’ll explain it with a few commands.
You are within your xen0 as root.
We will boot the xen-guest filesystem, use mount bind to reuse the kernel we booted within the environment we’ll chroot into, and that’s basically it.

mount /dev/xenguest /mnt
cp /etc/resolv.conf /mnt/etc/resolv.conf
mount -o bind /dev /mnt/dev
mount -t proc none /mnt/proc

# switch to the new environment
chroot /mnt /bin/bash

Done.
Don’t forget to unmount /mnt/{dev,proc} before umounting /mnt or restarting.

I needed a simple solution for PID files today. This is what [1] helped me build:

class pid_file:
    pidfile = os.path.join(PATHPREFIX, "%s.pid" % str(__name__))
    def __enter__(self):
        if os.path.exists(self.pidfile):
            with open(self.pidfile, 'r') as f:
                runningpid = f.read()
            raise Exception("%s is already running. Process number %s"
                            % (__name__, runningpid))
        pid = str(os.getpid())
        with open(self.pidfile, 'w') as f:
            f.write(pid)
        return pid
    def __exit__(self, type, value, traceback):
        assert os.path.exists(self.pidfile), "Pidfile should exist!"
        os.remove(self.pidfile)
 
with pid_file():
    code_that_should_check_for_other_running_instances_before()

Pretty straight forward in my opinion. If only python had been around when I used bash for day to day sysadmin tasks…

[1] http://effbot.org/zone/python-with-statement.htm

Munin is a monitoring tool, mostly used to monitor various parameters on servers. The Munin script crawl the Nodes configured there. I won’t get into configuring the server in detail because it’s just a script crawling the clients, the magic happens there.

To configure the server, we modify the /etc/munin/munin.conf and add

[clientserver]
address clientserver.example.com
port 1234

Thats it. Most of the times the first Client you will be using is installed on the server. I think this might be confusing at this point. We’ll understand it later.

Settings up the client’s munin-node

1. Install it: aptitude install munin-node (we don’t need “munin” here just munin-node)
2. Modify /etc/munin/munin-node.conf to allow external access: Allow ^123\.41\.23\.41$
3. Optional: Modify firewall to allow munin. ufw Modification could be:
   ufw allow proto tcp from 123.41.23.41 to any port 1234

The good stuff: plugins

Most confusing for me is/was how munin handles plugins.

Every client configures the parameters monitored with symlinks and configuration files.

Credits to http://agiletesting.blogspot.com/2010/09/getting-detailed-io-stats-with-munin.html for straight forward explaination: First, try

/usr/share/munin/plugins/diskstat_ suggest

then symlink the suggestions

ln -snf /usr/share/munin/plugins/diskstat_ /etc/munin/plugins/diskstat_latency_xvda2

xvda2 could be hda1 or whatever your configuration is.

Don’t forget to restart your munin-node and enjoy the results!

What is your favorite munin plugin?

“ rNews is a proposed standard for using RDFa to annotate news-specific metadata in HTML documents.  The rNews proposal has been developed by the IPTC, a consortium of the world’s major news agencies, news publishers and news industry vendors.”

Supervisor is a small python package and can save you the pain of managing background services on your server or locally. I’ll try to explain you how.

First: Installation, easy. You install it either via pypi

pip install supervisor

Or maybe there is a package included in your linux. The linux package comes with the init scripts to start automatically, I don’t want that on my local PC and prefer the latest version so I use pip.

How can supervisor help me?

Two different situations:

1. You are running nginx. Nice webserver, but you need to start separate processes, namely fastcgid, python (gunicorn/uwsgi/tornado), ruby scripts… –>  supervisord can manage them for you and handle the permissions as you like them, you don’t need to fight with initscripts.
2. You develop a webapp on your local mashine and as your app grows you need to start several damons to to specific tasks. Celery to manage your async tasks, Solr to handle fulltext indexing, Mysql, Postgresql, Virtuoso, nodejs. Starting all those by hand is a time consuming task. Supervisord can do that for you!

How does it work?

Supervisor comes with two programs: supervisord and supervisorctl. supervisord is the daemon, conrolled by supervisorctl.

All you need is a configuration file. You can put it in /etc/supervisor or just inside the directory you are working in right now. It might look like this:

supervisord.conf

[unix_http_server]
file=supervisor.sock
chmod=0700
[supervisord]
[rpcinterface:supervisor]
supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix://supervisor.sock

[program:djangoapp]
directory=/home/pyuser/app
command=virtualenv/bin/python manage.py runserver
stdout_logfile=supervisor_stdout.log
stderr_logfile=supervisor_stderr.log

[program:virtuoso]
directory=/path/where/db/is
command=/usr/local/virtuoso-opensource/bin/virtuoso-t +foreground +configfile /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.ini

[program:solr]
directory=/path/where/downloaded/apache-solr-3.1.0/example
command=java -Dsolr.solr.home=/Volumes/SSD/.solr -jar /DATA/apache-solr/example/start.jar;
stopsignal=QUIT
user=solr

This is just a simple example. Feel free to go wild here.

How do I use it?

Start it by running “supervisord” (if you installed the version packages with your distribution it might already run).

Control the Services with supervisorctl. “supervisorctl status” should give you a list of services.

More:

• supervisorctl start djangoapp — starts the djangoapp defined above
• supervisorctl stop|restart djangoapp
• supervisorctl stop all
• supervisorctl reread && supervisorctl update — rereads the configuration file and starts/stops/restarts the services that changed

That’s it, all you need to know. Did I promise too much?

1. Why?

Solr 1.4.0 included in Ubuntu is pretty old and even 1.4.1 too old to use. Solr merges into Lucene in Version 3.1 (http://wiki.apache.org/solr/Solr3.1) and usable enough to use it: http://stackoverflow.com/questions/4138834/solr-3-1-where-to-download, even includes Tika (http://lucene.grantingersoll.com/2008/12/06/tika-and-solr/) for indexing MS Office, .pdf and all those other formats, can index JSON directly: http://twitter.com/retresco/status/12739390895824896…

Time to switch!

2. How?

Get the latest RC from http://search-lucene.com/m/lUQr42C4gg7: http://people.apache.org/~rmuir/staging_area/lucene-solr-3.1RC0-rev1078688/solr-3.1RC0/apache-solr-3.1.0.zip

Unzip it.

java -Dsolr.solr.home=apache-solr-3.1.0/example/multicore -jar apache-solr-3.1.0/example/start.jar

Done.

If you run into an java.io.FileNotFoundException exception, try http://h3x.no/2010/08/13/java-io-filenotfoundexception-no-segments-file-found-solr.

Update:

Apache solr 3.1 was released! Download it here. “Installation” stays the same. Great work guys!

I found a pypy Formula in homebrew so I tried it out:

brew install pypy

Installs Version 1.4.1, good enough. So let’s play with it.

As I want to install ipython, I need a virtualenv to install it without causing harm to my system.
Luckily virtualenv.py is easy to use, wait: http://www.virtualenv.org/en/latest/index.html#pypy-support “Currently only PyPy trunk is supported.”, hm.

Two changes to Formula needed in order to install trunk:
1. Install the latest nightly build as of today (1.5.0a). You might probably want to look up the latest version for yourself, md5 will not match that of course:

    url 'http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-42803-143c77a3550f-osx64.tar.bz2'
    md5 '534d20d846d04b8003deebf1fdaea2cc'

2. Make sure to install “include” as I am pretty sure we will need it at some point:

    prefix.install ["bin", "lib-python", "lib_pypy", "include"]

Result:

require 'formula'
require 'hardware'
 
class Pypy < Formula
  url 'http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-42803-143c77a3550f-osx64.tar.bz2'
  md5 '534d20d846d04b8003deebf1fdaea2cc'
 
  homepage 'http://pypy.org/'
  version '1.5.0a'
 
  def install
    prefix.install ["bin", "lib-python", "lib_pypy", "include"]
  end
end

After installing it I used it to run programs I wrote.

Long story short: As long as you don’t need packages like PIL or pycurl (tornado needs it), pypy works like a charm

It’s not as “fast” (according to pystone, depends on what you want to do…) as the other Pythons I have, but that always depends on your usecase of course:

Python 2.7.0 (143c77a3550f, Mar 20 2011, 03:00:09)
[PyPy 1.5.0-alpha0] on darwin
Type "help", "copyright", "credits" or "license" for more information.
And now for something completely different: ``RPython: we use it so you don't
have to''
>>>> from test import pystone; pystone.main(10)
Pystone(1.1) time for 10 passes = 0.000379
This machine benchmarks at 26385.2 pystones/second

(1.4.1 get 26041.7 pystones/second).

What I usually use:

Python 2.7.1 (r271:86832, Jan  1 2011, 10:52:17)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from test import pystone; pystone.main(10)
Pystone(1.1) time for 10 passes = 0.000201
This machine benchmarks at 49751.2 pystones/second

Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
>>> from test import pystone; pystone.main(10)
Pystone(1.1) time for 10 passes = 0.000148
This machine benchmarks at 67567.6 pystones/second

Comparing apples and oranges, enough for today. Thanks to the Pypy Team for their great work!

Why I program Python?
 

Cussing in Commits: Which Programming Language Inspires the Most Swearing? via Webmonkey

Sometimes I need to add some more sophisticated checks to my bash scripts and a really simple way to achieve this is to embed python programms inside the bash script:

#!/bin/bash
set -e # this will cause the script to end on the first error
echo "Normal Bash operations here"
python - < END
import django
# do python coding here
END
echo "Bash again"

Hope that helps. Anything I forgot?