New blogging engine

Finally, after a year of almost no activity, I have updated my blog with new software. I used to run WordPress for the past seven or so years but finally decided it was time for something new. As of today, I am running on Pelican, a Python-based static website generator. This means good-bye to the php-based dynamic colution. I hope this will give this blog some more security. Comments will be via Disqus for the moment, for want of a better alternative. I still plan to eventually migrate to completely static comments, but
for now Disqus will do.

Also, you might notice that the theme I use looks a lot like the one from Octopress. This is because I use the fabolous Pelican Octopress theme by Maurizio Sambati. I don’t use Octopress itself at the moment because I still have a hard time working with Ruby on Windows. Also, Pelican easily supports reStructured Text out of the box which I do in fact like better than Markdown (and have used it more often).

My new setup includes some more nifty technical details.

For ease of deployment, I have devised a workflow that is based on RhodeCode and Jenkins. I have setup a RhodeCode instance on my local server where I store the source to my blog. At the moment I use Mercurial to manage the versions. Furthermore, I have a Jenkins installation on the same server that pulls from the RhodeCode instance, builds the blog contents and publishes the result here afterwards. To avoid frequently polling on Mercurial, I have also installed a trigger in Mercurial (or RhodeCode respectively) that triggers the rebuild of the blog. The following figure gives an overview.

new publishing workflow

Now, this might seem like a bit of overhead on the setup side, but I had a lot of fun automating it this way. Technically I could just as well pusblish the blog from my personal PC everytime, but I found that to be an unworthy alternative ;)

What to do if biber stops working on MacOS X

Yesterday evening I ran into a bit of a problem when I tried building my latex document that uses biblatex/biber. For some reason, biber would not generate any output but fail with

Olivers-MacBook-Air:~ porten$ biber
data source /var/folders/5c/y7ssnfgj2fz1fnx1gz_z7ng80000gn/T/par-porten/cache-5a7f3069e2a4d51fd3557003fc55ec74c554c947//inc/lib/Biber/LaTeX/recode_data.xml not found in .
Compilation failed in require at Biber/Utils.pm line 21.
BEGIN failed--compilation aborted at Biber/Utils.pm line 21.
Compilation failed in require at Biber/Internals.pm line 8.
BEGIN failed--compilation aborted at Biber/Internals.pm line 8.
Compilation failed in require at (eval 22) line 2.
    ...propagated at /opt/local/lib/perl5/5.14.1/base.pm line 94.
BEGIN failed--compilation aborted at Biber.pm line 5.
Compilation failed in require at script/biber-darwin line 20.
BEGIN failed--compilation aborted at script/biber-darwin line 20.

Now, what fixes this is simply deleting the par cache:

sudo rm -rf /var/folders/5c/y7ssnfgj2fz1fnx1gz_z7ng80000gn/T/par-porten/

Sources:

Running WindowTester Pro from Maven/Tycho

Running SWTBot tests as part of your maven/tycho build cycle is rather
well documented and pretty straight forward (see for example the
sonatype docs). Doing so with a WindowTester Pro recorded test is
not – but then again it is pretty straigt forward as well.

There are just two things that you need to change from the SWTBot tests
pom.xml (as presented in the sonatype docs):

  • have useUIThread = true
  • add a dependecy to com.windowtester.runtime.feature.group to pull
    in platform specific plugins

You will also need to tell maven/tycho where to find the latter feature,
of course, by adding
http://dl.google.com/eclipse/inst/windowtester/beta/3.7 to your
repositories.

A sample pom.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelversion>4.0.0</modelversion>
  <parent>
    <artifactid>de.van_porten.vrest.build</artifactid>
    <groupid>de.van_porten</groupid>
    <version>0.0.1-SNAPSHOT</version>
    <relativepath>../de.van_porten.vrest.build</relativepath>
  </parent>
  <groupid>de.van_porten</groupid>
  <artifactid>de.van_porten.vrest.tests.recorded</artifactid>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>eclipse-test-plugin</packaging>

  <repositories>
    <repository>
      <id>windowtester</id>
      <layout>p2</layout>
      <url>http://dl.google.com/eclipse/inst/windowtester/beta/3.7</url>
    </repository>
  </repositories>

  <build>
    <plugins>
      <plugin>
        <groupid>org.eclipse.tycho</groupid>
        <artifactid>tycho-surefire-plugin</artifactid>
        <version>${tycho-version}</version>

        <configuration>
          <testsuite>de.van_porten.vrest.tests.recorded</testsuite>
          <testclass>de.van_porten.vrest.tests.recorded.AllTests</testclass>

          <useuiharness>true</useuiharness>
          <useuithread>true</useuithread>
          <!-- use our product and application to launch the tests -->
          <product>de.van_porten.vrest.bundle.product</product>
          <application>org.eclipse.ui.ide.workbench</application>

          <dependencies>
            <dependency>
              <type>p2-installable-unit</type>
              <artifactid>com.windowtester.runtime.feature.group</artifactid>
              <version>0.0.0</version>
            </dependency>
          </dependencies>
        </configuration>

      </plugin>
    </plugins>
  </build>


  <profiles>
    <profile>
      <id>osx</id>
      <activation>
        <property>
          <name>java.vendor.url</name>
          <value>http://www.apple.com/</value>
        </property>
      </activation>
      <build>
        <pluginmanagement>
          <plugins>
            <plugin>
              <groupid>org.eclipse.tycho</groupid>
              <artifactid>tycho-surefire-plugin</artifactid>
              <version>${tycho-version}</version>
              <configuration>
                <argline>-XstartOnFirstThread</argline>
              </configuration>
            </plugin>
          </plugins>
        </pluginmanagement>
      </build>
    </profile>
  </profiles>

</project>

Running Tycho in Jenkins/Hudson

When I first started running my Tycho/Maven build of my visual editor in
Jenkins the build would always fail if a single test failed. That was
mainly because the Maven build would run the surefire test automatically
but cancel the build if there were any failures in them. In turn, this
would not keep artifacts from being generated and thus not create a new
snapshot release. To make matters worse, my tests are not that stable
yet – running them on Linux sometimes fails for no reason whatsoever,
leading to yet another broken build.

Now, this is obviously not what you (or Jenkins, for that matter) would
expect to happen. What I wanted and what I guess you would normally want
is for the build to pass but for Jenkins to mark the build as
"unstable".

To acomplish this on the command line, you simply need to tell Maven to
ignore test failures

mvn clean install -Dmaven.test.failure.ignore=true

With Jenkins, this is equally straight forward:

image0

Now Jenkins acts nicely and reports unstable builds while still
producing all artifacts.

vRest maven build finally working

I finally managed to get the Maven-based build of my Eclipse-based
graphical editor to work – including unit and swtbot-ui tests:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] de.van_porten.vrest.build ......................... SUCCESS [0.077s]
[INFO] de.van_porten.vrest.target-platform ............... SUCCESS [0.112s]
[INFO] de.van_porten.vrest.core .......................... SUCCESS [2.842s]
[INFO] de.van_porten.vrest.core.edit ..................... SUCCESS [0.869s]
[INFO] de.van_porten.vrest.core.editor ................... SUCCESS [0.829s]
[INFO] de.van_porten.vrest.bundle ........................ SUCCESS [0.105s]
[INFO] de.van_porten.vrest.help .......................... SUCCESS [0.076s]
[INFO] de.van_porten.vrest.ui ............................ SUCCESS [1.387s]
[INFO] de.van_porten.vrest.ui.properties ................. SUCCESS [3.650s]
[INFO] de.van_porten.vrest.feature ....................... SUCCESS [0.113s]
[INFO] de.van_porten.vrest ............................... SUCCESS [1:08.099s]
[INFO] de.van_porten.vrest.tests.core .................... SUCCESS [4.087s]
[INFO] de.van_porten.vrest.tests.ui ...................... SUCCESS [13.315s]
[INFO] de.van_porten.vrest.p2-repository ................. SUCCESS [2.498s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3:02.410s
[INFO] Finished at: Sat Nov 26 22:22:30 CET 2011
[INFO] Final Memory: 97M/231M
[INFO] ------------------------------------------------------------------------

I’ll push to Jenkins now and see if the continous build and test also
works there. Cross your fingers.

The one thing that is bugging me is an oddness with my target platform
defintion that for some reason doesn’t want to resolve
org.apache.batik.* properly – once I switch target to the development
platform everything works as expected. But I will figure that one out
eventually ..

More on the culprits and solutions I have to follow shortly.

CUnit-to-JUnit transform now on BitBucket

One of the most visited blog entries on this blog probably is my article
on transforming CUnit results to JUnit results: CUnit Tests in
Hudson
. There I write about how to use xsltproc to transform the
output of CUnit to be processed by Hudson/Jenkins.

Now, I decided to put the file on BitBucket to maybe consolidate the
contributions or changes others are making. I welcome everybody to make
contributions to the transform there.

The link to the BitBucket project is:
https://bitbucket.org/mcdeck/cunit-to-junit

DIY NAS #1: Introduction

I’ve been pondering the thought of buying a NAS system for quite some
time now but could not find a suitable one. My requirements are quite
challenging though, I must admit:

  • Silent: I want to run the system near my desk so it has to be very
    silent. (And I mean super-silent – not just a little silent)
  • Fast: Access to the drives has to be fast.
  • Feature-Rich Software: I want to use the NAS primarily as a NAS, but
    it should offer features like BT downloading and the like. It should
    also act as a iTunes and/or DLNA server.
  • Hardware Features:
    - Wake-On-Lan / Wake-on-Wan
    - Inegrated WLAN Adapter
    - Raid 1, 5 or 10; 10 preferred

I bought some off-the-shelf NAS, namely the Synology DS411slim and
the Acer easyStore H341. Both were decent devices but were not really
up to the challenge. I ended up returning both of them.

The easyStore was fast and feature-rich, running Windows Home Server
(not 2011, though, but the old one). However, it was noisy which I could
not stand.

The DS411slim was silent enough and had some nice features via
DiskStation Manager, but felt painfully slow when copying to and from
the device. Plus it was rather expensive, working with 2.5" HDDs, and
thus also very limited in storage capacity (4x1GB maximum).

After searching for some days now for suitably silent yet powerful
device I could not come up with something good. There is the Western
Digitial MyBook WorldEdition II
but that did not receive many positive
comments on Amazon (for example). Some even state it tends to loose
data, which is quite against the point of buying a NAS in the first
place. Also Buffalo seems to offers some decent devices but they are
either only single drive solutions like the LinkStation Live LS-CHL
or – like the LinkStation Pro Duo LS-WVLR1 – not very reliable (from
what reviews say).

Other devices from QNAP (TS-419P+), Synology (DS411+II, DS410)
and Thecus (N4200) all seem viable options. I feel though that any
single one of them has one flaw or another.

The QNAP TS-419P+ seems to be quite silent (though I could not find any
absolute numbers yet) and offers RAID5. It runs an embedded linux and
has a download manager built in. It feels quite pricy though at about
€480,- (Amazonimage0) without HDDs. The same holds basically for
the DS411+II, which is even more pricy at €520,-
(Amazonimage1)
without any drives again. The same holds for the DS410 at around €450,-
(Amazonimage2).
Sadly, also the Thecus N4200 runs at a stunning €440,-
(Amazonimage3).
Also none of the above come with an integrated WLAN adapter.

That said, you can begin to feel that you should check other options as
well, but which would that be? Easy: build your own NAS! That can’t be
more expensive, can it? The downside obviously is that it is a bit more
work to set up such a system and also the maintenance might be harder.
On the upside, you are free in choice of operating system and features
and you could think about transforming your NAS into a HTPC if you chose
components carefully. Also of interest is the question of power
consumption compared to the other, more integrated solutions and lastly
also the price of the rig.

In the next part of this article series I will present a selection of
components that could be used to build a DIY NAS/HomeServer to suit my
needs.

Setting default applications in Ubuntu 10.10

After installing Adobe Reader into my Ubuntu installation I noticed that
the Adobe Reader was actually making itself at home as the default
viewer for PDF files. However, I do only rarely use it because I feel
that the Ubuntu Document Viewer (aka evince) is the quicker and better
solution most of the time.

Now, I could change the default for double-clicking by going through any
PDF file’s Properties->Open With and selecting Document Viewer
there. Firefox and Thunderbird for one thing did not really seem to care
at all about this, though. Both kept using Adobe Reader to open PDF
files when double-clicking in the download window and only offered the
reader as the default in the file save dialog.

To work around this problem, I came up with a solution using
xdg-mime.

In a terminal, I did

porten@lx-porten:~$ xdg-mime query default application/pdfacroread.desktop
porten@lx-porten:~$ sudo xdg-mime default evince.desktop application/pdf
porten@lx-porten:~$ xdg-mime query default application/pdfevince.desktop

Obviously the important part is the xdg-mime default. I would
actually have expected setting the default via the file properties to
have the same effect, but alas, it does not.

References:

https://bbs.archlinux.org/viewtopic.php?pid=732724

http://manpages.ubuntu.com/manpages/maverick/man1/xdg-mime.1.html

Testing XMLRPC Controllers in Pylons

Testing is one of the essential tasks in modern software development, so
it is only natural to want to test an application as thoroughly as
possible.

I’m currently working on an application that offers a service via
xmlrpc. For a client side implementaion I use python’s xmlrpclib and
that is working well. Then again, I obviously want to test the xmlrpc
functionality also during regular unit and functional tests and not only
using the client. What’s more, having to instantiate a server to be able
to run tests is cumbersome at best.

Unfortunatly, Pylons does not offer a method to test xmlrpc controllers
out of the box. The solution I found is not quite as complicated as I
had feared at first. Read more after the break.

For the above mentiond reasons, here is my solution for testing a
XMLRPCController in Pylons using a mock transport for xmlrpclib.

This solution is largely based on the excellent post by Jean Schruger
on his Blog
.

from myapp.tests import *

import sys
from StringIO import StringIO
import xmlrpclib
from xmlrpclib import ServerProxy

class WSGILikeHTTP(object):
  def __init__(self, host, app):
    self.app = app
    self.headers = {}
    self.content = StringIO()

  def putrequest(self, method, handler):
    self.method = method
    self.handler = handler

  def putheader(self, key, value):
    self.headers[key] = value

  def endheaders(self):
    pass

  def send(self, body):
    self.body = body

  def getfile(self):
    return self.content

  def getreply(self):
    if self.method == "POST":
      r = self.app.post(self.handler,
                        headers = self.headers,
                        params = self.body )
      self.content = StringIO(r.response.unicode_body)
    return (200, None, None)

class WSGIAppTransport(xmlrpclib.Transport):
  # Only here to pass the 'app'
  def __init__(self, app):
    xmlrpclib.Transport.__init__(self)
    self.app = app

  # return the fake httplib.HTTP(host)
  def make_connection(self, host):
    host, extra_headers, x509 = self.get_host_info(host)
    return WSGILikeHTTP(host, self.app)


class TestXmlrpcController(TestController):

    def test_index(self):
      server = ServerProxy('http://admin:admin@dummy/xmlrpc',
                            transport=WSGIAppTransport(self.app))
      print >> sys.stderr, server.system.listMethods()

Pylons and Apache with repoze.who BasicAuth

To be able to use Basic Authentification in repoze.who running in a wsgi
app in your Apache installation, you need to tell Apache to
WSGIPassAuthorization. An Apache configuration like this will do:

# Setup mod_wsgi
WSGIPythonHome /var/www/pylons/runtime
WSGIScriptAlias /myApp /var/www/pylons/myApp/mod_wsgi/dispatch.wsgi
WSGIPassAuthorization On

<directory /var/www/pylons/myApp/mod_wsgi>
Order deny,allow
Allow from all
</directory>

The importanat part here is

WSGIPassAuthorization On

This will pass HTTP authorisation headers through to the WSGI
application.

Sources: Repoze-dev Mailing List, modwsgi Wiki

Software Architect