Friday, March 21, 2008

armenzg: Sess16 - Automated testing on Hera

This blog post announces that I have got my hands dirty adding the steps to do automated builds on buildbot with testings on the hera server.
I have got access this last past Tuesday and the last 4 weeks I was working on the buildbot configuration for CAIRO, and this is the summary of what I have tried to do:
  • Access the buildmaster in
  • Modify the master.cfg to add the build steps for the linux machine
  • Add file and the mozconfig-linux-tests
My goal was to force a complete build with tests without ruining Adam's configuration (he set up the try server)

NOTE: This buildbot setup will allow to force builds with tests and builds without tests

NOTE: You can see the second yellow column that indicates that is building - check it yourself OR force a build (and go for a coffe)


As always, I keep a section for problems and solutions for future reference:
  • I used "buildbot restart ." instead of "buildbot reconfig ." and therefore I did put the try server down :( - BTW, do not use "restart" if you want to just re-read master.cfg so your changes are applied, because instead of using the last working copy of master.cfg (as it says) it just dies and never starts again
  • I had to put some of the environment variables (MozillaEnvironments['centos']) inside buildcustom/ file since it was defined in there and also in my code but did not catch it this way
  • At some point my "mozconfig-linux" file contained CRLF and I would get "line 17 errors" when I had only 9 lines; I used file mozconfig-linux-tests to confirm and dos2unix mozconfig-linux-tests to fix it (thanks rhelmer) - I remember the "line" errors but I can't find the log that said it - When I do cat .mozconfig, you can actually see that there are extra new lines in the output)
  • There are has been a lot of small problems that I have run into but not relevant


I am in the Hera server, I can make changes and I have got a build with automated testing.

What is left?

  • Make the other machines to build too (I do not know if they are up and running)
  • Learn how to make it build following the "tryserver" way (give a patch and run the build with tests)
  • Find out why the compiling takes forever

Friday, March 07, 2008

armenzg: Session 15: When things go wrong

I was trying to write as a file containing steps commands which derived from ShellCommand so I could have logically related steps grouped together. For instance, "", "make" and "make install", since all of them have to run to count as a "good" build. Another reason was to have a summary of the tests run, instead of having a look at the STDIO to see the results of the tests.

Well, I decided to make it happen at school were we have computers (being setup by us, here and there) for our Open Source Projects, but things went wrong:

  1. I try to get buildbot and twisted on the mac mini
  2. I want to use package manager: macPorts cannot reach its rsync server; fink can reach a cvs repository to selfupdate but takes for ever
  3. I decided to get tarball and build it myself; Buildbot only works with Twisted 2.5 but I would only have 2.4 even tough I build it (sudo python ./ install)
  4. I realized that buildbot got installed under /Libraries/python/2.5/site-packages (I think) and twisted too, but it seemed to don't catch it as 2.5
  5. I found through a blog post (I left the link at school) talking about it and it seemed I had to export PYTHONPATH to point at the right place
  6. Now I should had been ready, but when I tried to start my buildbot master I got this message:
    Traceback (most recent call last):
    File "/Library/Python/2.5/site-packages/twisted/application/", line 379, in run
    File "/Library/Python/2.5/site-packages/twisted/scripts/", line 23, in runApp
    File "/Library/Python/2.5/site-packages/twisted/application/", line 157, in run
    self.application = self.createOrGetApplication()
    File "/Library/Python/2.5/site-packages/twisted/application/", line 207, in createOrGetApplication
    application = getApplication(self.config, passphrase)
    --- ---
    File "/Library/Python/2.5/site-packages/twisted/application/", line 218, in getApplication
    application = service.loadApplication(filename, style, passphrase)
    File "/Library/Python/2.5/site-packages/twisted/application/", line 341, in loadApplication
    application = sob.loadValueFromFile(filename, 'application', passphrase)
    File "/Library/Python/2.5/site-packages/twisted/persisted/", line 215, in loadValueFromFile
    exec fileObj in d, d
    File "buildbot.tac", line 9, in
    BuildMaster(basedir, configfile).setServiceParent(application)
    File "/Library/Python/2.5/site-packages/buildbot/", line 370, in __init__
    self.status = Status(self.botmaster, self.basedir)
    File "/Library/Python/2.5/site-packages/buildbot/status/", line 1779, in __init__
    assert os.path.isdir(basedir)
And this is where I basically left it and I should find why next time I go to school or just choose one of the Linux computers and run it there, but I can't promise that everything will be easy.

CairoTests class

class CairoTests(ShellCommand):
name = "Run Cairo Tests"
warnOnFailure = True
description = ["running tests"]
descriptionDone = ["tests completed"]
command = ["make", "test"]
workdir = "cairo"

def createSummary(self, log):
passCount = 0
failCount = 0
for line in log.readlines():
if "PASS" in line:
passCount = passCount + 1
if "FAIL" in line:
failCount = failCount + 1
summary = "TestsResults:" + str(passCount) + "/" + str(failCount) + "\n"
self.addCompleteLog('summary', summary)

def evaluateCommand(self, cmd):
superResult = ShellCommand.evaluateCommand(self, cmd)
if SUCCESS != superResult:
if None !='FAIL', cmd.logs['stdio'].getText()):
return SUCCESS

After this blog post I will have to change this code because I believe it does not capture properly how many tests failed or passed.
Cairo's tests have the keywords PASS, XFAIL, UNTESTED and FAIL at the end of some lines and the summary of each group of tests is after the block of tests and appears at the beginning of the line followed by a semicolon. Check the next quoted text:

TESTING svg-surface-source
Test using various surfaces as the source
svg-surface-source-image-argb32 [0]: PASS
svg-surface-source-image-argb32 [25]: PASS

<-- There are more lines but nothing relevant -->

svg-surface-source-xlib-fallback-rgb24 [0]: UNTESTED
Failed to open display: :0.0
Failed to open display: :0.0
svg-surface-source-xlib-fallback-rgb24 [25]: UNTESTED
PASS: svg-surface-source <-- This is what I call the summary of the block of tests

We also have at the end of the STDIO that shows the results of all tests and my summary should match that OR maybe I should just show that in the summary instead:

7 of 156 tests did not behave as expected (4 unexpected passes)
Please report to
make[4]: *** [check-TESTS] Error 1
Failed tests:
ft-text-vertical-layout-type1: image test-fallback test-meta test-paginated
Failures per surface - image: 1, test-fallback: 1, test-meta: 1, test-paginated: 1.

Next releases

It seems that the Hera Try Server will be up and running something really soon and I will have to make sure that the tests are being run properly and that it meets the desires of the Mozilla community. Adam has been doing a great job and as I have heard from Shaver they will be really happy to see this working.

In my next release I will expect to have the Cairo steps for Windows and the performance tests (which I thought I could have got done for this week but did not happen) and whatever is required for me to work on the Try Server