May 24, 2012

I don't belive in software complexity

Ok, building software could be complex, but it doesn't have to. Recently I read this quote

Complexity happens when you don't understand a problem fully
It's true. We, the software craftsmen, have a very difficult task upon us: building tools for proposes we don't understand fully.  I got it screeching my fingers with a bit of bioinformatics. In finance is no different. Today I saw poor JMS queues abused to dispatch themselves multiple times before aborting. It's not fair. How can you manage the development of a piece of software that does thing you don't understand or in which you put complexity without serious business reasons? Why we do it? Because we listen to stake holders and we do not try to understand: we do exacly what they say. Human beings thinks for use cases and even for exceptions. I do not want to be that kind of engineer ... I want to be the good kind.  I want to be the kind that extract general rules. I want to bring order into chaos. Because software loves order. And this is the beauty behind the screen and the keyboard, the chance we got to be artist instead of brainless coders.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.

May 9, 2012

Travis, give me my data back!

Inspired from this blog post about "Using Travis-CI With Python and Django" I started playing around to make this recipe fit my expectation from a CI enviroment. Running tests is the bare bottom of the whole thing. Email notifications and the handy badge to display in the project webpage are ok, but if you want to get somewhere metrics and coverage reports are the key to get the most out of the Continuous Integration approach.

So I started creating a Django project on Github with this Travis configuration

 language: python  
  - "2.6"  
  - "2.7"  
  - DJANGO=1.4  
  - pip install -q Django==$DJANGO --use-mirrors  
  - pip install pep8 --use-mirrors  
  - pip install  
  - pip install coverage  
  - pip install django_coverage  
  - "pep8 --exclude=migrations --ignore=E501,E225,E261,W191,E101 ."  
  - pyflakes -x W .  
  - python test_coverage  

Basicaly I added coverage and django_coverage to the depandecy and then ran the tests with coverage report enabled. ( in real world I would use a requirement file but I will keep dep. installation this way for the sake of clarity) This is the result:

 Name            Stmts  Miss Cover  Missing  
 trango.molten_core.models    0   0  100%    
 trango.molten_core.views    0   0  100%    
 TOTAL              0   0  100%   

Ok this is just and empty app with one test always passing. But my point is to put this data somewhere outside Travis ( and gh-pages seemed a very good place ) and have an easy way to see how coverage increase or decrease in particular builds ( and who is guilty for that!)

First we have to find an easy way to identify the build commit. I found that something like
export GIT_COMMIT_ID=`git rev-parse HEAD` works just fine. Then I started speculating what was the optimum ( or at least possible) way to move away the data and I ended up with these three solutions:

HUMAN PALADIN: Make Travis-CI push back the reports

This is the first, clean , way that came to my mind. I created the gh-pages branch on my repo ( if you are unfamiliar with this please read here ) and I set django_coverage to create the html reports in a specific folder (in file)

 if os.getenv('GIT_COMMIT_ID'):  
   COVERAGE_REPORT_HTML_OUTPUT_DIR = os.path.join("reports", os.getenv('GIT_COMMIT_ID'))  

Then I tried to meke Travis-CI to push back the reports

 language: python  
  - "2.6"  
  - "2.7"  
  - DJANGO=1.4  
  - pip install -q Django==$DJANGO --use-mirrors  
  - pip install pep8 --use-mirrors  
  - pip install  
  - pip install coverage  
  - pip install django_coverage  
  - "pep8 --exclude=migrations --ignore=E501,E225,E261,W191,E101 ."  
  - pyflakes -x W .  
  - mkdir reports  
  - export GIT_COMMIT_ID=`git rev-parse HEAD`  
  - mkdir reports/$GIT_COMMIT_ID  
  - python test_coverage  
  - cp -pR reports/$GIT_COMMIT_ID /var/tmp  
  - rm -rf reports/$GIT_COMMIT_ID  
  - git branch gh-pages  
  - cp -pR /var/tmp/$GIT_COMMIT_ID reports  
  - git add reports/*  
  - git commit -m "reports for $GIT_COMMIT_ID"  
  - git push origin gh-pages  

I had to move the reports in a temp directory to quietly switch from the master branch to the gh-pages branch. And sadly this is the output of the build
 $ git branch gh-pages  
 104$ cp -pR /var/tmp/$GIT_COMMIT_ID reports  
 106$ git add reports/*  
 108$ git commit -m "reports for $GIT_COMMIT_ID"  
 109[detached HEAD 5b018bc] reports for 4b01202c546546bc424af1828eb211a011bcc278  
 110 Committer: vagrant <vagrant@nettuno.(none)>  
 111Your name and email address were configured automatically based  
 112on your username and hostname. Please check that they are accurate.  
 113You can suppress this message by setting them explicitly:  
 115  git config --global "Your Name"  
 116  git config --global  
 118After doing this, you may fix the identity used for this commit with:  
 120  git commit --amend --reset-author  
 122 3 files changed, 384 insertions(+), 0 deletions(-)  
 123 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/excludes.html  
 124 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/index.html  
 125 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/modules/trango.molten_core.models.html  
 127$ git push origin gh-pages  
 128fatal: remote error:   
 129 You can't push to git://  
 130 Use  
 134after_script: 'git push origin gh-pages' returned false.  
 135Done. Build script exited with: 1  

As you can see when Travis-CI tries to commit we got evidence that git is not configured with a proper user nor have keys or Github user to add as collaborator to my repo. So also the push fails. I think up to now this is a dead end. Possible solutions are to run Travis-CI localy ( but I would go straight in Jenkins arms instead:  the good point in favor of Travis-CI is his cheap and cloudy nature ) or that Travis-CI allows you to add specific deployments keys to each building repository.

FORSAKEN WARRIOR: Use a curl friendly file host

Searching for options of  "push-back" I found a message in the Travis-CI mailing list suggesting to use a file hosting service to send your data via curl. For example Now the problem is that you have to put your API Key, username and clear text password in the .travis.yaml file commited in a public repository. Definitively not a secure way to store files online. One solution would be to create a proxy service that uses commit hashes as ONE TIME PASSWORD. But building such a thing would require a lot of time and the result would not be "vandal proof". Maybe, someday....

GNOME ROGUE: Ajax scraping

I think another option is to add to my gh-pages some javascript that reads tha Travis-CI api to list the builds and then perform some scraping to extract data from the dom of the build details. 100 Haking ponts, 10 Fun points, 0 usability points.


Travis-CI is an amazing project but without a feature to export build results I find hard to think it would fit a project with many developers and aiming a high code quality and test coverage. I found historical CI data very important and in some case also inspiring ("Look how crappy the project were a month ago!"). Travis-CI is handy and sleek but having reports on the project web page gives an instant direction on how the project is managed. Well also the Travis badge is a really really good indicator.

I will keep it failing just to remember to fix it!

May 3, 2012

Bob Builder and the Clean Coder

As a GeekDad I often have to watch "Bob the Builder" TV Show.
After watching the same episode several time my mind had found the way to sublime the concept of the builder and I discovered that Bob is a good metaphore for the so called "Clean Coder". He is skilled , he relays on tools but actualy know how to do the job, he select the right tool for each job.
But the most important feature is that Bos is focused on team playing and work in an evironment friendly way ( SCM , your user data , UI ,UX , integration with other systems). I'm the kind of dad that tries to build wooden railways with a TDD aproach, so I think this is a good TV show for my children that could teach them to build things right.  

Can we fix it? Yes we can!

For the ones who doesn't know "The clean coder" is a book written by Robert Martin. It is  " A Code of Conduct for Professional Programmers". It also lead to a series of interesting videos  and you can follow @unclebobmartin on twitter.