Filed under: Uncategorized
At work we have a small, but perfectly formed set of Java applications that help run our business, and we have a well practised methodology for developing, testing and deploying all our modules.
The perfectly formed claim above is a lie. The various modules are a rats nest of inter-dependencies and there’s an incestuous level of circular dependency between some of these modules. And while our methodology for testing and deployment might be well practiced, it’s certainly not straight forward and could easily be streamlined and improved with better tools.
On our old build box, which released it’s magic smoke some months ago, a build cycle would take over half an hour because it had to start with the application with the most dependants and work it’s way down the stack from there. That’s hardly good for instant feedback when your check in breaks the build.
In order to be able to implement Continuous Integration for our Java stack, there were several problems we needed to solve:
- Monolithic build
- Dependency Nightmare
- Deployments
- Visibility of build statistics.
There are others, but these were the main ones for me. The old build platform was based on ant and Cruise Control, using ant wasn’t bad, but for new developers there was quite a bit of work involved in getting a module built. This wasn’t all down to ant, but it didn’t make it easy to manage a projects dependencies.
Some of the 3rd party dependencies we have are ancient, and it’s hard to find the right jar file for most of them. There are reasons for this, but I’m not going to go into them here.
This is where Maven comes in, it’s ability to manage any dependencies a module might have was the biggest draw, tell it which library you want, and what version and let it worry about where it comes from. So I picked the module that had no dependencies on any of our other applications and started to convert that.
The biggest lesson I learned from that is do not try and munge the Maven pom to fit your file structure, change your file structure to match what Maven expects, once you’ve done a couple, it really doesn’t take that long. As well as making the conversion smoother, in the long run you get the benefit of every project looking the same, so new developers know where the source is, they know where the tests are and they know where the extra resources like jboss config files are. It’s a good example of convention over configuration.
The next problem you’ll come across is working out which of the many Maven repositories you’ll need to use, many of the libraries are duplicated across more than one repo, and in some instances the same class is available in multiple artifacts. I didn’t find a quick solution to this, it was instinct & trial and error. Some of them are obvious though.
Once we had the first module converted over, we started to look at the next one. We quickly noticed that many of our apps share dependencies, so I created a parent pom file that they can all share to get the same dependencies. This also made it easy to plug in the code quality checks later as well.
So I’m nearly at the end of this post, and I haven’t mentioned Hudson yet. Why did I choose Hudson over the incumbent Cruise Control? Well the build scripts that built the apps on the build box were different to the ones that developers used to build the same app on their dev box, and I wanted to unify that, also when I looked at the 2, Hudson was up and building something from scratch in about 30 minutes, and cruise control wasn’t even anywhere close. Initially I had some problems with stability, the Hudson process would become unresponsive over night, moving from the default Winstone app server to Tomcat seems to have sorted this out though. Watch out for file permissions in the Hudson directory though if you do the same, it made a plugin I use vanish, which made all my builds that depend on it disappear for a time too!
Using maven also solves the monolithic build problem because each module just becomes a dependency the same way log4j or junit are. It doesn’t help the circular dependencies either, you have to fix them yourself. Using a repository manager like Nexus is a really good idea too.
Creating a project in Hudson is easy, point it at your code repository and away you go. Simplicity is crucial if you want to get everyone in the department on board and it doesn’t get much simpler than that.
I’d like to say a massive thanks to the members of the Sheffield Java User Group who helped me get over some of the initial problems I had using Maven.
Next I’m going to cover the various Hudson & Maven plugins that we use.
Filed under: politics
For the past 6 months or so I have been delivering leaflets for the local Liberal Democrat party, and as you might imagine, the number of leaflets has increased recently, anyone would think there was an election on.
- Some people have driveways that are far too long
- Some people have too many steps.
- It’s a great time for listening to audio books, I like SciFi myself, perhaps a little Cory Docterow?, or Kim Stanley Robinson. Red Mars is an epic story and I can’t wait to listen to the next 2.
- Always open the inner letterbox flap by inserting you hand palm up, if you don’t, you might end up scratching your knuckles.
- Don’t ben too worried about dogs, unless they’re waiting for you on the other side of the letterbox you’ll have plenty of time to make your getaway.
- Marks and Spencer carrier bags are not strong enough, you’ll be lucky to get a half a round out of it before one of the handles goes. Blackburn Rovers store bags survive a few rounds.
- Keep your back straight while walking, don’t slouch.
- Make sure the leaflets go the whole way through, don’t leave them sticking out. My Paperboy training came in really useful here.
- Push other leaflets through too.
- Don’t be tempted to remove opposition leaflets if they’re poking out.
- Don’t drop your elastic bands.
- Take a drink.
- Don’t try and teach your son to ride his bike at the same time, you’ll get nowhere.
Do you have any tips? Any advice for fellow leafleteers?
I wrote to Angela Smith asking her not to rush through the Digital Economy Bill, here is the 2nd response I got, the first being a letter explaining my letter had been forwarded to another department.
Here is the letter in full:
07 April 2010
Our Ref: Digital Economy Bill
Dear Constituent,
Thank you for your recent email concerning the digital economy bill and the prospect of this bill becoming legislation.
Although I would have voted in favour of the bill at its second reading, none of the opposition parties provided any opposition to advancing the bill. As a consequence of this, no vote was held and so the bill progressed to committee stage unopposed. I do however feel compelled to explain the reasons why I was in favour of the bill’s passage at second reading.
It is the case that businesses crucial to the UK’s media and technology economy will greatly benefit from the implementation of the bill. Such companies include Channel 4, whose main benefit would be an extension of its public service remit. The Bill will legally require Channel 4 to invest in film, thereby securing its already considerable commitment to this medium.
The other provision I strongly support is that relating to the strengthening of online protection of children and young teenagers. There have been numerous cases of children downloading information, using internet gaming and using social networking sites without their parents’ supervision or permission. The internet is one of the most influential and therefore powerful tools in the world, and there is little doubt that this level of unregulated and free information can have dangerous consequences on an impressionable mind.
A major area of controversy that the bill seeks to readdress would be the issue of online gaming. Rather than there being two regulatory boards examining and classifying the suitability of games, there would now be just one, avoiding the confusion on gaming suitability that currently exists. Secondly, the bill would put in place a strict procedure when games rated for ages 12, 16 and 18 are supplied to people under these ages, as it simplifies and clarifies the law on age-related status for retailers, parents and children, ensuring that children are not exposed to any material that is not deemed appropriate for them. Finally, it will create an age ratings system harmonised across 30 European countries so that children will not become exposed to offensive gaming material from other countries. Whilst the majority of research conducted on the internet is useful and informative, I believe that where we can take reasonable measures to protect children then this action should be taken.
It is also the case that the Bill has been thoroughly debated in the Lords and amended in relation to the provisions on copyright. It is my understanding that a clear and effective appeal mechanism has been put in place and that before Regulations could be implemented there would be full consultation and Parliamentary scrutiny.
Yours sincerely
Angela Smith MP
Sheffield Hillsborough
So in short, my MP would have voted for the bill because it had a couple of provisions in it that protected children, regardless of how many bad provisions there are in the bill. Surely that is a reason NOT to vote for it and hold it over for proper debate and proper consultation in the next parliament!
Update: As Saul pointed out on Twitter, the last paragraph says it all “It is also the case that the Bill has been thoroughly debated in the Lords“, an unelected and unaccountable body, and that is one of the complaints against the bill.
I’ll update the post when I send a reply.
I’ve previously asserted that government should be assigning more of it’s ICT projects to smaller companies, and after reading the Digital Economy Bill and the various ammendments by Lords I’ve come to realise that politics needs more nerds.
It’s clear that the current batch of politicians and lords don’t understand technology both from the point of view of how it works, nor from the angle for how it has effected, changed and bettered society.
Chris Huhne’s call for police to be given the power to monitor sex offenders use of the internet is well intended, but the means by which the police are to acheive it expose the kind of lack of understanding that we see all to often.
Neither an IP address or even an email address are identity tokens, you cannot definitivly tie them back to an individual. In my house there are 2 computers, and one of those is shared so to imply that I have been browsing the CBBC website or reading email on Yahoo is completly impossible.
What if I were to access the sites form the local internet cafe? The free wifi you get in places like Starbucks (ah! the DEB will take care of that), what if I use the WIFI from the flat a couple of floors below because they havn’t secured it, or they use a service like FONera that offers free wifi?
Add on to that the fact that ISP don’t give out the same IP address to the same account each time, mix in a little anonymous proxy or two and services such as TOR it’s obvious that you can’t rely on an IP address being attached to an individual.
What about email addresses, everyone knows they’re an easy way to identify someone, right? Wrong! There are a multitude of service providers that will give you an email address, some even without having to sign up for an account, most of these are banned from being used as email addresses, but it’s not hard to imagine services that keep themselves off the radar.
None of that matters however because you can go and sign up for a Google Mail address right now and the recovery email feild is optional, you can have an email adress that will log you in in seconds, and that’s why you can’t rely on an email address as an identity token.
And this is why politics needs more geeks, so that politics as a whole will become more rounded and a better representation of the people who elect them and so the naive one in me would like to think they’d be consulted before statements like this are made or bills like the DEB are poposed or ammended.
Yesterday I gave my views on the apps and services I use most often on my iPhone and compared them to what the N900 had to offer, the iPhone won 3-1, but a lot of the comparisons were a draw.
So today I’m going to tell you what I think of the hardware & the OS. I’ll point out right now that I’m comparing the N900 to MY iPhone, which is a 3G, and not the newer 3Gs
In the box are the usual wires, some headphones, a wall wart charger and a USB lead. I was dissapointed that the N900 didn’t use a standard mini USB cable like so many other devices do, it means you are tied to their peripherals much like you are with Apple’s. The box also contains an AV lead which you can plug into a TV and play your media through, very nice.
In terms of looks, the iPhone wins hands down, it’s more polished, their apps all share the same UI and follow (in the main) the Apple guidelines. The N900 UI doesn’t look as polished and many of the apps handle similar tasks in different ways.
In terms of size, the iPhone is about half as thick, 10mm longer and the same width, but they weigh the same as far as I can tell. The screen on the iPhone is slightly larger and when you’re using your fingers it’s capacative screens is much more responsive than the N900, but you can use a stylus on the N900 for much finer control.
Call quality and making calls on both devices is very easy, and the N900 has excellent audio quality, but it has dropped 2 calls in the 3 days I’ve been using it.
The Maemo OS has one massive advantage over Apple’s iPhone OS, it’s free and open source. I can download tools and develop apps for it, just like the iPhone, but I have the choice of 3 languages and 2 GUI toolkits. The latter is what causes the inconsistency in part, but the next version of Maemo will switch to use Qt as it’s toolkit. Another barrier to entry for iPhone dev is that you need to fork out $99 a year to be able to run your apps on an actual device.
I’m trying to write a better Twitter client for the iPhone, but how far I’ll get in the 2 weeks before the phone goes back, I don’t know. Development on OSX has to be done using the provided VM image, which has a 800×600 resolution and runs like a dog on older Apple hardware. I’m hopefull that developing PyQt apps will be easier though, one demo suggests you can deploy to the device just by copying the source over, which will be a great improvment.
Apple also put another wall in between the developer and getting their apps out, and this is either a good thing, or a bad thing, depending on how you look at it. It’s a good thing as it can act as a quality filter (however low they set the bar) or it’s a bad thing because it means they can refuse any app they fancy, or make an eReader have an adult rating because you can use it to read The Karma Sutra.
With the N900 anyone can upload apps to Nokia’s Ovi, which has the same quality gates as the App Store, or you can install stuff via a normal debian package, as a company you can even create your own repository and people can add it to their phones. The Ovi store is where developers can sell their apps, and like Apple, Nokia take a 30% cut of what you sell it for (minus costs) and have a €50 fee to join the store.
The documentation for Maemo is also very good, with plenty of examples, an easy to follow introduction, but these are focused on C or C++, but Meego.com has some basic tutorials for Qt. MeeGo is what the next version of Meamo will be called, after it’s merger with Intel’s Moblin.
So with all these things in it’s favour the N900 is ideally positioned to explode, but there’s 1 thing standing in it’s way … adoption. Developers won’t write apps for it unless they can see they could make some money, and people won’t buy the phone unless they can see the apps they want available for it. Just look at all the apps that were available when the App Store launched, Nokia can’t rely on a build it and they will come strategy, they need to work with developers of all kinds to build the kind of apps people expect to see when they buy the phone. Offer developers the hardware & the support they need to build their apps and not just rely on creating a website and releasing the phone, “Build it and they will come” doesn’t apply unfortunatly.
So, out of the 2 phones, I think the N900 wins, it’s dev tools, better hardware spec and open source nature just beat the iPhone. If I was a new user looking at the 2 devices I’d probably go for the N900.
Last week I was getting quite pissed of with the amount of time my iPhone is taking to wake up from sleep mode, which means I miss quite a few calls because I can’t slide the answer slider!
So this week I took delivery of the new Nokia N900 for a 2 week trial, and here are my inital impressions.
I’ve drawn up a list of all the stuff I use my iPhone for to see how the N900 compares…
- Web Browsing
- Music
- Google Reader
- Maps
- Calendar
- Foursqaure/Gowala
- EBook reader
Web Browsing
The web browser that comes with the N900 is very slick, just as good as the Safari based one that comes on the iPhone, it uses the Gecko rendering engine which is the same one that firefox uses, so it’s probably safe to assume that anything that looks good in Firefox will look the same on the N900.
Verdict: A draw
None of the native twitter apps I tried are very good. Maeku doesn’t let you view @ replies or direct messages and kept telling me I had exceeded my rate limit when other apps worked fine.
Verdict: iPhone Wins
Google Reader
A side effect of the browser being so good is that some services don’t recognise it as a mobile browser, so won’t let you browse their mobile site. The Google Mobile suite of apps being a prime example. Visit http://m/google.com/reader (or any of the mobile sites) and you get redirected to http://www.google.com/mobile/more/ and as yet I havn’t found a workaround.
I tried to use the full site, but displayed on such a small screen, it doesn’t work very well.
Verdict: iPhone wins
Maps
The N900 uses Nokia’s Ovi for it’s mapping, and they look lovely, the N900 comes with a built in compas which lets the maps rotate as you move which is really nice, it helps so much with orientation. I was trying to find a building in Sheffield and the rotating maps made it much easier to work out where I was going. Admitidly, I only have a 3G iphone, not the newer 3Gs which also has the magnetometer.
Having said that, the directions are for driving only and the satalite view is much more detailed on the iPhone.
Verdict: In this limited test, the N900 won.
Calendar
There are many posts about people struggling to get their phone synced to Google for contacts and calendar, but I had no problems when I followed these instructions
Verdict: A Draw
FourSquare/Gowalla
Ok, so these 2 are pretty frivolous, but I’m using them quite a lot at the moment, so I’ll include them here. There are no native apps at the moment, but they both have mobile versions of the site.
Gowalla causes a javascript error at the moment, so I can’t use the rest of the site and the Foursquare mobile site doesn’t seem to use GPS at all, I have to enter it manually.
Verdict: Harsh because it’s not the N900’s fault, but the iPhone wins
eBook Reader
I havn’t had chance to use the ebook reader yet.
Round 1
So at the end of the first post, the iPhone is ahead on apps, but that’s not really a suprise to anyone is it. Part 2 will compare the device & OS itself, and I expect a much closer contest.
In the last post, I looked at moving all your data from one app to another, and today I’m going to go through using Django’s unit test extensions to test the app. This post also covers using signals to validate models that arn’t created from a form. You should of course also unit test the methods on your models outside of the views they’re called from, but there’s nothing special there, and there’s a whole bunch of blog posts that cover them.
The main aim of whos-playing.com is to make organising 5-a-side matches easy, so it’s main features are all around players saying they can or can’t make the game. A player can say they’ll play, but then change their mind, they can only play in matches for teams they are a member of, and they can only say THEY will play, not anyone else, the exception to this being the team organiser, he can add or remove any player. Lets look at the various things we might want to test.
Continue reading
I’ve written a Django app to help people who organise social 5 a side games, (yeah, I know I said it was dead, but I left it up and it started to get a few users, it’s still not a success, but it is being used). It was the first “large” scale Django app I’d attempted and so it’s got a lot of things wrong with it, for starters I didn’t really see where I could split the project up into seperate Django apps, but I understand that a lot better now, so as I add new features I’m also splitting the project out into seperate apps. This blog post covers how I do it, and hopefully you’ll find it usefull.
I’m making a start from this post on Stackoverflow.
I’m already using south for migrations, so it makes sense to use that for refactoring too. If you havn’t already, install it and convert your app over.
Everything I created originally was under an app called, ‘sheets’, and now I want to split the parts of this that control creating and playing matches into a seperate app, which I’m going to call it ‘match’, don’t forget to add it to your INSTALLED_APPS
./manage.py startapp match
Now move the models over to your new models.py file, you’ll have to add a related_name parameter to some of your models if you’re using Postgres because south will try and create a relation with the same name as the existing models relation.
stuart-grimshaws-macbook:teamsheet stuartgrimshaw$ ./manage.py startmigration match --initial Error: One or more models did not validate: sheets.match: Accessor for field 'location' clashes with related field 'Location.match_set'. Add a related_name argument to the definition for 'location'. sheets.matchplayers: Accessor for field 'player' clashes with related field 'User.matchplayers_set'. Add a related_name argument to the definition for 'player'. sheets.sides: Accessor for field 'player' clashes with related field 'User.sides_set'. Add a related_name argument to the definition for 'player'. sheets.sides: Accessor for field 'side' clashes with related field 'SideNames.sides_set'. Add a related_name argument to the definition for 'side'. match.match: Accessor for field 'location' clashes with related field 'Location.match_set'. Add a related_name argument to the definition for 'location'. match.matchplayers: Accessor for field 'player' clashes with related field 'User.matchplayers_set'. Add a related_name argument to the definition for 'player'. match.sides: Accessor for field 'player' clashes with related field 'User.sides_set'. Add a related_name argument to the definition for 'player'. match.sides: Accessor for field 'side' clashes with related field 'SideNames.sides_set'. Add a related_name argument to the definition for 'side'.
Don’t be tempted to dive right into your new migration and add your code to migrate data from the old models to the new ones, South recommends that you do a data migration in 3 steps, which makes sense when you consider that doing it in 1 big step might have it’s shortcomings.
The startmigration –initial command created the first migration, this simply created the tables in your database. Run migrate to make these alterations:
./manage.py migrate
Next, you need to create the migration that actually moves the data from one set of files to another:
./manage.py startmigration match sheets-to-match-data
The migration in the example on Stackoverflow is actually a lot more complicated than ours, we simply need to rename the database tables, like this:
def forwards(self, orm):
db.rename_table('sheets_match', 'match_match')
db.rename_table('sheets_matchplayers', 'match_matchplayers')
db.rename_table('sheets_sides', 'match_sides')
def backwards(self, orm):
db.rename_table('match_match', 'sheets_match')
db.rename_table('match_matchplayers', 'sides_matchplayers')
db.rename_table('match_sides', 'sides_sides')
So, that’s the data migrated over, now we need to migrate any views to the new app, just identify and copy them, one of the mistakes I made in the original attempt actually helps me out here, I created a seperate urls.py for some sections, that was my first clue I was doing something wrong.
After copying this file over to the new apps urls.py, it’s time to take the views it mentions and copy them to the new apps views file, once you’re happy that they’re working, you can delete them from the old views file too.
How do you know they’re working? Well, you have been using Django’s unit tests havn’t you? No? Well that’s another post then.
Filed under: Uncategorized
While trying to work out how the excellent tab completion works for git, I found /etc/bash_completion.d/git which mentions the __git_ps1 command, you can use this to put the current branch in your bash prompt
here’s mine:
PS1='${debian_chroot:+($debian_chroot)}\[\ 033\[01;34m\]
\u@\h\[\ 033\[00m\]:\[\ 033\[01;33m\]\w\[\ 033\[1;32m\]
$(__git_ps1 " (%s)")\[\ 033\[00m\]$ '
(Remove the space from \ 033, WordPress wouldn’t escape it properly)
The magic bit being
$(__git_ps1 " (%s)")
There are a couple of options you can set too to get it to show you the status of the repo too, see the file above for more info.
Filed under: music
Here’s a list of my Top 10 6 albums of the decade, in no particular order:
Reverend and the Makers – The State of Things, superb album from John “The Reverend” McClure and his band, The Makers. I went to see them in Sheffield for the second tim this year and it was easily the best gig I’ve been to.
Serj Tankian – Elect the Dead, the first album from ex System of a Down frontman Serj Takian. Pretty much another System of a Down album, which is no bad thing. Looking at my Last.fm stats he’s the single most played artist with over 2000 plays.
Queens of the Stone age – Songs for the Deaf joined on drums by Foo Fighters front man Dave Grohl this is easily the best of the QotS albums, with songs like No-one Knows and Hanging Tree are still on my playlist today.
Foo Fighters – Echoes, Silence, Patience & Grace I love this album from the Foo Fighters, when you look at the amount of material Dave Grohl puts out it’s amazing that so much of it is so good, and this is the best Foo Fighters album of the decade.
Reverend and the Makers – A French Kiss In The Chaos The only band to have 2 albums in my list, this 2nd album from the Sheffield band is just as good as the first and was on constant repeat for about a month after I bought it!
Batman Begins Soundtrack – maybe a bit of an odd one to include in a list of top albums, but this is one of the best soundtracks to a film ever.
What are your top album’s from the last 10 years?
Stubblog 