Category Archives: Agile

Links

There is a new version of gunicorn, 19.0 which has a couple of significant changes, including some interesting workers (gthread and gaiohttp) and actually responding to signals properly, which will make it work with Heroku.

The HTTP RFC, 2616, is now officially obsolete. It has been replaced by a bunch of RFCs from 7230 to 7235, covering different parts of the specification. The new RFCs look loads better, and it’s worth having a look through them to get familiar with them.

Some kind person has produced a recommended set of SSL directives for common webservers, which provide an A+ on the SSL Labs test, while still supporting older IEs. We’ve struggled to find a decent config for SSL that provides broad browser support, whilst also having the best levels of encryption, so this is very useful.

A few people are still struggling with Git.  There are lots of git tutorials around the Internet, but this one from Git Tower looks like it might be the best for the complete beginner. You know it’s for noobs, of course, because they make a client for the Mac :)

I haven’t seen a lot of noise about this, but the EU has outlawed pre-ticked checkboxes.  We have always recommended that these are not used, since they are evil UX, but now there’s an argument that might persuade everyone.

Here is a really nice post about splitting user stories. I think we are pretty good at this anyhow, but this is a nice way of describing the approach.

@monkchips gave a talk at IBM Impact about the effect of Mobile First. I think we’re on the right page with most of these things, but it’s interesting to see mobile called-out as one of the key drivers for these changes.

I’d not come across the REST Cookbook before, but here is a decent summary of how to treat PUT vs POST when designing RESTful APIs.

Fastly have produced a spectacularly detailed article about how to get tracking cookies working with Varnish.  This is very relevant to consumer facing projects.

This post from Thought Works is absolutely spot on, and I think accurately describes an important aspect of testing The Software Testing Cupcake.

As an example for how to make unit tests less fragile, this is a decent description of how to isolate tests, which is a key technique.

The examples are Ruby, but the principle is valid everywhere. Still on unit testing, Facebook have open sourced a Javascript unit testing framework called Jest. It looks really very good.

A nice implementation of “sudo mode” for Django. This ensures the user has recently entered their password, and is suitable for protecting particularly valuable assets in a web application like profile views or stored card payments.

If you are using Redis directly from Python, rather than through Django’s cache wrappers, then HOT Redis looks useful. This provides atomic operations for compound Python types stored within Redis.

Just Enough Documentation: an ideal web design process

A recent project reminded me how much I like the “just enough documentation” approach, and how inefficiently web design is often done.

Tasked with redesigning an Agile project management web application (to improve usability but not re-brand), I used the following process:

1. Paper sketching for a day or so to get my ideas straight about page and widget layout. This is the best time to explore wildly different alternatives, and make most of the big decisions.

2. Wireframes in Omnigraffle of the key pages. Since the visual style wasn’t changing, I could do high-fidelity wireframes closely resembling the final design. I work on a large format, so there’s ample space to overlay notes and state changes, as well as alternative ideas. The wireframes were shared with the client on a continuous basis for feedback.

Lots of things weren’t wireframed, including less critical pages, and many dynamic states.

3. I did only two pages in Photoshop, in which I did just enough to give me my final measurements, and the few image elements that needed cutting out. I refined designs a little from the wireframes, but most of this I left to the next stage. The wireframes and this stage together took about 5 days.

4. HTML and CSS. Because in my design I was switching from a fixed-width layout to a liquid layout, you can only really appraise the design once you implement it, so it was imperative to get to this stage as quickly as possible. It is also far, far more efficient to fine-tune margins, typography and colours in CSS than in Photoshop. Furthermore, I could quickly mock up most dynamic interactions – of which this app has plenty – in jQuery (which I had not used before but was ridiculously easy to pick up.) How much more effective than doing so in wireframes, where you (and the client) have to imagine how it will “feel” to the user and you can never be sure whether it will work?

By far the bulk of my time – about 2 weeks – was spent working in HTML and CSS. Pages or elements that weren’t wireframed were much quicker to design directly as HTML and CSS, without the duplication of effort. New ideas inevitably arise during this stage, and are immediately incorporated. I occasionally went back to paper or Photoshop or Omnigraffle to work out small details, but there was no need to keep those design documents “up to date” – they had served their purpose. The client was able to appraise the designs exactly as it renders in the browser, and the resulting artifact was also ready to be integrated into their back-end.

Not only could the client and I be confident about how the design will actually look once implemented, I could also ensure that accessibility measures were built in, as well as the ability for it to be easily re-skinned. (As the app is sometimes white-labelled.)

Conclusion

If I reflect on the design process on many past projects, most seem ridiculously inefficient. Where UX designers had to exhaustively wireframe every single page and every possible interaction, with complex state diagrams for all the dynamic elements. And to keep the wireframes up to date throughout the course of the project. Where designers had to mock up dozens of pages in Photoshop, with everything finalised to the last pixel (but usually omitting hover effects and never considering liquid layout or browser differences). All the foregoing needing to be done before a client will sign it off and a single line of markup can be written. The front-end developer reduced to an automaton incapable of taste or decisions, who just had to implement the Photoshop mockups to the pixel, and all interactions exactly as specified in the wireframes. It was usually impossible to refine aspects of design or interaction at this stage (everything having been signed off), let alone adding new ideas, despite it being the most fluid.

Obviously this won’t suit all projects or clients. For example, I had the advantage of not starting from scratch, and a client who was prepared to work in this way. It was a relatively small project with few stakeholders. On some projects there may be good reasons for an exhaustively documented waterfall process. But very often in my experience it was simply because it was the only way the agency knew how to work, due to the excessive compartmentalisation of skills, and an inadequate understanding of the meaning of web design. And my, what a lot of time and money it wastes.

QCon London 2010

A couple of us went to QCon London last week, which as usual had some excellent speakers and some cutting edge stuff.  QCon bills itself as “enterprise software development conference designed for team leads, architects and project management”, but it has a reputation for being an awful lot more interesting than that.  In particular it covers a lot of cutting-edge work in architecture.

Scale, scale, scale

What that means in 2010 is scale, scale, scale – how do you service a bazillion people.  In summary, nobody really has a clue.  There were presentations from Facebook, Skype, BBC, Sky and others on how they’ve scaled out, as well as presentations on various architectural patterns that lend themselves to scale.

Everyone has done it differently using solutions tailored to their specific problem-space, pretty much all using Open Source technology but generally building something in-house to help them manage scale.  This is unfortunate – it would be lovely to have a silver bullet for the scale problem.

Functional languages

From the academics there is a strong consensus that functional languages are the way forward, with loads of people championing Erlang.  I’m a big fan of Erlang myself, and we’ve got a few Erlang coders here at Isotoma.

There was also some interesting stuff on other functional approaches to concurrency, in Haskell specifically and in general.  One of the great benefits of functional languages is their ability to defer execution through lazy evaluation, which showed some remarkable performance benefits compared with more traditional data synchronisation approaches.  I’d have to wave my hands to explain it better, sorry.

Real-world solutions

Erlang is now being used in production in some big scale outs now too: the BBC are using CouchDB, which they gave a glowing report to.

Skype are using Postgres (our preferred RDBMS here) and achieving remarkable scale using pretty simple technologies like pgbouncer.  The architect speaking for Skype said one of their databases had 60 billion rows, spread over 64 servers, and that it was performing fine.  That’s a level of scale that’s outside what you’d normally consider sane.

They did need a dedicated team of seriously clever people though – and that’s one of the themes from all the really big shops who talked, that they needed large, dedicated teams of very highly-paid engineers.  Serious scale right now is not an off-the-shelf option.

NoSQL

Erlang starred in one of the other big themes being discussed, NoSQL databases.  We’ve had our own experience with these here, specifically using Oracle’s dbXML, with not fantastic results.  XML is really not suited to large scale performance unfortunately.  Some of the other databases being talked about now though: Cassandra from Facebook, CouchDB and Voldemort from Amazon.

None of these are silver bullets either though – many of them do very little heavy lifting for you – often your application needs custom consistency or transaction handling, or you get unpredictable caching (i.e. “eventual consistency”).  You need to architect around your user’s actual requirements, you can’t use an off-the-shelf architecture and deploy it for everyone.

The need to design around your user’s was put very eloquently by Udi Dahan in his Command-Query Responsibility Segregation talk.  This was excellent, and it was pleasant to discover that an architecture we’d already derived ourselves from first principles (which I can’t talk about yet) had an actual name and everything!  In particular he concentrated on divining User Intent rather than throwing in your normal GUI toolkit for building UIs – he took data grids to pieces, and championed the use of asynchronous notification.  The idea of a notification stream as part of a call-centre automation system, rather than hitting F5 to reload repeatedly, was particularly well told.

DevOps, Agile and Kanban

Some of the other tracks were particularly relevant to us.  The DevOps movement attempts to make it easier for development and operations teams to work closely together.  For anyone who has worked in this industry this will be familiar issue – development and ops have different definitions of success, and different expectations from their customers.  When these come into conflict, everyone gets hurt.

There was a great presentation from Simon Stewart of webdriver fame about his role as a System Engineer in Test at Google, where they have around one SET to 7 or 8 developers to help productionise the software, provide a proper test plan and generally improve the productivity and quality of code by applying ops and automated testing principles to development.

One of the things we’ve experienced a lot here over the last year, as we’ve grown, is that there are a lot of bottlenecks, pinch points and pain in areas outside development too.  Agile addresses a lot of the issues in a development team, but doesn’t address any of the rest of the process of going from nothing to running software in production.  We’ve experienced this with pain in QA, productionisation, documentation, project management, specification – in fact every area outside actual coding!

Lean Kanban attempts to address this, with methods adopted from heavy industry. I’m not going to talk about it here, but there’s definitely a role for this kind of process management, if you can get your customer on-side.

Training and Software Craftsmanship

Finally what I think was the most interesting talk of the conference and one directly relevant to my current work, Jason Gorman gave a fantastic talk about a training scheme he is running with the BBC to improve software craftsmanship using peer-review.  I’ll be trying this out at Isotoma, and I’ll blog about it too!

Agile modelling

A truly excellent article on Scaling Agile development through architecture over at Agile Journal. We’ve been discovering a lot of these rules ourselves are our project size and complexity increases, and I can recommend them all.

It’s still difficult though – we’ve certainly found that sometimes it’s virtually impossible to predict all the key points of an architecture early. The old agile stalwarts of good test coverage and lots-and-lots-of-refactoring have to come in.