Buildout Basics Part 1

Introduction to the series

This is the first in a 3 part series of tutorials about creating, configuring and maintaining buildout configuration files, and how buildout can be used to deploy and configure both python-based and other software.
During the course of this series, I will cover buildout configuration files, some buildout recipes and a simple overview structure of a buildout recipe. I will not cover creating a recipe, or developing buildout itself.

For a very good guide on the python packaging techniques that we will be relying on, see this guide: http://guide.python-distribute.org

All code samples will be python 2.4+ compatible, system command lines will be debian/ubuntu specific, but simple enough to generalise out to most systems (OSX and Windows included).
Where a sample project or code is required, I’ve used Django as it’s what I’m most familiar with, but this series is all about the techniques and configuration in buildout itself, it is not Django specific, so don’t be scared off if you happen to be using something else.

Buildout Basics

So, what’s this buildout thing anyway?

If you’re a python developer, or even just a python user, you will probably have come across either easy_install or pip (or both). These are pretty much two methods of achieving the same thing; namely, installing python software

$> sudo easy_install Django

This is a fairly simple command, it will install Django onto the system path, so from anywhere you can do

>>> import django
>>>

While this is handy, it’s not ideal for production deployment. System installing a package will lead to problems with maintenance, and probably also lead to version conflict problems, particuarly if you have multiple sites or environments deployed on the same server. One environment may need Django 1.1, the newest may need 1.3. There are significant differences in the framework from one major version to another, and a clean upgrade may not be possible. So, system-installing things is generally considered to be a bad idea, unless it’s guaranteed to be a dedicated machine.

So what do you do about it?
One answer is to use a virtualenv. This is a python package that will create a ‘clean’ python environment in a particular directory:

$> virtualenv deployment --no-site-packages

This will create a directory called ‘deployment’, in which is a clean python intepreter with only a local path. This environment will ignore any system-installed packages (--no-site-packages), and give you a fresh, self contained python environment.

Once you have activated this environment, you can then easy_install or pip install the packages you require, and then use them as you would normally, safe in the knowledge that anything you install in this environment is not going to affect the mission-critical sales (or pie-ordering website) process that’s running in the directory next door.

So, if virtualenv can solve this problem for us, why do we need something else, something more complex to solve essentially the same problem?

The answer, is that buildout doesn’t just solve this problem, it solves a whole lot more problems, particuarly when it comes to ‘how do I release this code to production, yet make sure I can still work on it, without breaking the release?’

Building something with buildout

The intent is to show you the parts for a buildout config, then show how it all fits together. If you want to see the final product, then dissemble it to find the overall picture, scroll to the end of this, have a look, then come back. Go on, it’s digital, this will still be here when you come back….

Config Files

Pretty much everything that happens with buildout is controlled by its config file (this isn’t quite true, but hey, ‘basics’). A config file is a simple ini (ConfigParser) style text file; that defines some sections, some options for those sections, and some choices in the options.

In this case, the sections of a buildout config file (henceforth referred to as buildout.cfg) are generally referred to as parts. The most important of these parts is the buildout part itself, which controls the options for the buildout process.

An absolute minimum buildout part looks something like this:

[buildout]
parts = noseinstall

While this is not a complete buildout.cfg, it is the minimum that is required in the buildout part itself. All is is doing is listing the other parts that buildout will use to actually do something, in this case, it is looking for a single part named noseinstall. As this part doesn’t exist yet, it won’t actually work. So, lets add the part, and in the next section, see what it does:

[buildout]
parts = noseinstall

[noseinstall]
recipe = zc.recipe.egg
eggs = Nose

An aside about bootstrap.py

We now have a config file that we’re reasonably sure will do something, if we’re really lucky, it’ll do something that we actually want it to do. But how do we run it? We will need buildout itself, but we don’t have that yet. At this point, there are two ways to proceed.

  1. sudo apt-get install python-zc.buildout
  2. wget http://python-distribute.org/bootstrap.py && python bootstrap.py

For various reasons, unless you need a very specific version of buildout, it is best to use the bootstrap.py file. This is a simple file that contains enough of buildout to install buildout itself inside your environment. As it’s cleanly installed, it can be upgraded, version pinned and generally used in the same manner as in a virtualenv style build. If you system-install buildout, you will not be able to easily upgrade the buildout instance, and may run into version conflicts if a project specifies a version newer than the one you have. Both approaches have their advantages, I prefer the second as it is slightly more self contained. Mixing the approaches (using bootstrap.py with a system-install is possible, but can expose some bugs in the buildout install procedure).

The rest of this document is going to assume that you have used bootstrap.py to install buildout.

Running some buildout

Now we have a method of running buildout, it’s time to do it in the directory where we left the buildout.cfg file created earlier:

$> bin/buildout

At this point, buildout will output something along the lines of:

Getting distribution for 'zc.recipe.egg'.
Got zc.recipe.egg 1.3.2.
Installing noseinstall.
Getting distribution for 'Nose'.
no previously-included directories found matching 'doc/.build'
Got nose 1.0.0.
Generated script '/home/tomwardill/tmp/buildoutwriteup/bin/nosetests-2.6'.
Generated script '/home/tomwardill/tmp/buildoutwriteup/bin/nosetests'.

Your output may not be exactly similar, but should contain broadly those lines.

The simple sample here is using the zc.recipe.egg recipe. This is probably the most common of all buildout recipes as it is the one that will do the heavy work of downloading an egg, analysing its setup.py for dependencies (and installing them if required), and then finally installing the egg into the buildout path for use. Recipes are just python eggs that contain code that buildout will run. The easiest way to think of this is that while a recipe is an egg, recipe contains instructions for the buildout process itself, and therefore will not be available to code at the end.

An analysis of the buildout output shows exactly what it has done. It has downloaded an egg for zc.recipe.egg and run the noseinstall part. Let’s take a closer look at that noseinstall part from before:

[noseinstall]
recipe = zc.recipe.egg
eggs = Nose

So, we can see why buildout has installed zc.recipe.egg, it is specified in the recipe option of this part, so buildout will download it, install it and then run it. We will take a closer look at the construction of a recipe in a later article, but for now, assume that buildout has executed a bunch of python code in the recipe, and we’ll carry on.
The python code in this case will look at the part that it is in, and look for an option called eggs. As we have specified this option, it will then look at this as a list, and install all the eggs that we have listed; in this case, just the one, the unittest test runner Nose.
As you can see from the bottom of the buildout output, the recipe has downloaded Nose, extracted it and created two files; bin/nosetests and bin/nosetests-2.6. Running one of those files like so:

$> bin/nosetests

----------------------------------------------------------------------
Ran 0 tests in 0.002s

OK
$>

We can see that this is nose, as we expect it to be. Two files have been generated because that is that the setup.py for Nose defines, a base nosetest executable, and one for the specifc python version that we have used (python 2.6 in my case). These are specified in the setup.py that makes up the nose egg, which will be covered in a later article.

Conclusion

We can install buildout into a development environment, and use a simple config file to install a python egg. The next article will cover a development example for using with django, and some niceties such as version pinning and running python in our buildouted environment.

Practical collective.xdv

Last year I published an overview of collective.xdv. A new and very promising way of theming Plone. Now that we’ve used it a few times I thought I’d document some the process we went though.

Firstly read the documentation:

It’s very good and shows you the best way to structure your theme.

Our setup:

  • Plone 3.3.5
  • CMF 2.1.2
  • PIL 1.1.6

Setting Up Collective.xdv Best Practice

The collective.xdv documentation has a very good description of the best way to setup  collective.xdv read it. I won’t go in to the full details as they are well documented but the basic points are:

  • Setup a Plone product
  • Use css registry to serve the css, instead of serving through XDV. This gives you the all the advantages of managing CSS through CSS registry (compression etc).
  • Create your html/css templates.

In our setup I turned off all stylesheets but kupustyles.css, authors.css, kupuplone.css, kupudrawerstyles.css when serving through XDV. I also created a new stylesheet to pull in what we needed from the admin theme. By adding back in on a case-by-case basis you can avoid CSS conflicts with your themes styles. Of course if you’re completely mad you could decide to restyle the admin interface (good luck with that).

Snags

There were a few snags we ran into. Here’s how we fixed them:

no styling with portal_factory

We ran in to an issue with our Plone setup and portal_factory. The transformation stopped working when you added content, leaving you seeing the default plone theme. Very confusing. Apparently it’s fixed in 3.3. but for whatever reason we had to use this monkey patch:

http://webteam.medsci.ox.ac.uk/integrators-developers/portalfactorypatch

Here are some great tips for using collective.xdv as well:

http://webteam.medsci.ox.ac.uk/integrators-developers/xdvtips

IE conditionals

By default I was removing comments from both the theme and content. (<rules />) . As the IE conditionals are wrapped in comment tags I had to add them back based on their order.

<append theme='/html/head' content='/html/head/comment()[3]' />
<append theme='/html/head' content='/html/head/comment()[4]' />

Lack of applyPrefix support for Plone 3

Plone 4 has a option to parse a stylesheet for relavtive URLs and apply an absolute prefix. You’ll have to do this manually with Plone 3.

Styling the Search Box

I’m not a fan of Plone search box so it was nice to be able to just pull out the action attribute to use:

<prepend theme='//*[@id="search-form"]' content="//*[@id='portal-searchbox']/form/@action" />

Using Conditionals to Select Templates

Using Plone’s default body classes or defining your own you can specify which template to use based on class name. This is a big win as you no longer have mess around with macros or the main template. For example:

<theme href="one-column.html" css:if-content="body.one-column" />
<theme href="two-column.html" css:if-content="body.two-column" />
<theme href="three-column.html" css:if-content="body.three-column" />

Pulling in Content

You could bring in content piece-by-piece – writing rules for each element. I found this was time consuming and difficult. In the end it made more sense to pull in the all content from each column. If you want to re-work the HTML of a standard portlet it’s better to use inline XSL (more on that in the next article)

<replace content='//*[@id="portal-column-content"]/div/*' theme='//*[@id="content"]' />

Conclusion

From my perspective:

  • It gives control of the templating to the front-end dev / designer.
  • It allows working in parallel with Plone devs rather then them waiting for templates to be completed.
  • Using collective.xdv dramatically sped up the time we had a skinned site we could show to the client.
  • We cut the time we spent doing donkey work and increased the time we could spend on worthwhile development.

I’ll leave it to the Senior Developer to give the pros from an experienced Plone developer:

  • No mangling of reference HTML templates as they are implemented in TAL by Plone Devs
  • improved team workflow as HTML/CSS Designer/Dev can work in parallel with Plone Devs
  • lighter workload for Plone Devs as there is much less HTML to implement in TAL
  • reduced need to explicitly remove unwanted elements from TAL templates – they can mostly be ignored by xdv
  • flexibility in layout between pages – i.e. one-column, two-column, three-column layouts don’t require a mass of macros and modification of main_template to achieve. This can instead be pushed up to the XDV layer.

Jonathan Stark – “Building mobile apps with HTML, CSS and JavaScript” workshop

Jonathan began with an overview of the current mobile landscape. In a nutshell it’s a minefield of platform fragmentation and device fragmentation.

“‘It’s brutal to develop natively across platforms”

Jonathan Stark
So what to do? Well, we then looked at three approaches; native, web and hybrid. All have their advantages but hybrid looked the way to go for most mobile use cases. In Jonathan’s words the advantages are:
  • Web files bundled in native executable
  • Access to device APIs
  • Sell in native app stores
  • Not perfect, but pretty damn good

Next we looked at some useful guidelines for the mobile use case. A nice summation would be “Do one thing and do it well”.

CSS3

We looked at various CSS3 features; advanced styling, transforms, transitions, animations etc. There’s a wealth of information out there on these so I won’t go into too much detail. Here are some interesting pointers:
  • android requires text-shadow to have a blur value.
  • box-shadow has performance issues, particularly on Blackberry (consider using media-query to turn off for Blackberry).
  • border-image is particularly interesting, as it does away with sliding doors css.
  • animations are hardware accelerated on iOS.
  • be careful of z-index on transforms. By default on it stacks above static positioned elements.
  • box-reflect works with form elements.
  • box-shadow takes up space like borders.
  • box-reflect works on <VIDEO> tag.
  • Talk of hardware acceleration for animation in the next version of Android  (Ice Cream Sandwich).
  • using lots of CSS3 gradients, animations, box-shadow will cause performance issues, tread careful and test often.
I liked the look of CSS animations. I have some reservations of how manageable they will become on medium to large projects. Having written code based animation sequences in Flash I know from experience it can get pretty complicated managing dependances and interactions. Using LESS, or SASS looks like an absolute must for using CSS animation on anything but the smallest projects.

HTML5

Next up was looking at some of the useful offline features of HTML5. Namely; web storage, web SQL database and application cache. My personal favourite was web storage – it looked easy to use and very useful for state maintenance. Application cache also looked great but I can see huge problems debugging, even Jonathan struggled to demo it properly.

Phonegap, jQTouch and Sencha Touch

Here we looked at the various frameworks for building hybrid apps. Again there is a ton of info out there. Jonathan was obviously keen on jQTouch but I actually think Phonegap looked the more promising platform for development. It seemed to have a good balance between ease of use, learning curve and features. Sencha Touch looked really powerful, whilst  jQTouch looked lightweight and quick to build with.

Debugging

In the final part of the day Jonathan showed some great debugging tools. If you’ve ever had to debug mobile CSS you know what a pain it is to debug on the device itself.  The stand out star for me was weinre, a really fantastic tool for debugging mobile apps.  The basic premise is it gives you access to the webkit inspector on your laptop which in turn is linked to the phone browser. It allows you to see exactly what is happening on the phone and allows live editing as well. Pretty amazing really.

Conclusion

Jonathan’s workshop has re-enforced my opinion that writing mobile apps for all formats is best done using HTML5 / CSS and JS.  The key benefits are:
  • write once*
  • The skills are already available and less costly than hiring native specific developers
  • one central code base
* They’ll always be some platform specific code needed.
All the slides and downloads from Jonathan’s talk are available here: http://jonathanstark.com/talks-wdx4.php

The Third Manifesto Implementers’ Workshop

Earlier this month I went to the Third Manifesto Implementers’ Workshop at Northumbria University in Newcastle. A group of us discussed recent developments in implementing the relational data model.

The relational data model was proposed by E. F. Codd in 1969 in response to the complex, hierarchical, data storage solutions of the time which required programs to be written and compiled for each database query. It was a powerful abstraction, but unfortunately SQL and its implementations missed out on important features, and broke it in fundamental ways. In response to this problem, and the industry’s approach towards object-databases, Chris Date and Hugh Darwen wrote “The Third Manifesto” (TTM) to put forward their ideas on how future database systems should work. I urge you to read their books (even if you’re not interested in the subject) – the language is amazing: precise, concise, comprehensive and easy to read – other technical authors don’t come close.

The relational model treats data as sets of logical propositions and allows them to be queried, manipulated and constrained declaratively. It abstracts away from physical storage and access issues which is why it will still be used a hundred years from now (and why NoSQL discussions like these http://wiki.apache.org/couchdb/EntityRelationship http://www.cmlenz.net/archives/2007/10/couchdb-joins are retrograde). If you’re writing loops to query your data, or having to navigate prescribed connection paths, then your abstractions are feeble and limited.

At the workshop, I talked about my project, Dee, which implements the relational ideas from TTM in Python. You can see my slides here (or here if you have an older browser).

Erwin Smout gave a couple of talks about implementing transition constraints and dispensing with data definition language.

David Livingstone walked us through the RAQUEL architecture – a layered approach along the lines of the OSI network model.

Hugh Darwen discussed the features and implementation of IBM’s Business System 12, one of the first ever relational database systems which had some surprisingly dynamic features, including key inferencing, so that view definitions could keep tabs on their underlying constraints.

Chris Date took us through his latest thoughts on how to update relational views in a generic way. The aim is for database users to be able to treat views and base tables in the same way, for both reading and writing. Lots to think about here, and my to-do list for Dee has grown another section.

Adrian Hudnott discussed a couple of research projects around optimising multiple relational assignments and tracking the source of updates so that transition constraints could be more effective.

Renaud de Landtsheer gave an insight into the work he’s been doing implementing first-order-logic constraints within Oracle databases.

I sat next to Toon Koppelaars (whose name went down well with the Geordies) and then I realised I had his book (Applied Mathematics for Database Professionals) waiting to be read on my desk at work, thanks to the eclectic Isotoma library (and Wes).

It was a packed couple of days with plenty of food for thought. Thank you to David Livingstone and Safwat Mansi for organising and hosting such an enjoyable and interesting event.

UX Lx: Day 1

Thoroughly enjoyed UX Lisbon, organised by ideas e imagens. I don’t think I’ve ever been at a conference where I didn’t regret a single session.

These notes are not necessarily complete summaries of the sessions; they focus on the aspects I found most noteworthy.

1. Prototyping with HTML5 – Todd Zaki Warfel

@zakiwarfel | Slides

Probably the workshop in which I learned least, as Zaki Warfel effectively described exactly the way I currently prototype, and for the same reasons. But was good to have this reinforced.

With all prototypes, including wireframes, set expectations with the client. Choose the level of visual and functional fidelity the project requires. I tend to choose HTML prototypes when I require a high level of functional fidelity, when I don’t know whether an interface works until I can use it.

HTML prototypes have many advantages:

  • closest to final delivery environment
  • browsers & text editors are ubiquitous – fewer things that can go wrong
  • my favourite: producing production-quality HTML for the prototype can shave 30-40% off production time. (This divided attendees as some considered it too high a hurdle. But aligns with my belief that well-structured semantic HTML is akin to IA and the IA is best placed to write it)

HTML + JavaScript is also more capable than any GUI prototyping tool like Axure. But remember, it is still more constrained than a sketch or a wireframe. When you are sketching you can invent things; you’re not constrained by the familiar toolset.

Zaki Warfel stressed the advantages of HTML5, but I came away still unconvinced of any real benefits for prototypes. Only new HTML5 form attributes (date, email, tel, url, placeholder, etc) are a no-brainer if you want to improve usability in iPhones.

In the tricky choice between HTML5‘s <section> and <article> elements, he suggested thinking of using <article> for things that you can imagine going into an RSS feed. But just “choose one road and don’t look back”. (I also find this very confusing, and agree with Jeremy Keith that article and section should be merged into a single element.)

To make HTML5 compatible with older browsers, Zaki Warfel recommended html5shiv instead of modernizr, to keep your HTML cleaner.

CSS3 he didn’t have to sell to me. I already know it saves major time not doing sliding doors or image-based corners, shadows or gradients. But note you have to go back later to fix for IE (or convince the client to accept visual compromises in IE, as Zaki Warfel, echoing Zeldman, advised.)

CSS3 selectors are also a huge time saver in prototyping, but (in my opinion) they will usually require extensive refactoring for IE later.

Zaki Warfel recommended using include files for repeated elements in your HTML, using PHP (already installed in OS X). At first I thought this was unnecessary, but then remembered that in later stages of prototyping I tend to do a lot of difficult find/replace operations across files. So I definitely want to try this in future.

Finally, Zaki Warfel demonstrated the magic of jQuery. I’ve only recently started learning it, and I think every interaction designer should. I use it for all show/hide, open/close, expand/contract, highlight, popup, etc. behaviours in HTML prototypes, and it is a pleasure being able to do this as easily as CSS.

2. Skeuomorphs: The Good, The Bad, and the Silly – Andrew Watterson

@andrewwatterson | Also summarised on Johnny Holland

Skeuomorphs refer to physical metaphors to ease transition to new technologies (mostly touch interfaces in this talk). Explicitly recommended in Apple’s HIG. Criticised by Adam Greenfield “patronizing crutches” and (implicitly) by Jakob Nielsen “Users don’t know where they can click.”

Skeuomorphs can be good:

  • They deemphasise technology in favour of utility (cf. Apple’s ads which suggest familiar usage contexts for new technology, vs. Android’s science-fiction approach.)
  • They can reassure and please, bridge gaps, ease transition. He paraphrases Don Norman’s adage that users have 2 needs, (1) to get something done, and (2) to smile.

But skeuomorphs can cause lots of problems (interestingly, most of the worst culprits are Apple apps)

  • They can mislead with inappropriate metaphors (Apple Calendar and Contacts on iPad have scrollable areas, concealed by the fact that they look like paper; Contacts has a “Groups” icon that looks like a bookmark but is actually a button; the Apple Sound Recorder shows a realistic microphone that conceals the location of the device’s actual microphone)
  • They can cause you to skip opportunities to innovate, preventing you from improving on existing tech (e.g. Apple’s Compass app, which has no more utility than a mechanical one, compared with AR Compass which uses augmented reality to give a far more useful view.)

Watterson didn’t even talk about Apple’s egregious Game Center, which boasts the worst skeuomorphic interface I’ve ever seen.

3. Serendipity: Beyond Recommendation – Pedro Fernandes

@betasolo

In Fernandes’ talk, serendipity refers to fortunate discoveries while looking for something unrelated.

The essence of his argument was that existing recommendation engines frequently resemble echo chambers, limited by their algorithms or the metadata they depend on. If item A always recommends items B and C, and items B and C recommend A/C and A/B respectively, then you’ll never discover items D or Z.

In e-commerce this can undermine “long tail” sales, but in social networking it can result in “cultural tribalism”, online spaces that just reinforce your preferred world view.

Fernandes showed two apps that try to inject more serendipity into the user journey: mowid.com for browsing films, and Serendipicity, a mobile app for tourists. Mowid relies heavily on tag-based browsing, drawn from a rich vocabulary of tags, and Serendipicity lets you explore places primarily via photos (inherently more open to interpretation) taken in the vicinity by other users.

While I agree about the central issue, I was a bit skeptical about both apps.

Designing for Touch – Josh Clark

@globalmoxie

Definitely one of the conference highlights (all the way from the Johnny Cash opening music). Clark proved himself a consummate educator, fitting in so much detailed, actionable information and examples that the fact it was a presentation rather than a workshop didn’t matter.

When designing for mobile devices, forget pixels: think of it as designing a physical device. It’s more like industrial design.

Many design rules follow directly from physical limitations. On small touchscreen devices, follow the rule of controls at bottom, content at top – so that content is not obscured by “meat sticks”. For the same reason, two rows of buttons at the bottom is also not ideal, something which Android is unfortunately stuck with.

Josh Clark demonstrating optimal thumb range on an iPhone

Optimal thumb range

Bottom-left is the top spot – for right-handed users at least, which is what you should optimise for. But consider offering a setting for the 10-15% of users who are left-handed.

Because it’s difficult to fix buttons at the bottom of the screen using JavaScript, he suggested a design pattern for web apps where the main menu is always presented at the bottom of the content, but with a “menu” anchor link at the top of the page.

For the iPad / tablets, rules are different. There are many ways to use them, and many use contexts, with no clear preference for portrait or landscape. It is therefore harder to predict hand positioning. Top-of-screen controls are better, to avoid controls at the bottom sinking into your belly :)

The Instapaper app, with its controls in the top two corners, was praised, and The Daily, where using the page scrubber at the top obscures the thumbnail images just below it, was criticised.

What is the optimal size for a button? The answer turns out to be 44 pixels high (29 wide). This happens to be (no coincidence) the height of the iPhone menu bars, buttons, and keys on the virtual keyboard.

But what is a pixel? Due to differing pixel densities, we should stop thinking of device pixels, and think rather of a physical measure on screen. This is called

  • iOS: points
  • CSS: pixel (which is device-independent)
  • Android: density-independent pixel (dp)

So if you continue to use the px unit in CSS, it automatically does the right thing. However, you need to start producing both a normal and high-density version of all images, e.g. image.png and image@2x.png
This is easy enough to apply using the IMG element, but for CSS background images you have to specify the higher-density image thus

@media -webkit-min-device-pixel-ratio:2 {
		.class { background-image: image@2x.png;
		         background-size: 50%;
		}
	}

It’s not just about size, though, but also spacing: the closer together, the bigger buttons need to be. A useful tip is to invisibly increase the hit area for small elements (Remember the Milk does this for checkboxes.)

You can also use animation to draw attention to or explain buttons or other elements that might otherwise be missed or misunderstood. For example, slide in a menu that can be swiped, or pop up the primary action button (Gowalla does this.)

Single-screen interfaces (utility apps) strengthen the illusion that it is a physical device. They should pass the “glance test” – right information hierarchy at arms’ length. Two examples: Tea Round and Umbrella Today. Strip out everything not needed when rushed and distracted. Clarity trumps Density (of features).

But mobile devices are not always used when rushed and distracted! People don’t want “dumbed down”. People want uncomplicated. Full-featured, just lighter interface. (Story of initial Facebook app which was billed as a companion app, which users rejected because they expected to be able to do everything they could do on the website.) Some mobile apps need to do more than the desktop versions.

Don’t fear extra taps! Web has made us squeamish about number of clicks. Latency not an issue in a native/cached app. Tap quality* trumps tap quantity (*unconfusing)

Similarly, don’t fear scrolling. E.g. USA Today used an accordion interface to avoid a long list of headlines scrolling, but not all users understood it. For long lists of content, scrolling is still better.

Changing orientation: think of it not just as a change in layout, but also a change in mindset. Landscape mode can also be seen as “focused mode”. But beware of depending on it, as it is hard to discover. (Personally, I believe orientation in iOS is flawed, invoked overwhelmingly by accident. I think Ben Summers has the right idea for how it should work.)

Affordances: gestures are the keyboard shortcuts of touch interfaces, they can be hard for users to discover. Shortcuts need backup plans – never rely on all users discovering them.

When you see your app being used, look for unsuccessful gesture attempts and repetitive interactions, and pave the cowpaths. (Have Apple never seen users try to swipe the Calendar app to change day?)

Multi-touch (on phones, not iPad) and the shake gesture generally considered bad.

Don’t get carried away with an impressive, glitzy interface. Showcase the content, not the form. Don’t underestimate the power of the humdrum and familiar – see NY Times and Flipboard apps. Similarly, skeuomorphs, while still window-dressing, can enhance a design; familiarity and intimacy invite touch. But if you’re aping a physical object, choose the right metaphor.

But try to avoid buttons altogether. Buttons are a hack. Look at how a toddler uses an iPhone/iPad – they try to interact directly with the content. Wherever possible, make the content the interface. (Example: see how Twitter for iPad eliminated the Back button.)

Clark signed off on an optimistic note, and an encouragement to experiment: new platforms don’t appear very often. This is the coolest job in the world.

Improving Harvest using seductive interactions

One of my favourite talks at UX London 2010 was Stephen Anderson’s Seductive Interactions – using basic psychological principles to bridge the gap between business goals and users’ behavioural goals. Usability alone, he argues, often merely decreases friction. Using psychology can also increase users’ motivation.

At UXLx, the recent UX conference in Lisbon, I attended Anderson’s workshop on the same topic. By luck, our exercises focused on improving one of my great bugbears, time tracking. And not just any time tracker: Harvest, which we use at Isotoma.

Don’t get me wrong: Harvest is by a long stretch the best time tracker I’ve used. But nevertheless, it remains an activity I hate and avoid, and routinely fill in my time weeks late.

Applying typical utilitarian thinking would result in improvements like –

  • Increasing the font sizes in Day view
  • Week view should highlight active rows and columns, and keep column headers and footers visible when scrolling

– helpful, but unlikely to change my attitude or behaviour.

In the workshop we used Anderson’s “Mental Notes”, each of which describe a psychological principle designers can take advantage of, and used them to generate ideas to improve Harvest. Here are the relevant notes, and what we came up with:

Trigger / Recognition Over Recall / Feedback Loops

Feedback Loops; Trigger; Recognition Over RecallThe application should listen and learn from your actions. Just like a good online supermarket will make it easy to re-order the things you frequently buy, Harvest should always automatically show the jobs you worked on previously.

It should also trigger you to take action. At the end of each day, it should pop up a form: “Did you work on these jobs today?” containing only the jobs you had put time on the day before.

A frequent problem is that a job might not exist in the system yet, because a project manager hasn’t created it, or hasn’t added you to it.

Firstly, in the Day and Week views, you should see all the jobs you’ve been added to, not only the ones you’ve chosen to show. (The dropdown menu to add jobs is a terrible interface.)

Secondly, you should be able to create “placeholder” jobs and assign times to them, which you can later reconcile with an actual job. Placeholder times will be used in your own totals, but not in project managers’ reports.

Harvest popup window

Status / Achievements / Competition / Appropriate Challenges

Competition; Status; Achievements; Appropriate ChallengesAnderson showed the example of Target’s supermarket checkout interface, which uses a game-like system to encourage cashiers to work more quickly (photo).

Similar principles can be used to encourage workers to fill their timesheets in sooner. Harvest should track how long, on average, it takes you to fill in your timesheet (same day, 1 day late, 7 days late, etc.) This should be shown to you as a chart over time, so you can see your average and your trend.

Then it should also show you the same timeliness charts for your colleagues, and where you rank (without names, as no-one would appreciate the feeling of being singled out.) This will act as a subtle but powerful spur to improve your timeliness. The company can also offer rewards for the most timely, or the greatest improvement.

Set Completion

Set CompletionHighlight it when a day totals up to 7.5h or more (or whatever your daily goal is). Completed days should stand out clearly on the Week view and reports. And how about a “Well done!” whenever you complete a day?

Another example: 750 Words encourages writers to stick to a writing schedule by writing only 750 words a day, checking off a box when they’ve done so. (See screenshot here.)

Delighters / Surprise / Humor Effect / Visual Imagery / Self Expression

Delighters; Surprise; Humor Effect; Visual Imagery; Self ExpressionIn a different UXLx talk, Andrew Watterson paraphrased usability pope Don Norman’s adage that users have 2 basic needs: (1) to get something done, and (2) to smile. Adding a touch of humour to an application can go a long way towards making it more pleasurable to use and memorable.

This is why our own Forkd.com uses tomatoes instead of asterisks on forms, amongst many other touches, and I’ve lost count of the number of people who have commented on that.

Forkd.com registration form (excerpt)Some ideas that cropped up here were for the application to offer occasional comments and quips, e.g. advising users to mind their posture, or go for a walk now and then, or pep talk suggesting how hard-working they are. Ex-colleague Jonathan Baker-Bates designed a custom-built time tracker that allowed employees to add their own quips, which rapidly became an impromptu means of humourous communication and teasing within the company (like the #isotoma IRC channel’s notorious topics, I imagine.)

Another ex-colleague, Karl Sabino and I came up with the idea of adding “themes” to the time entry interface. Instead of the current businesslike grey-and-orange, you could fill in your times on something resembling a ticking bomb, or a Superbowl scoreboard, or an Indian restaurant menu. To our surprise, it turned out that Dominos Pizza already had this idea on their patented Pizza Tracker:

Screenshot of Dominos Pizza Tracker

Many customers found this amusing enough to post videos of it on YouTube.

So there you have it: a fresh approach to a design problem that yielded many results I doubt I’d have come up with otherwise. When I tweeted about the workshop, Harvest immediately responded with interest. I hope you like the results! (And my employers will thank you if you can somehow improve my terrible timesheet habits.)

I’d like to acknowledge the input of Karl Sabino and Jonathan Baker-Bates in several of the ideas above.

Setting the modification date of an Archetype object in Plone.

Modification dates of Archetypes objects are set automatically to the current
date and time when an object is reindexed. Usually this is useful behaviour but if
,for example, you’re importing content from elsewhere into a Plone site and you’d
like to set the modification date to be some time in the past then it’s a problem.

Heres a method that sets the standard dates of an Archetypes content type ‘obj’
to DateTime ‘dt’.

def setObjDate(obj, dt):
    """Prevent update of modification date 
       during reindexing"""
    obj.setCreationDate(dt)
    obj.setEffectiveDate(dt)
    obj.setModificationDate(dt)
    od = obj.__dict__
    od['notifyModified'] = lambda *args: None
    obj.reindexObject()
    del od['notifyModified']

The trick, based on migration code found in the plone.app.blob package, is
to mask the ‘notifyModified’ class method with a no-op lambda on our
instance, call the reindex, then drop the mask so that the class method is visible
once again.

I’ve used this successfully for importing data into a Plone 4 instance but it should work at least as far back as Plone 2.5.

Tamper-protection for Bank Transactions

If you need to send electronic transactions to Swedish banks, you’ll be required to add anti-tampering seals to the files. The banks recommend you use a third-party system to create the HMAC SHA256-128 seals, but that could involve a fair amount of expensive server software and maintenance contracts (some linked to the number of people who work in your company).

Instead, you can do it yourself in Python like this:

import hmac
import hashlib
import string

NORMALISE = string.maketrans(
    '\xC9\xC4\xD6\xC5\xDC\xE9\xE4\xF6\xE5\xFC' + ''.join(
    [chr(x) for x in range(0,32)]) + ''.join(
    [chr(x) for x in range(127,256)
     if x not in (201,196,214,197,220,233,
                  228,246,229,252)]),
    '\x40\x5B\x5C\x5D\x5E\x60\x7B\x7C\x7D\x7E' + ''.join(
    [chr(195) for x in range(0,32)]) + ''.join(
    [chr(195) for x in range(127,256)
     if x not in (201,196,214,197,220,233,
                  228,246,229,252)]))

def hex_to_bytes(hexs):
    """Convert string of hex into bytes"""
    return ''.join(['%s' % chr(int(hexs[i:i+2], 16))
                    for i in range(0, len(hexs), 2)])

def get_signature(contents, key):
    """Calculate the HMAC SHA256-128 signature

       contents - an iso-8859-1 (latin-1) encoded string
       key - a string of hex characters

       Returns a 32 char string of hex characters (128 bits)
    """
    key = hex_to_bytes(key)

    #Normalise the contents
    contents = contents.translate(NORMALISE, '\r\n')

    dig = hmac.new(key, msg=contents,
                   digestmod=hashlib.sha256).digest()
    return ''.join(['%02X' % ord(x) for x in dig[:16]])

And then to calculate the signature for a file:

>>> print get_signature(open('bankfile.dat').read(),
...                     '1234567890abcdef1234567890abcdef')
25122AE4179BD51DC87AD6EA08D16D45

Seeking a Head of QA

Isotoma is looking for someone with solid testing and management experience to come and help us shape and build our QA function. The role will be to take on the formalisation of our quality control systems, from traditional manual testing to automated CI, build and deployment. We’re looking for someone who can run the testing function whilst also getting involved with all 3 teams within the company (dev, devops and ops), helping them improve their processes and approaches upstream.
Read more about it on the main site (Head of Quality Assurance vacancy) and please get in touch.

PloneConf2010 – and there’s more

I’m here at PloneConf2010 too, and I’ve been to mostly different talks to Mitch, so here’s my write up too.  It’s taken a bit of time to find the time to write this up properly!

Calvin Hendryx-Parker: Enterprise Search in Plone using Solr

We’ve been using Solr for a few years here at Isotoma, but so far we’ve not integrated it with Plone.  Plone’s built-in Catalog is actually pretty good, however one thing it doesn’t do fantastically well is full-text search.  It is passable in English, but has very limited stemming support – which makes it terrible in other languages.

Calvin presented their experience of using Solr with Plone. They developed their own software to integrate the Plone catalog with Solr, instead of using collective.solr, which up till then was the canonical way of connecting them. Their new product alm.solrindex sounds significantly better than collective.solr.  Based on what I’ve heard here, you should definitely use alm.solrindex.

To summarise how this all hangs together, you need an instance of Solr installed somewhere that you can use.  You can deploy a solr specifically for each site, in which case you can deploy it through buildout.  Solr is Java, and runs inside various Java application servers.

You can also run a single Solr server for multiple Plone sites – in which case you partition the Solr database.

You then configure Solr, telling it how to index and parse the fields in your content. No configuration of this is required within Plone.  In particular you configure the indexes in Solr not in Plone.

Then install alm.solrindex in your plone site and delete all the indexes that you wish to use with Solr. alm.solrindex will create new indexes by inspecting Solr.

Then reindex your site, and you’re done!  It supports a lot of more complex use cases, but in this basic case you get top-end full text indexing at quite low cost.

Dylan Jay, PretaWeb: FunnelWeb

Funnelweb sounds invaluable if you want to convert an existing non-Plone site into a Plone site, with the minimum effort.

Funnelweb is a tool based on transmogrifier. Transmogrifier provides a “pipeline” concept for transforming content. Pipeline stages can be inserted into a pipeline, and these stages then have the ability to change the content in various ways.

Dylan wrote funnelweb to use transmogrifier and provide a harness for running it in a managed way over existing websites.  The goal is to create a new Plone site, using the content from existing websites.

Funnelweb uploads remotely to Plone over XML-RPC, which means none of transmogrifier needs to be installed in a Plone site, which is a significant advantage.  It is designed to be deployed using buildout, so a script will be provided in your build that executes the import.

A bunch of pipeline steps are provided to simplify the process of importing entire sites.  In particular funnelweb has a clustering algorithm that attempts to identify which parts of pages are content and which are templates.  This can be configured by providing xpath expressions to identify page sections, and then extract content from them for specific content fields.

It supports the concept of ordering and sorts, so that Ordered Folder types are created correctly.  It supports transmogrify.siteanalyser.attach to put attachments closer to pages and transmogrify.siteanalyser.defaultpage to detect index pages in collections and to make them folder indexes in the created sites.

Finally it supports relinking, so that pages get sane urls and all links to those pages are correctly referenced.

Richard Newbury: The State of Plone Caching

The existing caching solution for Plone 3 is CacheFu, which is now pretty long in the tooth.  I can remember being introduced to CacheFu by Geoff Davis at the Archipelago Sprint in 2006, where it was a huge improvement on the (virtually non-existent) support for HTTP caching in Plone.

It’s now looking pretty long in the tooth, and contains a bunch of design decisions that have proved problematic over time, particularly the heavy use of monkeypatching.

This talk was about the new canonical caching package for Plone, plone.app.caching. It was built by Richard Newbury, based on an architecture from the inimitable Martin Aspeli.

This package is already being used on high-volume sites with good results, and from what I saw here the architecture looks excellent.  It should be easy to configure for the general cases and allows sane extension of software to provide special-purpose caching configuration (which is quite a common requirement).

It provides a basic knob to control caching, where you can select strong, moderate or weak caching.

It can provide support for the two biggest issues in cache engineering: composite views (where a page contains content from multiple sources with different potential caching strategies) and split views (where one page can be seen by varying user groups who cannot be identified entirely from a tuple of URL and headers listed in Vary).

It provides support for nginx, apache, squid and varnish.  Richard recommends you do not use buildout recipes for Varnish, but I think our recipe isotoma.recipe.varnish would be OK, because it is sufficiently configuration.  We have yet to review the default config with plone.app.caching though.

Richard recommended some tools as well:

  • funkload for load testing
  • browsermob for real browsers
  • HttpFox instead of LiveHttpHeaders
  • Firebug, natch
  • remember that hitting refresh and shift-refresh force caches to refresh.  Do not use them while testing!

Jens Klein: Plone is so semantic, isn’t it?

Jens introduced a project he’s been working on called Interactive Knowledge Stack (IKS), funded by the EU.  This project is to provide an open source Java component for Content Management Systems in Europe to help the adoption of Semantic concepts online.  The tool they have produced is called FISE. The name is pronounced like an aussie would say “phase” ;)

FISE provides a RESTful interface to allow a CMS to associate semantic statements with content.  This allows us to say, for example that item X is in Paris, and in addition we can state that Paris is in France.  We can now query for “content in France” and it will know that this content is in France.

The provide a generic Python interface to FISE which is usable from within Plone.  In addition it provides a special index type that integrates with the Plone Catalog to allow for updating the FISE triple store with the information found in the content.  It can provide triples based on hierarchical relationships found in the plone database (page X is-a-child-of folder Y).

Jens would like someone to integrate the Aloha editor into Plone, which would allow much easier control by editors of semantic statements made about the content they are editing.