The Third Manifesto Implementers’ Workshop

Earlier this month I went to the Third Manifesto Implementers’ Workshop at Northumbria University in Newcastle. A group of us discussed recent developments in implementing the relational data model.

The relational data model was proposed by E. F. Codd in 1969 in response to the complex, hierarchical, data storage solutions of the time which required programs to be written and compiled for each database query. It was a powerful abstraction, but unfortunately SQL and its implementations missed out on important features, and broke it in fundamental ways. In response to this problem, and the industry’s approach towards object-databases, Chris Date and Hugh Darwen wrote “The Third Manifesto” (TTM) to put forward their ideas on how future database systems should work. I urge you to read their books (even if you’re not interested in the subject) – the language is amazing: precise, concise, comprehensive and easy to read – other technical authors don’t come close.

The relational model treats data as sets of logical propositions and allows them to be queried, manipulated and constrained declaratively. It abstracts away from physical storage and access issues which is why it will still be used a hundred years from now (and why NoSQL discussions like these http://wiki.apache.org/couchdb/EntityRelationship http://www.cmlenz.net/archives/2007/10/couchdb-joins are retrograde). If you’re writing loops to query your data, or having to navigate prescribed connection paths, then your abstractions are feeble and limited.

At the workshop, I talked about my project, Dee, which implements the relational ideas from TTM in Python. You can see my slides here (or here if you have an older browser).

Erwin Smout gave a couple of talks about implementing transition constraints and dispensing with data definition language.

David Livingstone walked us through the RAQUEL architecture – a layered approach along the lines of the OSI network model.

Hugh Darwen discussed the features and implementation of IBM’s Business System 12, one of the first ever relational database systems which had some surprisingly dynamic features, including key inferencing, so that view definitions could keep tabs on their underlying constraints.

Chris Date took us through his latest thoughts on how to update relational views in a generic way. The aim is for database users to be able to treat views and base tables in the same way, for both reading and writing. Lots to think about here, and my to-do list for Dee has grown another section.

Adrian Hudnott discussed a couple of research projects around optimising multiple relational assignments and tracking the source of updates so that transition constraints could be more effective.

Renaud de Landtsheer gave an insight into the work he’s been doing implementing first-order-logic constraints within Oracle databases.

I sat next to Toon Koppelaars (whose name went down well with the Geordies) and then I realised I had his book (Applied Mathematics for Database Professionals) waiting to be read on my desk at work, thanks to the eclectic Isotoma library (and Wes).

It was a packed couple of days with plenty of food for thought. Thank you to David Livingstone and Safwat Mansi for organising and hosting such an enjoyable and interesting event.

UX Lx: Day 1

Thoroughly enjoyed UX Lisbon, organised by ideas e imagens. I don’t think I’ve ever been at a conference where I didn’t regret a single session.

These notes are not necessarily complete summaries of the sessions; they focus on the aspects I found most noteworthy.

1. Prototyping with HTML5 – Todd Zaki Warfel

@zakiwarfel | Slides

Probably the workshop in which I learned least, as Zaki Warfel effectively described exactly the way I currently prototype, and for the same reasons. But was good to have this reinforced.

With all prototypes, including wireframes, set expectations with the client. Choose the level of visual and functional fidelity the project requires. I tend to choose HTML prototypes when I require a high level of functional fidelity, when I don’t know whether an interface works until I can use it.

HTML prototypes have many advantages:

  • closest to final delivery environment
  • browsers & text editors are ubiquitous – fewer things that can go wrong
  • my favourite: producing production-quality HTML for the prototype can shave 30-40% off production time. (This divided attendees as some considered it too high a hurdle. But aligns with my belief that well-structured semantic HTML is akin to IA and the IA is best placed to write it)

HTML + JavaScript is also more capable than any GUI prototyping tool like Axure. But remember, it is still more constrained than a sketch or a wireframe. When you are sketching you can invent things; you’re not constrained by the familiar toolset.

Zaki Warfel stressed the advantages of HTML5, but I came away still unconvinced of any real benefits for prototypes. Only new HTML5 form attributes (date, email, tel, url, placeholder, etc) are a no-brainer if you want to improve usability in iPhones.

In the tricky choice between HTML5‘s <section> and <article> elements, he suggested thinking of using <article> for things that you can imagine going into an RSS feed. But just “choose one road and don’t look back”. (I also find this very confusing, and agree with Jeremy Keith that article and section should be merged into a single element.)

To make HTML5 compatible with older browsers, Zaki Warfel recommended html5shiv instead of modernizr, to keep your HTML cleaner.

CSS3 he didn’t have to sell to me. I already know it saves major time not doing sliding doors or image-based corners, shadows or gradients. But note you have to go back later to fix for IE (or convince the client to accept visual compromises in IE, as Zaki Warfel, echoing Zeldman, advised.)

CSS3 selectors are also a huge time saver in prototyping, but (in my opinion) they will usually require extensive refactoring for IE later.

Zaki Warfel recommended using include files for repeated elements in your HTML, using PHP (already installed in OS X). At first I thought this was unnecessary, but then remembered that in later stages of prototyping I tend to do a lot of difficult find/replace operations across files. So I definitely want to try this in future.

Finally, Zaki Warfel demonstrated the magic of jQuery. I’ve only recently started learning it, and I think every interaction designer should. I use it for all show/hide, open/close, expand/contract, highlight, popup, etc. behaviours in HTML prototypes, and it is a pleasure being able to do this as easily as CSS.

2. Skeuomorphs: The Good, The Bad, and the Silly – Andrew Watterson

@andrewwatterson | Also summarised on Johnny Holland

Skeuomorphs refer to physical metaphors to ease transition to new technologies (mostly touch interfaces in this talk). Explicitly recommended in Apple’s HIG. Criticised by Adam Greenfield “patronizing crutches” and (implicitly) by Jakob Nielsen “Users don’t know where they can click.”

Skeuomorphs can be good:

  • They deemphasise technology in favour of utility (cf. Apple’s ads which suggest familiar usage contexts for new technology, vs. Android’s science-fiction approach.)
  • They can reassure and please, bridge gaps, ease transition. He paraphrases Don Norman’s adage that users have 2 needs, (1) to get something done, and (2) to smile.

But skeuomorphs can cause lots of problems (interestingly, most of the worst culprits are Apple apps)

  • They can mislead with inappropriate metaphors (Apple Calendar and Contacts on iPad have scrollable areas, concealed by the fact that they look like paper; Contacts has a “Groups” icon that looks like a bookmark but is actually a button; the Apple Sound Recorder shows a realistic microphone that conceals the location of the device’s actual microphone)
  • They can cause you to skip opportunities to innovate, preventing you from improving on existing tech (e.g. Apple’s Compass app, which has no more utility than a mechanical one, compared with AR Compass which uses augmented reality to give a far more useful view.)

Watterson didn’t even talk about Apple’s egregious Game Center, which boasts the worst skeuomorphic interface I’ve ever seen.

3. Serendipity: Beyond Recommendation – Pedro Fernandes

@betasolo

In Fernandes’ talk, serendipity refers to fortunate discoveries while looking for something unrelated.

The essence of his argument was that existing recommendation engines frequently resemble echo chambers, limited by their algorithms or the metadata they depend on. If item A always recommends items B and C, and items B and C recommend A/C and A/B respectively, then you’ll never discover items D or Z.

In e-commerce this can undermine “long tail” sales, but in social networking it can result in “cultural tribalism”, online spaces that just reinforce your preferred world view.

Fernandes showed two apps that try to inject more serendipity into the user journey: mowid.com for browsing films, and Serendipicity, a mobile app for tourists. Mowid relies heavily on tag-based browsing, drawn from a rich vocabulary of tags, and Serendipicity lets you explore places primarily via photos (inherently more open to interpretation) taken in the vicinity by other users.

While I agree about the central issue, I was a bit skeptical about both apps.

Designing for Touch – Josh Clark

@globalmoxie

Definitely one of the conference highlights (all the way from the Johnny Cash opening music). Clark proved himself a consummate educator, fitting in so much detailed, actionable information and examples that the fact it was a presentation rather than a workshop didn’t matter.

When designing for mobile devices, forget pixels: think of it as designing a physical device. It’s more like industrial design.

Many design rules follow directly from physical limitations. On small touchscreen devices, follow the rule of controls at bottom, content at top – so that content is not obscured by “meat sticks”. For the same reason, two rows of buttons at the bottom is also not ideal, something which Android is unfortunately stuck with.

Josh Clark demonstrating optimal thumb range on an iPhone

Optimal thumb range

Bottom-left is the top spot – for right-handed users at least, which is what you should optimise for. But consider offering a setting for the 10-15% of users who are left-handed.

Because it’s difficult to fix buttons at the bottom of the screen using JavaScript, he suggested a design pattern for web apps where the main menu is always presented at the bottom of the content, but with a “menu” anchor link at the top of the page.

For the iPad / tablets, rules are different. There are many ways to use them, and many use contexts, with no clear preference for portrait or landscape. It is therefore harder to predict hand positioning. Top-of-screen controls are better, to avoid controls at the bottom sinking into your belly :)

The Instapaper app, with its controls in the top two corners, was praised, and The Daily, where using the page scrubber at the top obscures the thumbnail images just below it, was criticised.

What is the optimal size for a button? The answer turns out to be 44 pixels high (29 wide). This happens to be (no coincidence) the height of the iPhone menu bars, buttons, and keys on the virtual keyboard.

But what is a pixel? Due to differing pixel densities, we should stop thinking of device pixels, and think rather of a physical measure on screen. This is called

  • iOS: points
  • CSS: pixel (which is device-independent)
  • Android: density-independent pixel (dp)

So if you continue to use the px unit in CSS, it automatically does the right thing. However, you need to start producing both a normal and high-density version of all images, e.g. image.png and image@2x.png
This is easy enough to apply using the IMG element, but for CSS background images you have to specify the higher-density image thus

@media -webkit-min-device-pixel-ratio:2 {
		.class { background-image: image@2x.png;
		         background-size: 50%;
		}
	}

It’s not just about size, though, but also spacing: the closer together, the bigger buttons need to be. A useful tip is to invisibly increase the hit area for small elements (Remember the Milk does this for checkboxes.)

You can also use animation to draw attention to or explain buttons or other elements that might otherwise be missed or misunderstood. For example, slide in a menu that can be swiped, or pop up the primary action button (Gowalla does this.)

Single-screen interfaces (utility apps) strengthen the illusion that it is a physical device. They should pass the “glance test” – right information hierarchy at arms’ length. Two examples: Tea Round and Umbrella Today. Strip out everything not needed when rushed and distracted. Clarity trumps Density (of features).

But mobile devices are not always used when rushed and distracted! People don’t want “dumbed down”. People want uncomplicated. Full-featured, just lighter interface. (Story of initial Facebook app which was billed as a companion app, which users rejected because they expected to be able to do everything they could do on the website.) Some mobile apps need to do more than the desktop versions.

Don’t fear extra taps! Web has made us squeamish about number of clicks. Latency not an issue in a native/cached app. Tap quality* trumps tap quantity (*unconfusing)

Similarly, don’t fear scrolling. E.g. USA Today used an accordion interface to avoid a long list of headlines scrolling, but not all users understood it. For long lists of content, scrolling is still better.

Changing orientation: think of it not just as a change in layout, but also a change in mindset. Landscape mode can also be seen as “focused mode”. But beware of depending on it, as it is hard to discover. (Personally, I believe orientation in iOS is flawed, invoked overwhelmingly by accident. I think Ben Summers has the right idea for how it should work.)

Affordances: gestures are the keyboard shortcuts of touch interfaces, they can be hard for users to discover. Shortcuts need backup plans – never rely on all users discovering them.

When you see your app being used, look for unsuccessful gesture attempts and repetitive interactions, and pave the cowpaths. (Have Apple never seen users try to swipe the Calendar app to change day?)

Multi-touch (on phones, not iPad) and the shake gesture generally considered bad.

Don’t get carried away with an impressive, glitzy interface. Showcase the content, not the form. Don’t underestimate the power of the humdrum and familiar – see NY Times and Flipboard apps. Similarly, skeuomorphs, while still window-dressing, can enhance a design; familiarity and intimacy invite touch. But if you’re aping a physical object, choose the right metaphor.

But try to avoid buttons altogether. Buttons are a hack. Look at how a toddler uses an iPhone/iPad – they try to interact directly with the content. Wherever possible, make the content the interface. (Example: see how Twitter for iPad eliminated the Back button.)

Clark signed off on an optimistic note, and an encouragement to experiment: new platforms don’t appear very often. This is the coolest job in the world.

Improving Harvest using seductive interactions

One of my favourite talks at UX London 2010 was Stephen Anderson’s Seductive Interactions – using basic psychological principles to bridge the gap between business goals and users’ behavioural goals. Usability alone, he argues, often merely decreases friction. Using psychology can also increase users’ motivation.

At UXLx, the recent UX conference in Lisbon, I attended Anderson’s workshop on the same topic. By luck, our exercises focused on improving one of my great bugbears, time tracking. And not just any time tracker: Harvest, which we use at Isotoma.

Don’t get me wrong: Harvest is by a long stretch the best time tracker I’ve used. But nevertheless, it remains an activity I hate and avoid, and routinely fill in my time weeks late.

Applying typical utilitarian thinking would result in improvements like –

  • Increasing the font sizes in Day view
  • Week view should highlight active rows and columns, and keep column headers and footers visible when scrolling

– helpful, but unlikely to change my attitude or behaviour.

In the workshop we used Anderson’s “Mental Notes”, each of which describe a psychological principle designers can take advantage of, and used them to generate ideas to improve Harvest. Here are the relevant notes, and what we came up with:

Trigger / Recognition Over Recall / Feedback Loops

Feedback Loops; Trigger; Recognition Over RecallThe application should listen and learn from your actions. Just like a good online supermarket will make it easy to re-order the things you frequently buy, Harvest should always automatically show the jobs you worked on previously.

It should also trigger you to take action. At the end of each day, it should pop up a form: “Did you work on these jobs today?” containing only the jobs you had put time on the day before.

A frequent problem is that a job might not exist in the system yet, because a project manager hasn’t created it, or hasn’t added you to it.

Firstly, in the Day and Week views, you should see all the jobs you’ve been added to, not only the ones you’ve chosen to show. (The dropdown menu to add jobs is a terrible interface.)

Secondly, you should be able to create “placeholder” jobs and assign times to them, which you can later reconcile with an actual job. Placeholder times will be used in your own totals, but not in project managers’ reports.

Harvest popup window

Status / Achievements / Competition / Appropriate Challenges

Competition; Status; Achievements; Appropriate ChallengesAnderson showed the example of Target’s supermarket checkout interface, which uses a game-like system to encourage cashiers to work more quickly (photo).

Similar principles can be used to encourage workers to fill their timesheets in sooner. Harvest should track how long, on average, it takes you to fill in your timesheet (same day, 1 day late, 7 days late, etc.) This should be shown to you as a chart over time, so you can see your average and your trend.

Then it should also show you the same timeliness charts for your colleagues, and where you rank (without names, as no-one would appreciate the feeling of being singled out.) This will act as a subtle but powerful spur to improve your timeliness. The company can also offer rewards for the most timely, or the greatest improvement.

Set Completion

Set CompletionHighlight it when a day totals up to 7.5h or more (or whatever your daily goal is). Completed days should stand out clearly on the Week view and reports. And how about a “Well done!” whenever you complete a day?

Another example: 750 Words encourages writers to stick to a writing schedule by writing only 750 words a day, checking off a box when they’ve done so. (See screenshot here.)

Delighters / Surprise / Humor Effect / Visual Imagery / Self Expression

Delighters; Surprise; Humor Effect; Visual Imagery; Self ExpressionIn a different UXLx talk, Andrew Watterson paraphrased usability pope Don Norman’s adage that users have 2 basic needs: (1) to get something done, and (2) to smile. Adding a touch of humour to an application can go a long way towards making it more pleasurable to use and memorable.

This is why our own Forkd.com uses tomatoes instead of asterisks on forms, amongst many other touches, and I’ve lost count of the number of people who have commented on that.

Forkd.com registration form (excerpt)Some ideas that cropped up here were for the application to offer occasional comments and quips, e.g. advising users to mind their posture, or go for a walk now and then, or pep talk suggesting how hard-working they are. Ex-colleague Jonathan Baker-Bates designed a custom-built time tracker that allowed employees to add their own quips, which rapidly became an impromptu means of humourous communication and teasing within the company (like the #isotoma IRC channel’s notorious topics, I imagine.)

Another ex-colleague, Karl Sabino and I came up with the idea of adding “themes” to the time entry interface. Instead of the current businesslike grey-and-orange, you could fill in your times on something resembling a ticking bomb, or a Superbowl scoreboard, or an Indian restaurant menu. To our surprise, it turned out that Dominos Pizza already had this idea on their patented Pizza Tracker:

Screenshot of Dominos Pizza Tracker

Many customers found this amusing enough to post videos of it on YouTube.

So there you have it: a fresh approach to a design problem that yielded many results I doubt I’d have come up with otherwise. When I tweeted about the workshop, Harvest immediately responded with interest. I hope you like the results! (And my employers will thank you if you can somehow improve my terrible timesheet habits.)

I’d like to acknowledge the input of Karl Sabino and Jonathan Baker-Bates in several of the ideas above.

Setting the modification date of an Archetype object in Plone.

Modification dates of Archetypes objects are set automatically to the current
date and time when an object is reindexed. Usually this is useful behaviour but if
,for example, you’re importing content from elsewhere into a Plone site and you’d
like to set the modification date to be some time in the past then it’s a problem.

Heres a method that sets the standard dates of an Archetypes content type ‘obj’
to DateTime ‘dt’.

def setObjDate(obj, dt):
    """Prevent update of modification date 
       during reindexing"""
    obj.setCreationDate(dt)
    obj.setEffectiveDate(dt)
    obj.setModificationDate(dt)
    od = obj.__dict__
    od['notifyModified'] = lambda *args: None
    obj.reindexObject()
    del od['notifyModified']

The trick, based on migration code found in the plone.app.blob package, is
to mask the ‘notifyModified’ class method with a no-op lambda on our
instance, call the reindex, then drop the mask so that the class method is visible
once again.

I’ve used this successfully for importing data into a Plone 4 instance but it should work at least as far back as Plone 2.5.

Tamper-protection for Bank Transactions

If you need to send electronic transactions to Swedish banks, you’ll be required to add anti-tampering seals to the files. The banks recommend you use a third-party system to create the HMAC SHA256-128 seals, but that could involve a fair amount of expensive server software and maintenance contracts (some linked to the number of people who work in your company).

Instead, you can do it yourself in Python like this:

import hmac
import hashlib
import string

NORMALISE = string.maketrans(
    '\xC9\xC4\xD6\xC5\xDC\xE9\xE4\xF6\xE5\xFC' + ''.join(
    [chr(x) for x in range(0,32)]) + ''.join(
    [chr(x) for x in range(127,256)
     if x not in (201,196,214,197,220,233,
                  228,246,229,252)]),
    '\x40\x5B\x5C\x5D\x5E\x60\x7B\x7C\x7D\x7E' + ''.join(
    [chr(195) for x in range(0,32)]) + ''.join(
    [chr(195) for x in range(127,256)
     if x not in (201,196,214,197,220,233,
                  228,246,229,252)]))

def hex_to_bytes(hexs):
    """Convert string of hex into bytes"""
    return ''.join(['%s' % chr(int(hexs[i:i+2], 16))
                    for i in range(0, len(hexs), 2)])

def get_signature(contents, key):
    """Calculate the HMAC SHA256-128 signature

       contents - an iso-8859-1 (latin-1) encoded string
       key - a string of hex characters

       Returns a 32 char string of hex characters (128 bits)
    """
    key = hex_to_bytes(key)

    #Normalise the contents
    contents = contents.translate(NORMALISE, '\r\n')

    dig = hmac.new(key, msg=contents,
                   digestmod=hashlib.sha256).digest()
    return ''.join(['%02X' % ord(x) for x in dig[:16]])

And then to calculate the signature for a file:

>>> print get_signature(open('bankfile.dat').read(),
...                     '1234567890abcdef1234567890abcdef')
25122AE4179BD51DC87AD6EA08D16D45

Seeking a Head of QA

Isotoma is looking for someone with solid testing and management experience to come and help us shape and build our QA function. The role will be to take on the formalisation of our quality control systems, from traditional manual testing to automated CI, build and deployment. We’re looking for someone who can run the testing function whilst also getting involved with all 3 teams within the company (dev, devops and ops), helping them improve their processes and approaches upstream.
Read more about it on the main site (Head of Quality Assurance vacancy) and please get in touch.

PloneConf2010 – and there’s more

I’m here at PloneConf2010 too, and I’ve been to mostly different talks to Mitch, so here’s my write up too.  It’s taken a bit of time to find the time to write this up properly!

Calvin Hendryx-Parker: Enterprise Search in Plone using Solr

We’ve been using Solr for a few years here at Isotoma, but so far we’ve not integrated it with Plone.  Plone’s built-in Catalog is actually pretty good, however one thing it doesn’t do fantastically well is full-text search.  It is passable in English, but has very limited stemming support – which makes it terrible in other languages.

Calvin presented their experience of using Solr with Plone. They developed their own software to integrate the Plone catalog with Solr, instead of using collective.solr, which up till then was the canonical way of connecting them. Their new product alm.solrindex sounds significantly better than collective.solr.  Based on what I’ve heard here, you should definitely use alm.solrindex.

To summarise how this all hangs together, you need an instance of Solr installed somewhere that you can use.  You can deploy a solr specifically for each site, in which case you can deploy it through buildout.  Solr is Java, and runs inside various Java application servers.

You can also run a single Solr server for multiple Plone sites – in which case you partition the Solr database.

You then configure Solr, telling it how to index and parse the fields in your content. No configuration of this is required within Plone.  In particular you configure the indexes in Solr not in Plone.

Then install alm.solrindex in your plone site and delete all the indexes that you wish to use with Solr. alm.solrindex will create new indexes by inspecting Solr.

Then reindex your site, and you’re done!  It supports a lot of more complex use cases, but in this basic case you get top-end full text indexing at quite low cost.

Dylan Jay, PretaWeb: FunnelWeb

Funnelweb sounds invaluable if you want to convert an existing non-Plone site into a Plone site, with the minimum effort.

Funnelweb is a tool based on transmogrifier. Transmogrifier provides a “pipeline” concept for transforming content. Pipeline stages can be inserted into a pipeline, and these stages then have the ability to change the content in various ways.

Dylan wrote funnelweb to use transmogrifier and provide a harness for running it in a managed way over existing websites.  The goal is to create a new Plone site, using the content from existing websites.

Funnelweb uploads remotely to Plone over XML-RPC, which means none of transmogrifier needs to be installed in a Plone site, which is a significant advantage.  It is designed to be deployed using buildout, so a script will be provided in your build that executes the import.

A bunch of pipeline steps are provided to simplify the process of importing entire sites.  In particular funnelweb has a clustering algorithm that attempts to identify which parts of pages are content and which are templates.  This can be configured by providing xpath expressions to identify page sections, and then extract content from them for specific content fields.

It supports the concept of ordering and sorts, so that Ordered Folder types are created correctly.  It supports transmogrify.siteanalyser.attach to put attachments closer to pages and transmogrify.siteanalyser.defaultpage to detect index pages in collections and to make them folder indexes in the created sites.

Finally it supports relinking, so that pages get sane urls and all links to those pages are correctly referenced.

Richard Newbury: The State of Plone Caching

The existing caching solution for Plone 3 is CacheFu, which is now pretty long in the tooth.  I can remember being introduced to CacheFu by Geoff Davis at the Archipelago Sprint in 2006, where it was a huge improvement on the (virtually non-existent) support for HTTP caching in Plone.

It’s now looking pretty long in the tooth, and contains a bunch of design decisions that have proved problematic over time, particularly the heavy use of monkeypatching.

This talk was about the new canonical caching package for Plone, plone.app.caching. It was built by Richard Newbury, based on an architecture from the inimitable Martin Aspeli.

This package is already being used on high-volume sites with good results, and from what I saw here the architecture looks excellent.  It should be easy to configure for the general cases and allows sane extension of software to provide special-purpose caching configuration (which is quite a common requirement).

It provides a basic knob to control caching, where you can select strong, moderate or weak caching.

It can provide support for the two biggest issues in cache engineering: composite views (where a page contains content from multiple sources with different potential caching strategies) and split views (where one page can be seen by varying user groups who cannot be identified entirely from a tuple of URL and headers listed in Vary).

It provides support for nginx, apache, squid and varnish.  Richard recommends you do not use buildout recipes for Varnish, but I think our recipe isotoma.recipe.varnish would be OK, because it is sufficiently configuration.  We have yet to review the default config with plone.app.caching though.

Richard recommended some tools as well:

  • funkload for load testing
  • browsermob for real browsers
  • HttpFox instead of LiveHttpHeaders
  • Firebug, natch
  • remember that hitting refresh and shift-refresh force caches to refresh.  Do not use them while testing!

Jens Klein: Plone is so semantic, isn’t it?

Jens introduced a project he’s been working on called Interactive Knowledge Stack (IKS), funded by the EU.  This project is to provide an open source Java component for Content Management Systems in Europe to help the adoption of Semantic concepts online.  The tool they have produced is called FISE. The name is pronounced like an aussie would say “phase” ;)

FISE provides a RESTful interface to allow a CMS to associate semantic statements with content.  This allows us to say, for example that item X is in Paris, and in addition we can state that Paris is in France.  We can now query for “content in France” and it will know that this content is in France.

The provide a generic Python interface to FISE which is usable from within Plone.  In addition it provides a special index type that integrates with the Plone Catalog to allow for updating the FISE triple store with the information found in the content.  It can provide triples based on hierarchical relationships found in the plone database (page X is-a-child-of folder Y).

Jens would like someone to integrate the Aloha editor into Plone, which would allow much easier control by editors of semantic statements made about the content they are editing.

Plone Conference 2010: Day 1

So it’s the end of day one halfway through day two (this took a lot longer to write than I thought!) at this year’s Plone Conference and between the two of us, we’ve seen 13 talks so far. Whilst I can’t comment on the ones that Doug has been to, I’m going to summarize my experience of today’s talks. You might also want to take a look at Rick Hurst’s summary of day 1.

Keynote: The Future of Plone by Alexander Limi & Alan Runyan

Slides | Stream

TL;DR: Plone 4 is actually pretty awesome; Deliverance/XDV is now Diazo; we need to engage more with other communities and concentrate on design for Plone 5.

In an opening keynote not dissimilar to last year’s in Budapest, Alexander Limi set out once again to summarize the previous and the next year of Plone. The major differences being, Plone 4 is now out and the heavier focus on strategic goals rather than technological ones. Lots of congratulations for Plone 4 release going so smoothly. It’s a lot faster, better out of the box and easier to install. There should be a few more minor version increases over the next year or so and as of release 4.1, Plone 3 will no-longer be supported. Looking forward to Plone 5 (which Hanno Schlichting hesitantly put a date of March 2011 on) a big deal was made of the 3 “D”s of Plone: Dexterity, Deco and Diazo; Diazo being the new, back-of-a-cigarette-packet name for XDV / Deliverance in Plone 5. Whilst Alex focussed on what will be new in Plone 5, Alan spoke of the need to kill off things that are no-longer used or are replaced by new technologies, one example being writing restricted Python through the web:

“Who uses editing Restricted Python through the web?”

The sound of a room full of people putting no hands up.

Laughter.

One thing that particularly threw me as a new developer was the number of different ways to do things and to a certain degree, I think a limiting of these options is a good thing.

The need to make Plone more appealing to the decision makers was also expressed. Alex believes that over the next few years, designers will be making the choice of CMS to use (a company goes to a designer for a website, they come up with a great design, then have to pick the CMS onto which that design should be applied by developers or a separate company). Other places to focus on getting the Plone word out there: other conferences, local user groups and getting Plone available to people using cheaper hosting solutions, such as getting a cPanel installer working, a stark contrast to last year saying “we’re not really in the same market as Joomla and Drupal and WordPress [...] it will always be easier to pay $5 and get a PHP host somewhere; that’s not something we can fight”.

Mad About Mobile by Mikko Ohtamaa

Slides | Stream

TL;DR: Your clients want mobile sites, if they don’t you should persuade them. WebAndMobile from mFabrik will help you solve a lot of common problems and add a lot of cool features so you can deliver.

Mikko opened by saying something along the lines of, “Your clients want mobile sites, if they don’t you should persuade them” and I couldn’t agree more. Not only will a mobile site increase the number of users you can reach (or the times at which they can use your website), it can also add some really cool features (Mikko lists geolocation, touch and multi-touch events & phone interactions like click to call & contact card download). Furthermore it’s a good way to provide a minimal version of your site to users who perhaps have problems with their sight or mobility.

He touched on the different levels of mobilization that are achievable starting with custom CSS and going all the way up to writing a completely new mobile app.

The Plone extension route provides a nice middle ground that will get you most of what you probably want from a mobile version of your Plone site (mobile browser detection, image resizing and serving different content for mobile platforms). Mikko has been working on WebAndMobile for the past year and a half, it actually has documentation and there are plans to merge in common components of Infrae’s
mobi.*

group of packages. It is currently live on about 6 Plone sites and has some crossover capability for other frameworks such as Django. It:

  • Will automatically redirect to a mobile site based on information in the HTTP request and can scale the site to the features of the user’s platform;
  • Provides support for theming different platforms with different capabilities;
  • Allows integration of mobile-features such as opening a waypoint in a smartphone’s maps or navigation application;
  • Provides the option of serving alternate or no content for content type fields and integrates seamlessly into the existing editor interface.

Neat. For me it looks like the best way to get most of the way towards adding a mobile version of your Plone site. Whilst it won’t be a free-ride, it’s a hell of a good step in the right direction.

High Performance Sites Made Easy by Matthew Wilkes

Slides | Stream

TL;DR: Everything you thought before about optimization is probably wrong. You need to use the catalog less, work off real-world numbers rather than profiling and cache the hell out of everything you can, no matter how insignificant it may seem.

For me this was probably the most useful talk of the day. It really busted quite a few myths I believed in and exposed new techniques that I thought weren’t possible with Plone, namely Edge-Side Includes in Varnish. The talk started with a brief summary of what is and isn’t important, backed up by some interesting stats. For example, whilst not waking objects is fine as a rule of thumb, it’s actually slower to use the catalog to access data up to about 50 objects. There was a big focus on optimization based on real data and putting it off until you can get some. Matthew eschews profiling in favour of analysis of page load time and server logs of normal site usage. One thing which will be of particular use is the tool Jarn have written for analyzing HAProxy logs.

Catalog tips included removing as many indexes as possible, generating better brains using adapters for the
ICatalogBrain

interface and being particularly careful with ZODB cache size: walking the catalog may in fact clear the ZODB cache given the huge number of tiny objects the catalog uses.

Quick wins for caching can be achieved by implementing ESIs (particularly implementing them from
BrowserView __call__

methods so they are more flexible and can be disabled quickly in case of emergency) and avoiding status messages (not only does they make a page not cacheable by Varnish, but Varnish caches that it isn’t cacheable for 3 mins!).

Finally he makes the excellent point that it can be worth haggling with the client over arbitrary decisions they may have made about things such as list size, pagination, long form-based processes & workflow, all of which can inhibit caching.

Lose weight now, ask me how! by Roché Compaan

Slides | Stream

TL;DR: The catalogs are massive bottlenecks; we rewrote some common parts of Plone that overuse them.

Continuing in a similar vein to Matthew Wilke’s talk, but taking a more extreme approach, Roché Compaan almost seems like he is out to destroy the catalogs, and not without justification. He began by asking the question, “can Plone scale to 10 million objects?” Starting at the lowest level, he produced some benchmarks of the ZODB using
collective.zodbbench

in 2007, compared them to similar operations for Postgres and found that ZODB performs extremely well, holding steady at about 250 writes per second achieved on a sample database of 10m objects.

When it came to analyzing Plone (using
ZopeProfiler


experimental.catalogqueryplan

and 
ZODB.FileStorage.fsdump

) he found some pretty horrifying stats about the catalog. For example, 90% of a Data.fs for a database of 10,000 content objects was dedicated to objects supporting the catalogs. Unfortunately he didn’t seem to have any stats on how the catalog affects speed.

upfront.diet

is a suite of products that offer replacements for parts of Plone that either cut out catalogs entirely or drastically reduce their usage. The most useful of these seem to be the upfront.simplereferencefield and
upfront.catalogblacklist

.
upfront.catalogblacklist

seems like something that should have been offered in Plone to begin with; it allows the developer to limit the indexing of specific content types through genericsetup.
upfront.simplereferencefield

is a replacement for the Archetypes reference field that doesn’t use the UID catalog. According to his stats, importing 50k objects using the archetypes reference field took 50 mins, while importing the same data set using the
upfront.simplereferencefield

took 5 mins. Impressive results, but this slash and burn approach to catalog usage leaves me uneasy, particularly when it’s something that is under consideration for Plone 5 anyway.

Others

Other talks I went to were Giving light to dark corners of z3c.form by Rok Garbas which was a good, quick intro to what seems to be the most flexible of all the Plone form frameworks; Laying Pipe with Transmogrifier by Clayton Parker which was a brief summary of the features of
collective.transmogrifier

which seems like a very powerful tool for importing data from various sources, but as Rick Hurst says, probably a bit heavyweight for what most people will want; and Tools and techniques for a successful Plone project by Martin Aspeli in which he shared a sneak preview of some of the tools and techniques he will be recommending in the new Plone 4 edition of his book.

Finally, at the end of day 1 was the guest keynote from Richard Noble, a very entertaining affair on the spirit of engineering and how teamwork matters far more than any amount of funding you might have.

An excellent start to the conference!

Twitter redesign: first impressions

Wow, my last post became obsolete fast. Good to know that the talented designers at Twitter have indeed been working on a root-and-branch revamp of Twitter.com all this time. Sorry if I cast aspersions that you weren’t! Let’s see how they fared against my wishlist:

New home page

1. How is “Profile” a suitable label for your Twitter stream?
Well, you still have to click Profile to see your tweets. But they’re clearly titled “Timeline” now, and this is not the only way to get to it.
The tab is also highlighted when selected.
Score: 7/10

2. Why is there not a decent Profile page?
We still don’t have detailed Profile pages, with more than 140 characters and 1 link, and room for more personal information, account statistics and analysis. Curious to know the rationale for this.
Score: 0/10

3. Why on earth is there only a Tweet box at the top of the Home page?
and
4. As soon as you scroll down, you lose the Tweet box.
These problems have been fixed 100%. Not only is the box above both Home and Profile, but there at any point you can invoke a floating, repositionable tweet box from the button in the header.
Score: 10/10, 10/10

New Profile page

5. Inconsistency in the right-hand navigation between the Home page and the Profile page.
Well, the right-hand navigation has been completely changed. These links are now tabs on the Home and Profile pages respectively, and differ between these two pages. At least some thought appears to have gone into which tabs are shown. Would have to see if I find them intuitive. I may still end up wondering how to get to @Mentions while on the Profile page, or where to find things I’ve retweeted. Messages are in a more appropriate location, well separated, in the header.
Score: will have to see.

6. The Home right-hand navigation area is also polluted by, essentially, promotions
They have now been clearly separated from your content, and no longer pushing some of it down. Still occupies a lot of prime real estate, though.
Score: 8/10

7. The right-hand navigation could generally be used a lot better.
OK, still can’t browse my hashtags. And don’t have totals for @Mentions or Retweets, but that’s not really important. The search box is now in a sensible place, in the header. And they’ve done plenty of other useful things with the right-hand column.
Score: 8/10

8. Only when you’re on the Home page (twitter.com) can you easily see who you’re signed in as
Fixed. This is now in a sensible place, top right.
Score: 10/10

9. Twitter.com does a terrible job of showing you recent activity.
Still terrible. Still no alert for an @Mention. Not even a new DM is highlighted in any way.
Score: 0/10

10. Inability to Search inside your own Twitter stream, or inside someone else’s.
No change here, by the looks of it.
Score: 0/10

11. When you look at your, or someone else’s Profile, Twitter could display much, much more useful information.
Besides room for more personal information, I was hoping for statistics and analysis, such as tweet frequency and tweet type ratios, frequently-used hashtags, time on Twitter, etc. No change here either.
Score: 0/10

12. The “More” page-down experience is atrocious.
OK, AJAX (sort of) to the rescue. The new “infinite scroll” is very impressive and (in my tests) super-fast. However, it still does not allow me to easily make big leaps into the past, the way paging links would’ve. When you navigate away, and click Back, you’re also not (instantly) back to where you were, although it tries to do so and catches up after a few seconds.
Score: 6/10

13. The display of user lists is bad, both in the mini form (avatars in the right-hand column) and the full page listings.
This is also improved by the AJAX infinite scroll, and the layout is improved, but still no sorting, or searching for a name within a list of contacts. Also, the experience of mousing over mini-avatars to see names hasn’t been improved.
Score: 3/10

14. Automatically hyperlink a Tweet ID.
Unsurprisingly, no. (I’d be curious to know whether this is feasible, incidentally.)
Score: 0/10

So, in total, of my 14 criticisms, 7 were completely or mostly corrected, and 6 only imperfectly or not at all. One still needs further testing.

My analysis above looks only at the points I raised in my previous post, and does not take into account the smart new layout and many new features, some of them quite wonderful. I especially like the photos, videos and conversations in the new right-hand pane. (Although the pane often seems to be filled up with not-really-relevant stuff just because there’s room.) I want to experience the beta a bit longer to get a feel for everything that’s new.

Why is Twitter.com so badly designed?

Since I’m always complaining about Twitter.com, it’s just fair for me to list what I think is wrong with it and how it should be improved. Please note: I’m only criticising the Twitter.com website, not the service itself or any other Twitter client.

1. Twitter Profile page highlighting top navigationHow is “Profile” a suitable label for your Twitter stream? Can you imagine someone talking about their “profile” on Twitter, when they mean their updates? Making it even more confusing, this menu item is not even highlighted after it’s selected.

2. For that matter, why is there not a decent Profile page? (In the normal social networking sense of the word.) Tweets are restricted to 140 characters for a good reason; there is no reason to have only a 140 character bio and a single link.

3. Why on earth is there only a Tweet box at the top of the Home page? At the very least, there should be one on the Profile page. Really, you should be able to tweet from any page on Twitter.com.

4. As soon as you scroll down, you lose the Tweet box. So if an update lower down on the page spurred you to write, or if you click Reply, bad luck: you have to lose it from sight in order to write.

5. Comparison of right-hand navigation on Twitter Home and Profile pages respectivelyLook at the inconsistency in the right-hand navigation between the Home page and the Profile page. Home has @replies, DMs, Favorites and Retweets (itself a disaster). Profile has Tweets and Favorites only. Is there any reason for them to be different on your own Profile? (On someone else’s Profile it will be Tweets and Favorites only, of course.) I’ve often been confused at the disappearance of, say, @replies, only to realise it’s because I’m on the Profile, rather than the Home page.

6. The Home right-hand navigation area is also polluted by, essentially, promotions, such as “Who to follow” and “Trending”. They are not visually distinguished from real navigation.

7. The right-hand navigation could generally be a lot better. Why not have the hashtags you’ve used, since they’re effectively a way of categorising your tweets? Why not display the total numbers of @replies, Favorites and Retweets (as for DMs)? The Search box is in a silly place, and being directly below your navigation links, implies it’s searching your tweets, not all tweets.

8. Only when you’re on the Home page (twitter.com) can you easily see who you’re signed in as (badge at top right). On the Profile page, only the “That’s you!” under the photo at the top tells you. On other pages, nothing tells you. In a household with multiple Twitter accounts, this is rather frustrating.

9. Twitter.com does a terrible job of showing you recent activity. When you log in to Twitter, you expect bright highlights alerting you of new @replies, if one of your tweets was favorited or retweeted, or if you have new DMs. When done right, this really boosts addictiveness: Facebook and Flickr are just two examples.

10. The inability to Search inside your own Twitter stream, or inside someone else’s, is simply crippling. I assume that this is a technical limitation due to Twitter’s scale, but it should be a top priority. If Google had bought Twitter, I imagine this would’ve been the first thing they’d fix.

11. When you look at someone else’s Profile, Twitter could display much, much more useful information. (Having a real Profile page would help.) Basically, the sort of information that 3rd-party services like MrTweet offers: their tweet frequency, how long they’ve been on Twitter, the nature of their Twitter usage (ratio of posts with links, ratio of posts that are @replies, recurrent hashtags, etc.)

On your own Profile, you’d want to see statistics. Number of views of your stream, number of favourites and retweets, ideally graphed over time (like Flickr does it).

12. The “More” page-down experience is atrocious. It’s slow, you can’t skip by more than a page at a time, so if you want to go back a few days it takes forever. And if you navigate away, and click Back, the entire stream is gone again. Presumably this is another technical limitation.

Ideally, I want actual paging links like on Flickr or Vimeo. (Only for a single person’s stream, not the firehose.)

13. The display of user lists is bad, both in the mini form (avatars in the right-hand column) and the full page listings.

On the mini version, relying on tooltips or the browser status bar to read a person’s name is very user-unfriendly. You want to scrub your mouse over the avatars and easily read the names as you do so.

On the full-page versions, where is the paging? Like the Twitter stream, there’s only a “Next” link. How are you expected to navigate through more than a 100 people? They are in no discernable order, and there’s no ability to order them by username, first or last name, or search within the list (like Facebook).

14. An obvious, if geeky, enhancement: automatically hyperlink a Tweet ID. Currently if you want to link to another tweet, you have to use an URL shortener. (A tweet ID is 9 characters shorter than a bit.ly link.)

And here are just some shortcomings that were fixed scandalously late:

  • Native URL shortening
  • The Follow button used to be a gear for ages
  • Emails alerting you to new followers contained no useful info (the follower’s bio was just added a few weeks ago)

I understand Twitter is focusing on growth, and many shortcomings are unavoidable results of its scale (such as Search and proper paging), but they cannot afford to lose sight of the user experience. But mostly I can’t understand how the remarkably talented UX folks they’ve been hiring — people like Kevin Cheng and Doug Bowman — can allow these problems to persist for so long. (While simultaneously rolling out unimpressive features like the infernal hovercards and “Who to follow”.)

Perhaps Twitter’s attitude is to focus on the API, and leave the user experience to third-party services and Twitter clients. That would be a pity: it’s a jungle out there if you’re looking for decent Twitter services. For every decent 3rd-party service or client there’s a plethora of ones that are ramshackle, spammy or downright malware. I also find this explanation implausible, given their investment in UX design talent.