Setting the modification date of an Archetype object in Plone.

Modification dates of Archetypes objects are set automatically to the current
date and time when an object is reindexed. Usually this is useful behaviour but if
,for example, you’re importing content from elsewhere into a Plone site and you’d
like to set the modification date to be some time in the past then it’s a problem.

Heres a method that sets the standard dates of an Archetypes content type ‘obj’
to DateTime ‘dt’.

def setObjDate(obj, dt):
    """Prevent update of modification date 
       during reindexing"""
    obj.setCreationDate(dt)
    obj.setEffectiveDate(dt)
    obj.setModificationDate(dt)
    od = obj.__dict__
    od['notifyModified'] = lambda *args: None
    obj.reindexObject()
    del od['notifyModified']

The trick, based on migration code found in the plone.app.blob package, is
to mask the ‘notifyModified’ class method with a no-op lambda on our
instance, call the reindex, then drop the mask so that the class method is visible
once again.

I’ve used this successfully for importing data into a Plone 4 instance but it should work at least as far back as Plone 2.5.

Tamper-protection for Bank Transactions

If you need to send electronic transactions to Swedish banks, you’ll be required to add anti-tampering seals to the files. The banks recommend you use a third-party system to create the HMAC SHA256-128 seals, but that could involve a fair amount of expensive server software and maintenance contracts (some linked to the number of people who work in your company).

Instead, you can do it yourself in Python like this:

import hmac
import hashlib
import string

NORMALISE = string.maketrans(
    '\xC9\xC4\xD6\xC5\xDC\xE9\xE4\xF6\xE5\xFC' + ''.join(
    [chr(x) for x in range(0,32)]) + ''.join(
    [chr(x) for x in range(127,256)
     if x not in (201,196,214,197,220,233,
                  228,246,229,252)]),
    '\x40\x5B\x5C\x5D\x5E\x60\x7B\x7C\x7D\x7E' + ''.join(
    [chr(195) for x in range(0,32)]) + ''.join(
    [chr(195) for x in range(127,256)
     if x not in (201,196,214,197,220,233,
                  228,246,229,252)]))

def hex_to_bytes(hexs):
    """Convert string of hex into bytes"""
    return ''.join(['%s' % chr(int(hexs[i:i+2], 16))
                    for i in range(0, len(hexs), 2)])

def get_signature(contents, key):
    """Calculate the HMAC SHA256-128 signature

       contents - an iso-8859-1 (latin-1) encoded string
       key - a string of hex characters

       Returns a 32 char string of hex characters (128 bits)
    """
    key = hex_to_bytes(key)

    #Normalise the contents
    contents = contents.translate(NORMALISE, '\r\n')

    dig = hmac.new(key, msg=contents,
                   digestmod=hashlib.sha256).digest()
    return ''.join(['%02X' % ord(x) for x in dig[:16]])

And then to calculate the signature for a file:

>>> print get_signature(open('bankfile.dat').read(),
...                     '1234567890abcdef1234567890abcdef')
25122AE4179BD51DC87AD6EA08D16D45

Seeking a Head of QA

Isotoma is looking for someone with solid testing and management experience to come and help us shape and build our QA function. The role will be to take on the formalisation of our quality control systems, from traditional manual testing to automated CI, build and deployment. We’re looking for someone who can run the testing function whilst also getting involved with all 3 teams within the company (dev, devops and ops), helping them improve their processes and approaches upstream.
Read more about it on the main site (Head of Quality Assurance vacancy) and please get in touch.

PloneConf2010 – and there’s more

I’m here at PloneConf2010 too, and I’ve been to mostly different talks to Mitch, so here’s my write up too.  It’s taken a bit of time to find the time to write this up properly!

Calvin Hendryx-Parker: Enterprise Search in Plone using Solr

We’ve been using Solr for a few years here at Isotoma, but so far we’ve not integrated it with Plone.  Plone’s built-in Catalog is actually pretty good, however one thing it doesn’t do fantastically well is full-text search.  It is passable in English, but has very limited stemming support – which makes it terrible in other languages.

Calvin presented their experience of using Solr with Plone. They developed their own software to integrate the Plone catalog with Solr, instead of using collective.solr, which up till then was the canonical way of connecting them. Their new product alm.solrindex sounds significantly better than collective.solr.  Based on what I’ve heard here, you should definitely use alm.solrindex.

To summarise how this all hangs together, you need an instance of Solr installed somewhere that you can use.  You can deploy a solr specifically for each site, in which case you can deploy it through buildout.  Solr is Java, and runs inside various Java application servers.

You can also run a single Solr server for multiple Plone sites – in which case you partition the Solr database.

You then configure Solr, telling it how to index and parse the fields in your content. No configuration of this is required within Plone.  In particular you configure the indexes in Solr not in Plone.

Then install alm.solrindex in your plone site and delete all the indexes that you wish to use with Solr. alm.solrindex will create new indexes by inspecting Solr.

Then reindex your site, and you’re done!  It supports a lot of more complex use cases, but in this basic case you get top-end full text indexing at quite low cost.

Dylan Jay, PretaWeb: FunnelWeb

Funnelweb sounds invaluable if you want to convert an existing non-Plone site into a Plone site, with the minimum effort.

Funnelweb is a tool based on transmogrifier. Transmogrifier provides a “pipeline” concept for transforming content. Pipeline stages can be inserted into a pipeline, and these stages then have the ability to change the content in various ways.

Dylan wrote funnelweb to use transmogrifier and provide a harness for running it in a managed way over existing websites.  The goal is to create a new Plone site, using the content from existing websites.

Funnelweb uploads remotely to Plone over XML-RPC, which means none of transmogrifier needs to be installed in a Plone site, which is a significant advantage.  It is designed to be deployed using buildout, so a script will be provided in your build that executes the import.

A bunch of pipeline steps are provided to simplify the process of importing entire sites.  In particular funnelweb has a clustering algorithm that attempts to identify which parts of pages are content and which are templates.  This can be configured by providing xpath expressions to identify page sections, and then extract content from them for specific content fields.

It supports the concept of ordering and sorts, so that Ordered Folder types are created correctly.  It supports transmogrify.siteanalyser.attach to put attachments closer to pages and transmogrify.siteanalyser.defaultpage to detect index pages in collections and to make them folder indexes in the created sites.

Finally it supports relinking, so that pages get sane urls and all links to those pages are correctly referenced.

Richard Newbury: The State of Plone Caching

The existing caching solution for Plone 3 is CacheFu, which is now pretty long in the tooth.  I can remember being introduced to CacheFu by Geoff Davis at the Archipelago Sprint in 2006, where it was a huge improvement on the (virtually non-existent) support for HTTP caching in Plone.

It’s now looking pretty long in the tooth, and contains a bunch of design decisions that have proved problematic over time, particularly the heavy use of monkeypatching.

This talk was about the new canonical caching package for Plone, plone.app.caching. It was built by Richard Newbury, based on an architecture from the inimitable Martin Aspeli.

This package is already being used on high-volume sites with good results, and from what I saw here the architecture looks excellent.  It should be easy to configure for the general cases and allows sane extension of software to provide special-purpose caching configuration (which is quite a common requirement).

It provides a basic knob to control caching, where you can select strong, moderate or weak caching.

It can provide support for the two biggest issues in cache engineering: composite views (where a page contains content from multiple sources with different potential caching strategies) and split views (where one page can be seen by varying user groups who cannot be identified entirely from a tuple of URL and headers listed in Vary).

It provides support for nginx, apache, squid and varnish.  Richard recommends you do not use buildout recipes for Varnish, but I think our recipe isotoma.recipe.varnish would be OK, because it is sufficiently configuration.  We have yet to review the default config with plone.app.caching though.

Richard recommended some tools as well:

  • funkload for load testing
  • browsermob for real browsers
  • HttpFox instead of LiveHttpHeaders
  • Firebug, natch
  • remember that hitting refresh and shift-refresh force caches to refresh.  Do not use them while testing!

Jens Klein: Plone is so semantic, isn’t it?

Jens introduced a project he’s been working on called Interactive Knowledge Stack (IKS), funded by the EU.  This project is to provide an open source Java component for Content Management Systems in Europe to help the adoption of Semantic concepts online.  The tool they have produced is called FISE. The name is pronounced like an aussie would say “phase” ;)

FISE provides a RESTful interface to allow a CMS to associate semantic statements with content.  This allows us to say, for example that item X is in Paris, and in addition we can state that Paris is in France.  We can now query for “content in France” and it will know that this content is in France.

The provide a generic Python interface to FISE which is usable from within Plone.  In addition it provides a special index type that integrates with the Plone Catalog to allow for updating the FISE triple store with the information found in the content.  It can provide triples based on hierarchical relationships found in the plone database (page X is-a-child-of folder Y).

Jens would like someone to integrate the Aloha editor into Plone, which would allow much easier control by editors of semantic statements made about the content they are editing.

Plone Conference 2010: Day 1

So it’s the end of day one halfway through day two (this took a lot longer to write than I thought!) at this year’s Plone Conference and between the two of us, we’ve seen 13 talks so far. Whilst I can’t comment on the ones that Doug has been to, I’m going to summarize my experience of today’s talks. You might also want to take a look at Rick Hurst’s summary of day 1.

Keynote: The Future of Plone by Alexander Limi & Alan Runyan

Slides | Stream

TL;DR: Plone 4 is actually pretty awesome; Deliverance/XDV is now Diazo; we need to engage more with other communities and concentrate on design for Plone 5.

In an opening keynote not dissimilar to last year’s in Budapest, Alexander Limi set out once again to summarize the previous and the next year of Plone. The major differences being, Plone 4 is now out and the heavier focus on strategic goals rather than technological ones. Lots of congratulations for Plone 4 release going so smoothly. It’s a lot faster, better out of the box and easier to install. There should be a few more minor version increases over the next year or so and as of release 4.1, Plone 3 will no-longer be supported. Looking forward to Plone 5 (which Hanno Schlichting hesitantly put a date of March 2011 on) a big deal was made of the 3 “D”s of Plone: Dexterity, Deco and Diazo; Diazo being the new, back-of-a-cigarette-packet name for XDV / Deliverance in Plone 5. Whilst Alex focussed on what will be new in Plone 5, Alan spoke of the need to kill off things that are no-longer used or are replaced by new technologies, one example being writing restricted Python through the web:

“Who uses editing Restricted Python through the web?”

The sound of a room full of people putting no hands up.

Laughter.

One thing that particularly threw me as a new developer was the number of different ways to do things and to a certain degree, I think a limiting of these options is a good thing.

The need to make Plone more appealing to the decision makers was also expressed. Alex believes that over the next few years, designers will be making the choice of CMS to use (a company goes to a designer for a website, they come up with a great design, then have to pick the CMS onto which that design should be applied by developers or a separate company). Other places to focus on getting the Plone word out there: other conferences, local user groups and getting Plone available to people using cheaper hosting solutions, such as getting a cPanel installer working, a stark contrast to last year saying “we’re not really in the same market as Joomla and Drupal and WordPress [...] it will always be easier to pay $5 and get a PHP host somewhere; that’s not something we can fight”.

Mad About Mobile by Mikko Ohtamaa

Slides | Stream

TL;DR: Your clients want mobile sites, if they don’t you should persuade them. WebAndMobile from mFabrik will help you solve a lot of common problems and add a lot of cool features so you can deliver.

Mikko opened by saying something along the lines of, “Your clients want mobile sites, if they don’t you should persuade them” and I couldn’t agree more. Not only will a mobile site increase the number of users you can reach (or the times at which they can use your website), it can also add some really cool features (Mikko lists geolocation, touch and multi-touch events & phone interactions like click to call & contact card download). Furthermore it’s a good way to provide a minimal version of your site to users who perhaps have problems with their sight or mobility.

He touched on the different levels of mobilization that are achievable starting with custom CSS and going all the way up to writing a completely new mobile app.

The Plone extension route provides a nice middle ground that will get you most of what you probably want from a mobile version of your Plone site (mobile browser detection, image resizing and serving different content for mobile platforms). Mikko has been working on WebAndMobile for the past year and a half, it actually has documentation and there are plans to merge in common components of Infrae’s mobi.* group of packages. It is currently live on about 6 Plone sites and has some crossover capability for other frameworks such as Django. It:

  • Will automatically redirect to a mobile site based on information in the HTTP request and can scale the site to the features of the user’s platform;
  • Provides support for theming different platforms with different capabilities;
  • Allows integration of mobile-features such as opening a waypoint in a smartphone’s maps or navigation application;
  • Provides the option of serving alternate or no content for content type fields and integrates seamlessly into the existing editor interface.

Neat. For me it looks like the best way to get most of the way towards adding a mobile version of your Plone site. Whilst it won’t be a free-ride, it’s a hell of a good step in the right direction.

High Performance Sites Made Easy by Matthew Wilkes

Slides | Stream

TL;DR: Everything you thought before about optimization is probably wrong. You need to use the catalog less, work off real-world numbers rather than profiling and cache the hell out of everything you can, no matter how insignificant it may seem.

For me this was probably the most useful talk of the day. It really busted quite a few myths I believed in and exposed new techniques that I thought weren’t possible with Plone, namely Edge-Side Includes in Varnish. The talk started with a brief summary of what is and isn’t important, backed up by some interesting stats. For example, whilst not waking objects is fine as a rule of thumb, it’s actually slower to use the catalog to access data up to about 50 objects. There was a big focus on optimization based on real data and putting it off until you can get some. Matthew eschews profiling in favour of analysis of page load time and server logs of normal site usage. One thing which will be of particular use is the tool Jarn have written for analyzing HAProxy logs.

Catalog tips included removing as many indexes as possible, generating better brains using adapters for the ICatalogBrain interface and being particularly careful with ZODB cache size: walking the catalog may in fact clear the ZODB cache given the huge number of tiny objects the catalog uses.

Quick wins for caching can be achieved by implementing ESIs (particularly implementing them from BrowserView __call__ methods so they are more flexible and can be disabled quickly in case of emergency) and avoiding status messages (not only does they make a page not cacheable by Varnish, but Varnish caches that it isn’t cacheable for 3 mins!).

Finally he makes the excellent point that it can be worth haggling with the client over arbitrary decisions they may have made about things such as list size, pagination, long form-based processes & workflow, all of which can inhibit caching.

Lose weight now, ask me how! by Roché Compaan

Slides | Stream

TL;DR: The catalogs are massive bottlenecks; we rewrote some common parts of Plone that overuse them.

Continuing in a similar vein to Matthew Wilke’s talk, but taking a more extreme approach, Roché Compaan almost seems like he is out to destroy the catalogs, and not without justification. He began by asking the question, “can Plone scale to 10 million objects?” Starting at the lowest level, he produced some benchmarks of the ZODB using collective.zodbbench in 2007, compared them to similar operations for Postgres and found that ZODB performs extremely well, holding steady at about 250 writes per second achieved on a sample database of 10m objects.

When it came to analyzing Plone (using ZopeProfilerexperimental.catalogqueryplan and ZODB.FileStorage.fsdump) he found some pretty horrifying stats about the catalog. For example, 90% of a Data.fs for a database of 10,000 content objects was dedicated to objects supporting the catalogs. Unfortunately he didn’t seem to have any stats on how the catalog affects speed.

upfront.diet is a suite of products that offer replacements for parts of Plone that either cut out catalogs entirely or drastically reduce their usage. The most useful of these seem to be the upfront.simplereferencefield and upfront.catalogblacklist. upfront.catalogblacklist seems like something that should have been offered in Plone to begin with; it allows the developer to limit the indexing of specific content types through genericsetup. upfront.simplereferencefield is a replacement for the Archetypes reference field that doesn’t use the UID catalog. According to his stats, importing 50k objects using the archetypes reference field took 50 mins, while importing the same data set using the upfront.simplereferencefield took 5 mins. Impressive results, but this slash and burn approach to catalog usage leaves me uneasy, particularly when it’s something that is under consideration for Plone 5 anyway.

Others

Other talks I went to were Giving light to dark corners of z3c.form by Rok Garbas which was a good, quick intro to what seems to be the most flexible of all the Plone form frameworks; Laying Pipe with Transmogrifier by Clayton Parker which was a brief summary of the features of collective.transmogrifier

which seems like a very powerful tool for importing data from various sources, but as Rick Hurst says, probably a bit heavyweight for what most people will want; and Tools and techniques for a successful Plone project by Martin Aspeli in which he shared a sneak preview of some of the tools and techniques he will be recommending in the new Plone 4 edition of his book.

Finally, at the end of day 1 was the guest keynote from Richard Noble, a very entertaining affair on the spirit of engineering and how teamwork matters far more than any amount of funding you might have.

An excellent start to the conference!

Twitter redesign: first impressions

Wow, my last post became obsolete fast. Good to know that the talented designers at Twitter have indeed been working on a root-and-branch revamp of Twitter.com all this time. Sorry if I cast aspersions that you weren’t! Let’s see how they fared against my wishlist:

New home page

1. How is “Profile” a suitable label for your Twitter stream?
Well, you still have to click Profile to see your tweets. But they’re clearly titled “Timeline” now, and this is not the only way to get to it.
The tab is also highlighted when selected.
Score: 7/10

2. Why is there not a decent Profile page?
We still don’t have detailed Profile pages, with more than 140 characters and 1 link, and room for more personal information, account statistics and analysis. Curious to know the rationale for this.
Score: 0/10

3. Why on earth is there only a Tweet box at the top of the Home page?
and
4. As soon as you scroll down, you lose the Tweet box.
These problems have been fixed 100%. Not only is the box above both Home and Profile, but there at any point you can invoke a floating, repositionable tweet box from the button in the header.
Score: 10/10, 10/10

New Profile page

5. Inconsistency in the right-hand navigation between the Home page and the Profile page.
Well, the right-hand navigation has been completely changed. These links are now tabs on the Home and Profile pages respectively, and differ between these two pages. At least some thought appears to have gone into which tabs are shown. Would have to see if I find them intuitive. I may still end up wondering how to get to @Mentions while on the Profile page, or where to find things I’ve retweeted. Messages are in a more appropriate location, well separated, in the header.
Score: will have to see.

6. The Home right-hand navigation area is also polluted by, essentially, promotions
They have now been clearly separated from your content, and no longer pushing some of it down. Still occupies a lot of prime real estate, though.
Score: 8/10

7. The right-hand navigation could generally be used a lot better.
OK, still can’t browse my hashtags. And don’t have totals for @Mentions or Retweets, but that’s not really important. The search box is now in a sensible place, in the header. And they’ve done plenty of other useful things with the right-hand column.
Score: 8/10

8. Only when you’re on the Home page (twitter.com) can you easily see who you’re signed in as
Fixed. This is now in a sensible place, top right.
Score: 10/10

9. Twitter.com does a terrible job of showing you recent activity.
Still terrible. Still no alert for an @Mention. Not even a new DM is highlighted in any way.
Score: 0/10

10. Inability to Search inside your own Twitter stream, or inside someone else’s.
No change here, by the looks of it.
Score: 0/10

11. When you look at your, or someone else’s Profile, Twitter could display much, much more useful information.
Besides room for more personal information, I was hoping for statistics and analysis, such as tweet frequency and tweet type ratios, frequently-used hashtags, time on Twitter, etc. No change here either.
Score: 0/10

12. The “More” page-down experience is atrocious.
OK, AJAX (sort of) to the rescue. The new “infinite scroll” is very impressive and (in my tests) super-fast. However, it still does not allow me to easily make big leaps into the past, the way paging links would’ve. When you navigate away, and click Back, you’re also not (instantly) back to where you were, although it tries to do so and catches up after a few seconds.
Score: 6/10

13. The display of user lists is bad, both in the mini form (avatars in the right-hand column) and the full page listings.
This is also improved by the AJAX infinite scroll, and the layout is improved, but still no sorting, or searching for a name within a list of contacts. Also, the experience of mousing over mini-avatars to see names hasn’t been improved.
Score: 3/10

14. Automatically hyperlink a Tweet ID.
Unsurprisingly, no. (I’d be curious to know whether this is feasible, incidentally.)
Score: 0/10

So, in total, of my 14 criticisms, 7 were completely or mostly corrected, and 6 only imperfectly or not at all. One still needs further testing.

My analysis above looks only at the points I raised in my previous post, and does not take into account the smart new layout and many new features, some of them quite wonderful. I especially like the photos, videos and conversations in the new right-hand pane. (Although the pane often seems to be filled up with not-really-relevant stuff just because there’s room.) I want to experience the beta a bit longer to get a feel for everything that’s new.

Why is Twitter.com so badly designed?

Since I’m always complaining about Twitter.com, it’s just fair for me to list what I think is wrong with it and how it should be improved. Please note: I’m only criticising the Twitter.com website, not the service itself or any other Twitter client.

1. Twitter Profile page highlighting top navigationHow is “Profile” a suitable label for your Twitter stream? Can you imagine someone talking about their “profile” on Twitter, when they mean their updates? Making it even more confusing, this menu item is not even highlighted after it’s selected.

2. For that matter, why is there not a decent Profile page? (In the normal social networking sense of the word.) Tweets are restricted to 140 characters for a good reason; there is no reason to have only a 140 character bio and a single link.

3. Why on earth is there only a Tweet box at the top of the Home page? At the very least, there should be one on the Profile page. Really, you should be able to tweet from any page on Twitter.com.

4. As soon as you scroll down, you lose the Tweet box. So if an update lower down on the page spurred you to write, or if you click Reply, bad luck: you have to lose it from sight in order to write.

5. Comparison of right-hand navigation on Twitter Home and Profile pages respectivelyLook at the inconsistency in the right-hand navigation between the Home page and the Profile page. Home has @replies, DMs, Favorites and Retweets (itself a disaster). Profile has Tweets and Favorites only. Is there any reason for them to be different on your own Profile? (On someone else’s Profile it will be Tweets and Favorites only, of course.) I’ve often been confused at the disappearance of, say, @replies, only to realise it’s because I’m on the Profile, rather than the Home page.

6. The Home right-hand navigation area is also polluted by, essentially, promotions, such as “Who to follow” and “Trending”. They are not visually distinguished from real navigation.

7. The right-hand navigation could generally be a lot better. Why not have the hashtags you’ve used, since they’re effectively a way of categorising your tweets? Why not display the total numbers of @replies, Favorites and Retweets (as for DMs)? The Search box is in a silly place, and being directly below your navigation links, implies it’s searching your tweets, not all tweets.

8. Only when you’re on the Home page (twitter.com) can you easily see who you’re signed in as (badge at top right). On the Profile page, only the “That’s you!” under the photo at the top tells you. On other pages, nothing tells you. In a household with multiple Twitter accounts, this is rather frustrating.

9. Twitter.com does a terrible job of showing you recent activity. When you log in to Twitter, you expect bright highlights alerting you of new @replies, if one of your tweets was favorited or retweeted, or if you have new DMs. When done right, this really boosts addictiveness: Facebook and Flickr are just two examples.

10. The inability to Search inside your own Twitter stream, or inside someone else’s, is simply crippling. I assume that this is a technical limitation due to Twitter’s scale, but it should be a top priority. If Google had bought Twitter, I imagine this would’ve been the first thing they’d fix.

11. When you look at someone else’s Profile, Twitter could display much, much more useful information. (Having a real Profile page would help.) Basically, the sort of information that 3rd-party services like MrTweet offers: their tweet frequency, how long they’ve been on Twitter, the nature of their Twitter usage (ratio of posts with links, ratio of posts that are @replies, recurrent hashtags, etc.)

On your own Profile, you’d want to see statistics. Number of views of your stream, number of favourites and retweets, ideally graphed over time (like Flickr does it).

12. The “More” page-down experience is atrocious. It’s slow, you can’t skip by more than a page at a time, so if you want to go back a few days it takes forever. And if you navigate away, and click Back, the entire stream is gone again. Presumably this is another technical limitation.

Ideally, I want actual paging links like on Flickr or Vimeo. (Only for a single person’s stream, not the firehose.)

13. The display of user lists is bad, both in the mini form (avatars in the right-hand column) and the full page listings.

On the mini version, relying on tooltips or the browser status bar to read a person’s name is very user-unfriendly. You want to scrub your mouse over the avatars and easily read the names as you do so.

On the full-page versions, where is the paging? Like the Twitter stream, there’s only a “Next” link. How are you expected to navigate through more than a 100 people? They are in no discernable order, and there’s no ability to order them by username, first or last name, or search within the list (like Facebook).

14. An obvious, if geeky, enhancement: automatically hyperlink a Tweet ID. Currently if you want to link to another tweet, you have to use an URL shortener. (A tweet ID is 9 characters shorter than a bit.ly link.)

And here are just some shortcomings that were fixed scandalously late:

  • Native URL shortening
  • The Follow button used to be a gear for ages
  • Emails alerting you to new followers contained no useful info (the follower’s bio was just added a few weeks ago)

I understand Twitter is focusing on growth, and many shortcomings are unavoidable results of its scale (such as Search and proper paging), but they cannot afford to lose sight of the user experience. But mostly I can’t understand how the remarkably talented UX folks they’ve been hiring — people like Kevin Cheng and Doug Bowman — can allow these problems to persist for so long. (While simultaneously rolling out unimpressive features like the infernal hovercards and “Who to follow”.)

Perhaps Twitter’s attitude is to focus on the API, and leave the user experience to third-party services and Twitter clients. That would be a pity: it’s a jungle out there if you’re looking for decent Twitter services. For every decent 3rd-party service or client there’s a plethora of ones that are ramshackle, spammy or downright malware. I also find this explanation implausible, given their investment in UX design talent.

Scaffolding template tags for Django forms

We love Django here at Isotoma, and we love using Django’s awesome form classes to generate self-generating, self-validating, [X]HTML forms.

However, in practically every new Django project I find myself doing the same thing over and over again (and I know others do too): breaking the display of a Django form instance up into individual fields, with appropriate mark-up wrappers.

Effectively I keep recreating the output of BaseForm.as_p/as_ul/as_table with template tags and mark-up.

For example, outputting a login form, rather than doing:

{{ form.as_p }}

We would do:

<p>
{% if form.username.errors %}
  {% for error in form.username.errors %}
    {{ error }}
  {% endfor %}
{% endif %}
{{ form.username.label }} {{ form.username }}
</p>
<p>
{% if form.password.errors %}
  {% for error in form.password.errors %}
    {{ error }}
  {% endfor %}
{% endif %}
{{ form.password.label }} {{ form.password }}
</p>

Why would you want to do this? There are several reasons, but generally it’s to apply custom mark-up to a particular element (notice I said mark-up, not styling, that can be done with the generated field IDs), as well as completely customising the output of the form (using <div>‘s instead etc.), and also because some designers tend to prefer this way of looking at a template.

“But”, you might say, “Django already creates all this for us with the handy as_p/as_ul/as_table methods, can you just take the ouput from that?”
Well, yes, in fact on a project a couple of weeks ago that’s exactly what I did, outputting as_p in a template, and then editing the source chucked out in a browser.
Which gave me the idea to create a simple little tool to do this for me, but with the Django template tags for dynamically outputting the field labels and fields themselves.

I created django-form-scaffold to do just this, and now I can do this from a Python shell:

>>> from dfs import scaffold
>>> from MyProject.MyApp.forms import MyForm
>>> form = MyForm()
>>> # We can pass either an instance of our form class
>>> # or the class itself, but better to pass an instance.
>>> print scaffold.as_p(form)

{% if form.email.errors %}{% for error in form.email.errors %}
{{ error }}{% endfor %}{% endif %}
<p>{{ form.email.label }} {{ form.email }}</p>
{% if form.password1.errors %}{% for error in form.password1.errors %}
{{ error }}{% endfor %}{% endif %}
<p&gtl{{ form.password1.label }} {{ form.password1 }}</p>
{% if form.password2.errors %}{% for error in form.password2.errors %}
{{ error }}{% endfor %}{% endif %}
<p>{{ form.password2.label }} {{ form.password2 }}</p>

Copy and paste this into a template, tweak, and Robert’s your mother’s brother.

As well as as_p(), the dfs.scaffold module also has the equivalent functions as_ul(), as_table, and an extra as_div() function.

Annoying CSS3 Baseline Alignment Problem in Firefox

CSS3 transform enables the rotation of elements including HTML text. If you intend to use it you should be aware that Firefox 3.6.8 and below has very poor baseline alignment.

The heading is just about acceptable. Content text is not.

Firefox 3.6.8:

Webkit:

So be warned if you intend to use CSS transform on text.

CSS3 Flexbox

CSS3 brings us a host of new features. Whilst most people will be familiar with gradients, text-shadow etc. I’d like to draw your attention to one of the lesser known modules in CSS3: flexbox. What’s interesting about Flexbox is that you can use it now if you use a Gecko or Webkit based browser (no IE sadly and Opera’s support isn’t there yet).

What does Flexbox bring us that we didn’t have before? Well it gives us a whole raft of new ways of controlling layout and flow. What we now achieve with floats we can do far more successfully and with more control with Flexbox.

I’ve never liked floats, they always seemed a hacky way of structuring HTML. Unfortunately with the meager positional CSS available, floats ended up being the best way of creating flexible fluid layouts. Let’s look at some examples.

display: box

To define an element as a flexbox we set the display to box.

.flexbox {
display: -moz-box;
display: -webkit-box;
display: box;
}

Example 1

box-orient

Notice that by default flexbox elements are aligned horizontally. If we want to change that to vertical:

.rule {
-moz-box-orient: vertical;
-webkit-box-orient: vertical;
box-orient: vertical;
}

Example 2

Box-orient is inherited i.e. child elements will also be aligned horizontally.

Example 3

Observant viewer will have noticed that Firefox (3.6.8) ignores the width and height setting on the child boxes, whilst Webkit respects it but overflows the content.

So if you want child elements to be aligned vertically, you’ll need to set the containing element to box-orient:vertical.

box-ordinal-group

We can reorder elements as we like, using box-ordinal-group.

#box-2 {
-moz-box-ordinal-group: 1;
-webkit-box-ordinal-group: 1;
box-ordinal-group: 1;
}

Example 4

The immediate application of this should be obvious, for the first time we have way of defining display order without reverting to all the hacks we’ve had to use up to now (floats, relative positioning, negative margins etc). Powerful stuff.

box-flex

Box-flex allow the content to expand to fill the available space.

.rule {
-moz-box-flex: 1;
-webkit-box-flex: 1;
box-flex: 1;
}

example 5

Notice once again the difference between Webkit and Gecko; whilst Webkit expands the main block to the browser window, Gecko expands to fit the content.

Box-flex fills space in proportion to the value given. Quoting the specs:

“All flex is relative. For example, a child with a box-flex of 2 is twice as flexible as a child with a box-flex of 1.”

To explain this better lets look at an example:

#box-1 {
box-flex: 1;
}
#box-2 {
box-flex: 2;
}
#box-3 {
box-flex: 7;
}

Example 6

For ease of calculation I’ve given the H1 child element a width of 50px, and the containing element a width of 600px. So the available space is:
600px – 150px = 450px

Working out the ratio is:
1 + 2 + 7 = 10

450px / 10 = 45px

this produces:

box 1 width: 50px + (45px * 1) = 95px
box 2 width: 50px + (45px * 2) = 120px
box 3 width: 50px + (45px * 7 ) = 365px

Further more you can set relative spacing whilst having fixed width paddings and margins. If you tried that with floats you’d either have to use percentage margins or have boxes bouncing all over the place.

box-align

Box-align works in the opposite direction to the orient setting. i.e. if the element is set to horizontal, box-align controls alignment in the vertical and vica versa.

#main {
-moz-box-align: stretch;
-webkit-box-align: stretch;
box-align: stretch;
}
#box-1 {
-moz-box-align: start;
-webkit-box-align: start;
box-align: start;
}
#box-2 {
-moz-box-align: center;
-webkit-box-align: center;
box-align: center;
}
#box-3 {
-moz-box-align: end;
-webkit-box-align: end;
box-align: end;
}

example 7

box-direction

This reverses the order of elements.

#main {
-moz-box-direction: reverse;
-webkit-box-direction: reverse;
box-direction: reverse;
}

Example 8

Notice the inconsistency between Gecko and Webkit again. Gecko aligns the right hand column, whilst Webkit aligns to left but reorders the elements.

box-pack

Box-pack controls alignment in the direction set by orient.

#main {
-moz-box-pack: justify;
-webkit-box-pack: justify;
box-pack: justify;
}
#box-1 {
-moz-box-pack: center;
-webkit-box-pack: center;
box-pack: center;
}
#box-2 {
-moz-box-pack: start;
-webkit-box-pack: start;
box-pack: start;
}
#box-3 {
-moz-box-pack: end;
-webkit-box-pack: end;
box-pack: end;
}

example 9

Justify is a nice attribute, equally spacing elements within the parent container.

Final Example

example 10

The classic 3 column layout. Notice the navigation appearing first, also note that the background colour extends to the full height of all boxes.

Sadly flexbox is not supported by IE and looks unlikely to be supported by IE9. So it will be a while before these techniques become mainstream.