Author Archives: Richard Mitchell

Plone Conference 2010: Day 1

So it’s the end of day one halfway through day two (this took a lot longer to write than I thought!) at this year’s Plone Conference and between the two of us, we’ve seen 13 talks so far. Whilst I can’t comment on the ones that Doug has been to, I’m going to summarize my experience of today’s talks. You might also want to take a look at Rick Hurst’s summary of day 1.

Keynote: The Future of Plone by Alexander Limi & Alan Runyan

Slides | Stream

TL;DR: Plone 4 is actually pretty awesome; Deliverance/XDV is now Diazo; we need to engage more with other communities and concentrate on design for Plone 5.

In an opening keynote not dissimilar to last year’s in Budapest, Alexander Limi set out once again to summarize the previous and the next year of Plone. The major differences being, Plone 4 is now out and the heavier focus on strategic goals rather than technological ones. Lots of congratulations for Plone 4 release going so smoothly. It’s a lot faster, better out of the box and easier to install. There should be a few more minor version increases over the next year or so and as of release 4.1, Plone 3 will no-longer be supported. Looking forward to Plone 5 (which Hanno Schlichting hesitantly put a date of March 2011 on) a big deal was made of the 3 “D”s of Plone: Dexterity, Deco and Diazo; Diazo being the new, back-of-a-cigarette-packet name for XDV / Deliverance in Plone 5. Whilst Alex focussed on what will be new in Plone 5, Alan spoke of the need to kill off things that are no-longer used or are replaced by new technologies, one example being writing restricted Python through the web:

“Who uses editing Restricted Python through the web?”

The sound of a room full of people putting no hands up.

Laughter.

One thing that particularly threw me as a new developer was the number of different ways to do things and to a certain degree, I think a limiting of these options is a good thing.

The need to make Plone more appealing to the decision makers was also expressed. Alex believes that over the next few years, designers will be making the choice of CMS to use (a company goes to a designer for a website, they come up with a great design, then have to pick the CMS onto which that design should be applied by developers or a separate company). Other places to focus on getting the Plone word out there: other conferences, local user groups and getting Plone available to people using cheaper hosting solutions, such as getting a cPanel installer working, a stark contrast to last year saying “we’re not really in the same market as Joomla and Drupal and WordPress [...] it will always be easier to pay $5 and get a PHP host somewhere; that’s not something we can fight”.

Mad About Mobile by Mikko Ohtamaa

Slides | Stream

TL;DR: Your clients want mobile sites, if they don’t you should persuade them. WebAndMobile from mFabrik will help you solve a lot of common problems and add a lot of cool features so you can deliver.

Mikko opened by saying something along the lines of, “Your clients want mobile sites, if they don’t you should persuade them” and I couldn’t agree more. Not only will a mobile site increase the number of users you can reach (or the times at which they can use your website), it can also add some really cool features (Mikko lists geolocation, touch and multi-touch events & phone interactions like click to call & contact card download). Furthermore it’s a good way to provide a minimal version of your site to users who perhaps have problems with their sight or mobility.

He touched on the different levels of mobilization that are achievable starting with custom CSS and going all the way up to writing a completely new mobile app.

The Plone extension route provides a nice middle ground that will get you most of what you probably want from a mobile version of your Plone site (mobile browser detection, image resizing and serving different content for mobile platforms). Mikko has been working on WebAndMobile for the past year and a half, it actually has documentation and there are plans to merge in common components of Infrae’s
mobi.*

group of packages. It is currently live on about 6 Plone sites and has some crossover capability for other frameworks such as Django. It:

  • Will automatically redirect to a mobile site based on information in the HTTP request and can scale the site to the features of the user’s platform;
  • Provides support for theming different platforms with different capabilities;
  • Allows integration of mobile-features such as opening a waypoint in a smartphone’s maps or navigation application;
  • Provides the option of serving alternate or no content for content type fields and integrates seamlessly into the existing editor interface.

Neat. For me it looks like the best way to get most of the way towards adding a mobile version of your Plone site. Whilst it won’t be a free-ride, it’s a hell of a good step in the right direction.

High Performance Sites Made Easy by Matthew Wilkes

Slides | Stream

TL;DR: Everything you thought before about optimization is probably wrong. You need to use the catalog less, work off real-world numbers rather than profiling and cache the hell out of everything you can, no matter how insignificant it may seem.

For me this was probably the most useful talk of the day. It really busted quite a few myths I believed in and exposed new techniques that I thought weren’t possible with Plone, namely Edge-Side Includes in Varnish. The talk started with a brief summary of what is and isn’t important, backed up by some interesting stats. For example, whilst not waking objects is fine as a rule of thumb, it’s actually slower to use the catalog to access data up to about 50 objects. There was a big focus on optimization based on real data and putting it off until you can get some. Matthew eschews profiling in favour of analysis of page load time and server logs of normal site usage. One thing which will be of particular use is the tool Jarn have written for analyzing HAProxy logs.

Catalog tips included removing as many indexes as possible, generating better brains using adapters for the
ICatalogBrain

interface and being particularly careful with ZODB cache size: walking the catalog may in fact clear the ZODB cache given the huge number of tiny objects the catalog uses.

Quick wins for caching can be achieved by implementing ESIs (particularly implementing them from
BrowserView __call__

methods so they are more flexible and can be disabled quickly in case of emergency) and avoiding status messages (not only does they make a page not cacheable by Varnish, but Varnish caches that it isn’t cacheable for 3 mins!).

Finally he makes the excellent point that it can be worth haggling with the client over arbitrary decisions they may have made about things such as list size, pagination, long form-based processes & workflow, all of which can inhibit caching.

Lose weight now, ask me how! by Roché Compaan

Slides | Stream

TL;DR: The catalogs are massive bottlenecks; we rewrote some common parts of Plone that overuse them.

Continuing in a similar vein to Matthew Wilke’s talk, but taking a more extreme approach, Roché Compaan almost seems like he is out to destroy the catalogs, and not without justification. He began by asking the question, “can Plone scale to 10 million objects?” Starting at the lowest level, he produced some benchmarks of the ZODB using
collective.zodbbench

in 2007, compared them to similar operations for Postgres and found that ZODB performs extremely well, holding steady at about 250 writes per second achieved on a sample database of 10m objects.

When it came to analyzing Plone (using
ZopeProfiler

, 
experimental.catalogqueryplan

and 
ZODB.FileStorage.fsdump

) he found some pretty horrifying stats about the catalog. For example, 90% of a Data.fs for a database of 10,000 content objects was dedicated to objects supporting the catalogs. Unfortunately he didn’t seem to have any stats on how the catalog affects speed.

upfront.diet

is a suite of products that offer replacements for parts of Plone that either cut out catalogs entirely or drastically reduce their usage. The most useful of these seem to be the upfront.simplereferencefield and
upfront.catalogblacklist

.
upfront.catalogblacklist

seems like something that should have been offered in Plone to begin with; it allows the developer to limit the indexing of specific content types through genericsetup.
upfront.simplereferencefield

is a replacement for the Archetypes reference field that doesn’t use the UID catalog. According to his stats, importing 50k objects using the archetypes reference field took 50 mins, while importing the same data set using the
upfront.simplereferencefield

took 5 mins. Impressive results, but this slash and burn approach to catalog usage leaves me uneasy, particularly when it’s something that is under consideration for Plone 5 anyway.

Others

Other talks I went to were Giving light to dark corners of z3c.form by Rok Garbas which was a good, quick intro to what seems to be the most flexible of all the Plone form frameworks; Laying Pipe with Transmogrifier by Clayton Parker which was a brief summary of the features of
collective.transmogrifier

which seems like a very powerful tool for importing data from various sources, but as Rick Hurst says, probably a bit heavyweight for what most people will want; and Tools and techniques for a successful Plone project by Martin Aspeli in which he shared a sneak preview of some of the tools and techniques he will be recommending in the new Plone 4 edition of his book.

Finally, at the end of day 1 was the guest keynote from Richard Noble, a very entertaining affair on the spirit of engineering and how teamwork matters far more than any amount of funding you might have.

An excellent start to the conference!

Generating sample unicode values for testing

When writing tests, there’s one thing we’re now sure to do because it’s caught us out so many times before: use unicode everywhere, particularly in Python.

Often, people will just go to Wikipedia and paste bits of unicode chosen at random into sample values, but that doesn’t always make for good readability when the tests you forgot to comment break two months later and you have to revisit them.

I’ve found a simple way to generate test unicode values that make sense, is to use an upside-down text generator, or some other l33t text transformer which produces unicode. Using text detailing whatever the sample value is supposed to represent, it’s still pretty legible at a glance and you’ll hopefully flag up those pesky
UnicodeDecode

errors quicker.

There’s a handy list of different text transformations and websites that will perform them for you on Wikipedia.

Javascript localization within Plone

For a while now, I’ve been trying to think of the best way to get localized strings into Javascript under Plone. Many existing packages and parts of Plone use one Javascript file per language, containing all translations. Whilst this is effective, it doesn’t “feel right” having all your translations distributed to the user on each page load, much less having them drawn from two different sources. Worse still (in my mind) is having translations dumped out into hidden HTML elements and then recalled by Javascript using
getElementById

or some such.

The solution I’ve eventually settled on is one that combines AJAX and client-side caching in cookies to minimize page load time. The implementation consists of a single view, a Javascript file and a bunch of XML & ZCML to tie it all together.

The powers that be here at Isotoma, have kindly agreed to let me open source this under the Apache 2.0 license.

Full documentation can be found for the code can be found in the
README.txt

, however I’ve included a quickstart guide below.

Continue reading →

Beginning development with Plone 4 & Dexterity

Over the past few days, I’ve been tinkering with the latest alphas of Plone 4, particularly with an eye to trying out Dexterity on the latest version.

I started out, as many people will, by downloading the unified installer which will install Python 2.6, Zope 2.12 and the Plone 4.0 alpha for you. After a few teething problems with multiple versions of Python on my Hardy host, I had my Plone install up and running.

First impressions among myself and my colleagues here at Isotoma were that firstly, it was a heck of a lot faster than its predecessor. In fact, John Stahl recently blogged that Plone 4 is potentially three times faster than Drupal, Joomla and WordPress. The other main, marked difference was the default theme, which is a lot slicker, though in my own opinion with its blocks of bright colours and rounded corners, a little too overtly “Web 2.0” (insert air-quotes here).

My next stop was Martin Aspeli’s Dexterity developer manual which whilst up-to-date for the current stable release of Plone, required some tweaking to get going with Plone 4.

The unified installer, by default, makes use of several config files for buildout, which keeps a lot of the core settings in separate files (
base.cfg

&
versions.cfg

). I hear that roadrunner is almost ready for Plone 4, but it’ll be a little while before we’re getting it without checking out the source so that had to be chopped. The
extends

entry for Dexterity also required updating to the latest alpha.

Otherwise, things went very straightforwardly. My
buildout.cfg

for use with the unified installer can be found below the fold.
Continue reading →

Useful Plone template debugging functions

As Plone developers, a lot of the problems we have when writing code and templates are only revealed with cryptic, sometimes misleading error messages from somewhere way down the stack from their underlying cause, if at all. When an error is raised, by some template rendering, Zope does provide some useful traceback information specifying the template with line numbers and expressions and whatnot. But why shouldn’t we be able to access this information without raising an error? For example, to diagnose security or redirection problems that aren’t necessarily obvious even with extra logging & verbose security enabled.

The functions provided below allow the developer to gather this kind of feedback and output wherever he or she wishes, without having to provide any arguments that might not always be easy to get from the current part of source code. They work under Zope 2.7 but are untested under other versions. If you do try them out on other versions, please report back in the comments if they do work!

import sys
from Products.CMFCore.utils import getToolByName
from zope.tales.tales import TALESTracebackSupplement

def get_current_template_position():  
    """ If called from a stack frame which has been called from template evaluation, 
        returns a tuple of template filename, line number, column number and 
        TALES expression closest in the stack to the caller. Otherwise, returns None. 
    """  
    i = 0  
    curframe = sys._getframe(i)  
    while True:  
        locals = curframe.f_locals  
        globals = curframe.f_globals  
        if '__traceback_supplement__' in locals:  
            # Use the supplement defined in the function.  
            tbs = locals.get('__traceback_supplement__')  
        elif '__traceback_supplement__' in globals:  
            # Use the supplement defined in the module.  
            # This is used by Scripts (Python).  
            tbs = globals.get('__traceback_supplement__')  
        else:  
            tbs = None  
        if tbs is not None:  
            factory = tbs[0]  
            args = tbs[1:]  
            try:  
                supp = factory(*args)  
            except:  
                continue  
            if type(supp) is TALESTracebackSupplement:  
                return (supp.context, supp.source_url, supp.line, supp.column, supp.expression)  
  
        i=i+1  
        try:  
            curframe = sys._getframe(i)  
            if curframe is None:  
                return None  
        except:  
            return None  
  
def dump_current_template_position(context=None, return_string=False):  
    """ When called, attempts to print to the console the URL of the current request, the 
        authenticated user, the currently executing template file, the line and column 
        currently being evaluated in the file and the expression being evaluated. 
 
        Will not print if called from a stack frame which has been called from template 
        evaluation. May not print if called from a .cpy or .vpy file, depending on 
        permissions to 'print'.
        
        Wherever possible, this function should be called with the 'context' arguement
        specified.
 
        If the optional argument 'return_string' is set to True, the function returns the 
        message that would be output, rather than printing. 
    """  
    tpos = get_current_template_position()  
    if tpos is not None:  
        (ctx, template, line, col, expr) = tpos  
        url = 'Unknown'
        if context is not None:
            try:  
                request = hasattr(context, 'request') and context.request or context.REQUEST  
                url = request.get('ACTUAL_URL')  
            except AttributeError:
                pass
        if url == 'Unknown':
            try:  
                request = hasattr(ctx, 'request') and ctx.request or ctx.REQUEST  
                url = request.get('ACTUAL_URL')  
            except AttributeError:
                pass
                
        member = 'Unknown'
        if context is not None:
            try:
                mtool = getToolByName(context, 'portal_membership')
                member = mtool.getAuthenticatedMember()  
            except AttributeError:
                pass
        if member == 'Unknown':
            try:
                mtool = getToolByName(ctx, 'portal_membership')
                member = mtool.getAuthenticatedMember()  
            except AttributeError:
                pass
            
        output = "\tURL: %s\n\tAuth'd as: %s\n\tFile: %s\n\tLine: %s\n\tColumn: %s\n\tExpression: %s" % (url, member, template, line, col, expr)  
        if return_string:  
            return output  
        print output

This may also be called from templates, provided the template has sufficient permissions to call the module it lies in:

<tal:block tal:define="dummy python:modules['myproject.app.utils'].dump_current_template_position(context)" />

Or, as it is mostly used, from code called by templates, simply by importing the function(s) as necessary and calling them with options of your choice. If calling from
.cpy

or
.vpy

files, the print command may not work properly, so the
dump_current_template_position

function may be called with the optional argument
return_string

set to
True

and then the result may be logged or printed using alternate methods.

Get a content type class by name in Zope

I recently had the problem of finding out if a content type in Plone implemented a certain interface, but only given the name of the content type rather than an instantiated object or its class. Whilst it’s easy to create an instance given a content type name, using the Portal Type Tool, creating an instance to test and then deleting it felt wrong. Unfortunately the Types Tool seems to provide no easy way of determining the class of a given type.

The solution comes from a combination of the Portal Types Tool’s type information and the Archetypes Tool’s information about the implementation:

from Products.CMFCore.utils import getToolByName

def has_interface(portal, typeName, interface):
    """ Portal is the root portal object, typeName is a
        string such as 'MyDocumentType' and interface
        is the interface class to be tested against. """

    pt = getToolByName(portal, 'portal_types')
    at = getToolByName(portal, 'archetype_tool')
    typeinfo = pt.get(typeName)
    if typeinfo:
        package = typeinfo.product
        type = at.lookupType(package, typeName)
        if type:
            klass = type['klass']
            return interface.isImplementedByInstancesOf(klass)
    return False

LinguaPlone and redirection

One of our current projects working with Plone requires multilingual content using the LinguaPlone product. One of the features that has been lost since LinguaPlone was first released was the redirection to content in the user’s preferred language when given a hard link to content in another language. As mentioned in this rejected bug report from 2006, the desired functionality is that there should be no redirection however this approach doesn’t leave any room for flexibility in the name of usability.

One suggested solution in the bug report is to re-add old code from previous versions of LinguaPlone but what if one doesn’t want redirection on all translatable views?

The solution we came up with was to override the
__call__

method of the view using our own base class, which could be inherited by views which we wished to redirect:

from Products.Five.browser import BrowserView
from Products.CMFCore.utils import getToolByName

class MyBaseView (BrowserView):

    def __call__(self, *args, **kwargs):
        
        if self.context == self.context.request.other['PARENTS'][0]:
            # Don't try to redirect if we're not in the context of the object originally called.
            # Otherwise, if child objects' views are called the whole view may be redirected
            # to that child in exceptional circumstances.
            
            try:
                # Import may fail if LinguaPlone not enabled.
                from Products.LinguaPlone.interfaces import ITranslatable
                
                redirect = self.context.request.get('i18nredirect')
                if (not redirect or redirect.lower() not in ['no','false']) \
                        and ITranslatable.providedBy(self.context):
                    
                    language_tool = getToolByName(self.context, 'portal_languages', None)
                    
                    # Check current selected language, language of object sought & available translations
                    if language_tool is not None:
                        preferred_lang = language_tool.getPreferredLanguage()
                        if self.context.getLanguage() != preferred_lang and self.context.hasTranslation(preferred_lang):
                            translation = self.context.getTranslation(preferred_lang)
                            return self.context.request.RESPONSE.redirect(translation.absolute_url_path())
            except ImportError:
                if 'LinguaPlone' not in e.message:
                    raise e
        
        # Resume normal behaviour
        return super(MyBaseView,self).__call__(args, kwargs)

This allows us add the redirect functionality arbitrarily if we wish, and also retains a method of hard-linking using the
i18nredirect

GET variable. This functionality could of course be reversed, leaving the redirection as the special case rather than the default.