Some of you may know Tim Bray. He’s been a major player in some important technologies of the present (XML) and the future (Atom). He also has a really good blog.
He’s posted a good summary of some of the big issues in software and systems architecture. These are some of the points that occupy anyone involved in longer-term technology strategy, and it’s sobering to see them listed together like that. These are very exciting times to be in technology – but it’s probably easier now than it has ever been to back the wrong horse.
A lot of these issues are ones that we struggle with here at Isotoma, and as Chief Software Architect it’s my job to try and anticipate some of these trends, for the benefit of our clients. This seems like a good opportunity to respond to Tim, and to show how we’re thinking about technology strategy.
(Apologies if I lapse at times into gobbledegook. Some of the things I’ll talk about are just plain technical. I’ll try and link them appropriately, so at least there’s some context.)
Up till not too damn long ago, for a big serious software project you could pick Java or .NET or, if you really liked pain, C++. Today you’d be nuts not to look seriously at PHP, Python, and Ruby. What’s the future mind-share of all these things? I have no idea, but that decision is being made collectively by the community right now.
He’s absolutely right, but obviously we’ve done rather more than look seriously. We’ve been a pretty much pure Python shop right from the outset. We use some PHP, when it makes sense, but Python is our clear choice, and it’s one we’re more than happy with. It’s a significant competitive advantage for us, in all sorts of ways.
Python has delivered for us in developer productivity, and on a larger scale it’s delivered in elegance – it scales very well as a developer language. Also, perhaps unlike Ruby, it scales very well in terms of performance, so I’m comfortable building very large systems in Python.
No, I don’t think relational databases are going away anytime soon. But I think that SQL’s brain-lock on the development community for the past couple of decades has been actively harmful, and I’m glad that it’s now OK to look at alternatives.
Will the non-relational alternatives carve out a piece of the market? I suspect so, but that decision is being made by the community, right now.
Brain-lock is right. It’s been the case for twenty years that for every single IT project your architect would open his toolbox and would pull out an RDBMS.
Relational Databases are still suited to a whole suite of applications and classes of problem. When you have highly structured data and very tight performance criteria they’re going to be a good choice for a long time to come. But for many of the problems they’ve been used to solve they are terminally ill-suited.
We’ve been using ZODB as part of Zope since 2004 (and I used it myself for several years before that). ZODB has some excellent characteristics for whole classes of problem that an RDBMS has problems with. It’s a lot more flexible, and it’s hierarchical nature provides a natural fit for web projects.
More recently we’ve been making heavy use of DB XML, Oracle’s Open Source XML database. This is a fantastic product, and it’s a much better model for most of the applications we build. A good example, oddly, would be Forkd which we built using a traditional RDBMS. If we had the experience of DB XML then that we have now then there’s no question that we’d use it for Forkd. Fitting recipes into a relational database is an exercise in ultimately pointless contortion.
I’m very confident XML databases are going to be huge.
CORBA is dead. DCOM is dead. WS-* is coughing its way down the slope to dusty death. REST, they say, is the way to go. Which I believe, actually. Still, there’s not much yet in tooling or best practices or received wisdom or blue-suit consultants or the other apparatus of a mainstream technology.
So what are they going to be teaching the kids, a few years hence, the right way is to build an application across a network full of heterogeneous technology? That’s being worked out by the community, right now.
The lack of tooling and blue-suit consultants (ignore my blue suit for the second) is, I think, a good thing. REST is not a technology stack, it’s an architectural style. It pares down the network programming model to fit the harsh realities of a stateless, highly concurrent, open system. We’re big fans of REST, and it’s a natural fit for how we work.
It’s not the whole story though, and there are a whole bunch of recurring problems in RESTful interfaces that are awaiting smart people to solve them. There’s some good work going on with URI Templates and PATCH, and of course Atom that I think are part of the solution yet.
Some relatively common orchestrations are horribly contorted in REST too, and it wouldn’t surprise me if here, to handle specific cases of lock acquisition and release and so forth we see some tooling.
Moore’s law is still holding, but the processors get wider not faster. Now that the best and the brightest have spent a decade building and debugging threading frameworks in Java and .NET, it’s increasingly starting to look like threading is a bad idea; don’t go there. I’ve personally changed my formerly-pro-threading position on this 180º since joining Sun four years ago.
We still haven’t figured out the right way for ordinary people to program many-core processors; check out the inconclusive results of my Wide Finder project last year. (By the way, I’ve now got an Internet-facing T2000 all of my own and will be re-launching Wide Finder as soon as I get some data staged on it; come one, come all).
And I can’t even repeat my crack about the right answer being worked out right now, because I’m not actually sure that anyone has a grip on it just yet. But we’re sure enough at an inflection point.
We’re a lot further down this particular inflection curve than most, I think. We make heavy use of Twisted, a single-threaded cooperatively multitasking network programming system that specifically addresses the threading problem.
I don’t think it’s the whole answer though, but nor is Erlang, which Tim championed in his Wide Finder project, with fascinating results.
Erlang has some marvellous attributes when it comes to large scale concurrent systems, and I’m very impressed with it. But adopting Erlang throws too much away, I think, losing the large-scale structural advantages of the Object Oriented approach that is pretty much the default for software architecture today.
Perhaps something like Stackless is the longer term solution here. An OO, message-passing, naturally distributed language using Python syntax and standard library but with some core functional changes (variables not being variable, for example) is the answer.
Or maybe even Mozart, which solves a lot of these problems too. It’s the current first-year MIT language [update: this is probably a lie, see comments], so expect to hear more of it in time.
Tim is right though, nobody really knows the answer here. All we know is that it certainly isn’t traditional multi-threaded programming, a la Java or C++.
Used to be, it was Java EE or Perl or ASP.NET. Now all of a sudden it’s PHP and then Rails and a bunch of other frameworks bubbling up over the horizon; not a month goes buy that I don’t see a bit of buzz over something that includes the term “Rails-like”.
It seems obvious to me that pretty soon there’s going to be a Rails++ that combines the good ideas from RoR with some others that will be obvious once we see them.
Also, that some of those “Rails-like” frameworks, even if they’re not a huge step forward, will get some real market share because they’ll have some combination of of minor advantages.
Once again, I can’t say it’s being worked out right now, because for right now I see a pretty uniform picture of Rails’ market share advancing steadily. It won’t last.
We use a couple of rails-like frameworks ourselves, Turbogears being the most obviously MVC. The big ideas in Rails, and similar frameworks, is the combination of MVC with an Object Relational layer. Since, as I’ve said, I don’t think the Relational stuff is needed at all, there’s an obvious first place where Rails and friends should look. Ditch the RDBMS.
Second, MVC maps pretty well to a lot of applications, and it’s a natural architectural style for a lot of people. MVC isn’t the only architectural style though, and it’s not necessarily the best fit for some though. The well-documented problems at Twitter, for example, I think just show a poor fit between MVC and Twitter’s fascinating (and well chosen) Jabber back-end. I know for certain, I’d not have used anything Rails-like for that.
I think it’s likely that the notional “Rails++” will probably not be MVC, and nor will it have an Object Relational layer. I think Rails, and it’s imitators, are just not suited long-term to the challenges of scale and distribution. That said, they clearly work well for a whole host of small and medium-sized projects right now.
Servers, they’re easy to understand. Blue-suited salesmen sell them to CIOs a few hundred thousand dollars’ worth at a time, they get loaded into data centers where they suck up too much power and HVAC.
Well, unless you’re gonna do your storage and compute and load-balancing and so on out in the cloud. Are you? The CIOs and data-center guys are wrestling this problem to the ground right now.
And as for software, used to be you shipped binaries on magnetic media and charged ’em a right-to-use license. Nope, nowadays it’s open-source and they download it for free and you charge them a support contract. Nope, that was last century; maybe the software’s all going to be out there in the cloud and you never download anything, just pay to use what’s there.
Personally, I don’t think any of those models are actually going to go away. But which works best where? The market’s working that out, right now.
Obviously we’re Open Source throughout, which takes us a long long way down this road already. We’ve got one secret squirrel project that’s successfully deployed using a massive Amazon EC2 back-end too, but I can’t say more about it.
Lets just say the massive economic advantages of the cloud are so conclusive that this is an obvious bet. Something like Google’s AppEngine is only a first step down this road, but it’s visionary and appropriate. And it’s in Python
As I wrote a couple of months ago: how long can the public and private sector IT management continue to go on ignoring the fact that in OS X and Ubuntu, there are not one but two alternatives to the Windows desktop that are more reliable, more secure, more efficient, and cheaper? More or less everybody now has a friend or relative that’s on Mac or Linux and is going to be wondering why their desktop can’t be that slick.
What’s going to happen? I don’t know, but it’s going to be dramatic once we get to the tipping point, and I think we’re approaching it right now.
We use Ubuntu throughout, on all our desktops and laptops. Well, nearly. The machine that runs Sage for accounting is Windows. And our front-end guys need Windows or OSX to run PhotoShop and Flash. But everything else? Ubuntu works really well, and saves us an absolute fortune.
Will It Always Be Like This?
You know, just maybe. Our mastery of the IT technologies is still in its brawny youth, with lots of low-hanging fruit to be snatched and big advances to be made. And these days, with the advent of blogs and unconferences and all those new communication channels, our thought leaders are busy chattering at each other about all these problems all the time, 24/7/365. The gap between the leading edge and technology that’s actually deployed in the enterprise is as wide as it’s ever been and to me, that feels like a recipe for permanent disruption. Cowabunga!
Our industry has the greatest community of practice that has existed, perhaps, in the history of mankind. Every profession has it’s conferences and papers and journals, but only in our part of IT is it normal to share and discuss all of our work, all of the time, even to the extent of giving away the very code we write.
I can’t see an end to this cycle of innovation yet – it’s just too damed valuable to everyone concerned. Cowabunga indeed