How Twitter Integrates New Architecture

I've been experimenting with the Cassandra datastore lately and came across an interesting interview with Ryan King discussing Twitter's use of Cassandra.  I thought this tidbit was worth sharing even if you don't care about Cassandra:

A philosophical note here — our process for rolling out new major infrastructure can be summed up as “integrate first, then iterate”. We try to get new systems integrated into the application code base as early in their development as possible (but likely only activated for a small number of people). This allows us to iterate on many fronts in parallel: design, engineering, operations, etc.

He also describes in detail how they used that philosophy to migrate Twitter status updates (the toughest part of their app to scale) to Cassandra.  This also reminded me of Flickr's Feature Flags ... all of which can be used to support continuous deployment, something I've become very interested in after encountering it in the lean startup literature.

I'm planning to writeup my findings on using Cassandra with Ruby sometime soon if there's interest.

Speaking at Lone Star Ruby Conference

I will be speaking at the Lone Star Ruby Conference in September about how we use Ruby to deploy, monitor, and manage a cluster of servers running in the Amazon Web Services virtual cloud.   Below is a summary of what I'll be talking about.

In OtherInbox, almost every system administration task imaginable is carried out using Ruby, meaning we as developers can enjoy all of Ruby's expressive benefits and spend less time scripting the shell, writing cron tasks, or using other languages. Because we make fewer context switches from thinking in Ruby to thinking in other languages, we also reap a big productivity benefit.

Using Ruby throughout our cloud also means that porting the application to run in different production environments is a trivial task, because Ruby is the glue connecting the Ruby components together, thus all we require is a Ruby interpreter to deploy.

Two key Ruby technologies have matured in the previous 18 months which make it ideal for almost every layer of managing a cluster of servers:

• god.rb allows fine-grained process monitoring and daemon control (a la monit)
• rufus-scheduler enables Ruby-based scheduling (replacing cron, and providing a great facility for running daemons that must be executed on a recurring basis)

When combined with these Ruby workhorses, developers today can spend much more of their time writing Ruby code, and less time struggling with the vagaries of their production environment:

• Ruby standard library utilities (File, FileUtils, etc.)
• Rake
• Capistrano (for any kind of remote application, not just Rails)

The talk will also include a discussion of using several different AWS gems to make cloud computing simple, by illustrating the use of Amazon's S3 and SQS services to distribute asychnronous work and handle communication between servers.

(cross-posted from the OtherInbox blog)