Staying fast and good when making enterprise software

My startup Staq builds enterprise software using the techniques I've learned from building consumer web apps. As we bring more customers on and our revenue increases, we have more to lose when there's a bug in our code, and there's pressure to revert to a more traditional development process where the developers have less control. Specifically we're debating whether to move away from our current fast-paced "manual continuous deployment" process (where we're pushing code throughout the day, every day, and getting immediate feedback) to a more traditional commit-review-QA-release cycle. I have pretty strong feelings against going in that direction that I wanted to document here.

I know from experience that a process like that will inevitably slow us down, which we can't afford during this phase of hyper-growth. It can produce a boring, toxic work environment in the long run. Code you wrote today doesn't end up getting in front of a user until next week, and you've already forgotten the context to explain why you wrote that code the way you did. When there's a bug, you have to start from scratch when figuring out the problem. Feedback gets delayed, if you get it at all. 

So how we stay fast while keeping the software quality high? How do we stay fast and good?

I intend to establish a core value at Staq where we never, ever stop deploying. We make software at such a high level of quality, and our cluster immune system is so awesome, that it's nearly impossible to kill the app. When defects occur, they are so obvious and so easy to correct, that users rarely notice the problem.

In the early days there may be times when we have to compromise on this value. When that happens, I want it to feel like an ugly compromise. I want the situation to become a sign that we need to improve our systems to get back to the platonic ideal of Always Be Deploying. ("Coffee is for committers!")

Here's a partial list of strategies we came up with to make this work, most of them drawn from the lean startup movement:

• Work in small batches: smaller changes are less likely to cause a disruption, and are easier to pinpoint when they do cause problems.
• Don't take shortcuts: skipping tests and hacking things together ultimately add unacceptable risk and technical debt.
• Practice five whys/root cause analysis: don't make the same mistake twice; gradually develop a cluster immune system
• Expect 100% unit test code coverage, backed by appropriate integration and feature tests: catch problems as early as possible
• Setup a continuous integration server: we have too many modules for any one developer to test all of them at once, so this server will become a backstop, alerting us when there is a problem in one of our modules.
• Use Feature flags: while we are testing new features, hide them from users until they are fully baked.
• Build staging environments that make it easy for developers to double-check their work
• Since we're on Heroku we plan to use their awesome pipelines lab for this
• We can also fork our database to create a sandbox for features that depend on database changes
• This should never be construed as a required step before code can be released to production
• Code reviews
• Two times a day I review every commit in our system, really helps spot issues before they become problems
• Build informative, actionable alerting systems

What else should we add to the list?

Fetching your Heroku production database schema into local storage

Updated Aug 23rd, 2014: the old commands I posted here did not work with Postgres 9.3's pg_dump. New commands listed below, including schema migrations dump.

I noticed recently that the structure.sql checked-in to the repository for Staq's main Rails app had fallen slightly out of sync with what was running on our Heroku production database. After a little digging, I came up with the following command. Hopefully this saves you a bit of time - it just mimics what rake db:structure:dump would do for you, but connected to your production database. Will also update the list of schema migrations:

pg_dump -v -i -s -x -O -f db/structure.sql --exclude-table=****** --exclude-table=***** -d DATABASE_NAME -U USER_NAME -h HOSTNAME && pg_dump -v -a -t schema_migrations -x -O -d DATABASE_NAME -U USER_NAME -h HOST_NAME >> db/structure.sql

We want to pay you to learn Ruby in Baltimore

Update 7/29/14: We are not accepting new applications at this time. When we do this again I will definitely post details here. Check out this summary of how things went!

Update 9/3/13: Pando Daily wrote an article about this, includes a few more details about our motivation

My startup Staq has a big technical challenge to solve, and we are assembling a team of people in Baltimore to tackle it. We're looking for people who have basic technical knowledge and want to become proficient in Ruby programming. This would be a temporary engagement lasting about 3 to 6 months.

The team will be responsible for building our core technology comprising hundreds of small API and web-scraping integration projects using tools like Mechanize, Typhoeus, and CasperJS. We built an internal DSL to make this code easy(-ish) to write, test, and debug, so that once you know what you're doing, most projects take a day or less. Except some of them can take days, so this work is not for the faint of heart! 

These projects are ideal work for an entry-level programmer: they're pretty fun and don't usually require deep knowledge of Ruby. They are very important to the company and core to how we grow the business - which means your work has a great deal of meaning, unlike the busy work many programmers are forced to do. 

We are experiencing rapid growth. In order to support our customers, we need to temporarily accelerate the number of integrations we do per week. Since it's temp work suitable for new programmers, we'd like to use this as a training ground for new Ruby developers in Maryland. We want people to start on the integration team as hourly employees, with the goal of either graduating to a salaried position within Staq or moving on to work at another software company in the area with a reference from me.

We can pay you a decent hourly wage, but you would also get unlimited mentoring and training from me and other Staq programmers. I personally consider talent development one of my main jobs as a CTO, and I will work hard to help you reach you career goals, whatever they are.

General prerequisites: We need people who can solve problems and thrive in a startup environment of low supervision, high accountability, max flexibility, who want to be coached. It can be challenging to figure out the best way to scrape some of these sites, so you'll need to be dogged and persistent. We also want people who are good at communicating and keeping the rest of the team updated. You should have a track record as a volunteer, an organizer, a professional, a parent, a student, a veteran, etc. Bonus: you have a reference from someone I know. Rummage my connections on LinkedIn!

Technical prerequisites: You must be able to prove that you have basic knowledge of programming and working at the command line. You know what variables and methods are, you've written a few small programs, and you understand the basic principals of object-oriented programming. You have a graduation certificate from an online programming course, proof of attendance at a RailsGirls event, a college transcript, or you've worked through all of the exercises in Learn Ruby The Hard Way. Or you have a body of work on Github I can check out. Bonus: you have used any of these before: RVM, Rspec, git, homebrew, or Bundler.

Education prerequisites: None! Some of the best programmers I know are not college graduates, or even high school graduates! Please don't let that hold you back from applying. I didn't go to school for this stuff; I am entirely an audodidact.

Geographic prerequisites: Remote work is possible later on but at the beginning you must be able to commute to our office in Hampden.

Hardware prerequisites: We're a small company and unfortunately, right now we can't afford to buy you a development machine. Most of us us use OS X but that's not a requirement. You should plan to work in a Unix OS though, so if you're on Windows definitely get something like Virtualbox running a Linux variant.

Time commitment: We are looking to work with people from about 3 to 6 months. We can offer you up to 40 hours of work per week and need a minimum of 20 hours.

Interested? Please fill out an application.

A web product startup playbook

I'm presenting "A web product playbook" at BohConf 2013 today. It's a distillation of what I'm considered the most important lessons I've learned during the past seven years of succeeding and failing as an entrepreneur. It's packed with links, and posted to Github! Hope you like it.

RailsGirlsBaltimore infographic

I really love this awesome infographic that Jess Gartner created for the recent RailsGirls Baltimore event, showing what we learned in organizing it and what the experience was like for everyone involved.

The best thing about it, for me, is that it's a list of "to-dos" for making inclusive events. I feel like there's already a lot of information about what not to-do (although sadly, that information still hasn't reached many members of the tech industry, as Jess' previous infographic shows) and it's nice to get some specific, tactical advice about how to convert good intentions into reality.

More ways to keep the American dream in America

I read Dave Troy's post Keeping the American Dream in America with interest. I agree with everything he says about needing to get aspiring entrepreneurs to focus on solving more important, technically-challenging problems instead of devising endless variations of incrementally-better consumer software.

There are other obstacles to achieving the kind of entrepreneurship Dave wants to see, besides the ones he mentioned. Those hard problems are going to end up being solved by older people not currently enmeshed in "McStartup" culture. Innovation in fields like healthcare, education, biotechnology, or government are likely to come from people working in those fields already. I'm thinking of people like Jess Gartner, a former schoolteacher who just received funding from Accelerate Baltimore to create an new product that will revolutionize the way we fund schools and track funding outcomes. That's not an idea that Dave's "Red Bull-drinking, Ramen-eating young programmers" could think of, unless they had worked as teachers.

To solve the "McStartup" problem, we need more entrepreneurs like Jess or my partner James Curran, people who have established a career and built up some work experience, who have firsthand knowledge of screwed-up situations that could be improved with new technology (using James hard-won expertise, we are attacking a tough problem in the online media industry).

Because these are also people who may have families, who may be the breadwinner, I have a theory that we could encourage more mid-career people to start or join serious startup companies by increasing our support for risk-taking behavior in the United States. Off of the top of my head, some ways to support risk-taking behavior would include:

• Better access to affordable, quality childcare: A big reason I can afford to take the risks I do is that we have a lot of nearby family help, and we were lucky to get our kids into an awesome, affordable preschool.
• More widely-available, affordable healthcare: I know someone who delayed starting his now-highly successful, real-world-problem-solving tech company because he needed to provide health coverage for his wife who had a pre-existing condition.
• Well-funded, well-managed public retirement system: Middle-aged people might be willing to take more risks if they felt they didn't have to save every penny in order to stave off penury in old age.
• Restored public support for higher education: Someone who graduated from college years ago with outstanding, crushing debt is probably not going to jump out of a comfortable job to start a tech company.

You get the idea. I realize these are all liberal causes, and I wonder if liberals have ever tried to position initiatives like these as supporting an innovative culture. Have there have been any studies done that would support my theory? If you take an entrepreneurial, self-starting culture like we have in the United States, and increase support for mid-career risk-taking, do you get more innovation and more new companies?

Ignite Baltimore Proposals and Grants

We're now accepting proposals to speak at Ignite Baltimore #12 on 3/28, and one of my co-organizers, Neal Shaffer, wrote a comprehensive explanation of the kinds of talks we're looking for. Please consider submitting your idea for consideration! Proposals are due 2/17/13.

We also use the ticket proceeds to give out Ignition Grants at every Ignite, aimed at funding a new projects in the city. This time we're giving out $4500 in grants. Grant applications are due 2/22/13. If you've got a worthy idea that needs funding, please apply!

Invitation to RailsGirls

I'm helping organize RailsGirls Baltimore on 3/2 to introduce computer programming to girls and women in the Baltimore area. No prior experience is required! It's a fun day of learning new skills in a small group with a coach helping you progress through a series of exercises.

Please help me spread the word as much as possible. We'd like to have as many women attend as possible!

Details are here:

http://railsgirls.com/baltimore

Hope to see you there!

Rspec matchers for Sidekiq

I've been writing a lot of specs lately to make sure my code properly enqueues Sidekiq jobs. I ended up duplicating a lot of code, and after reading the Rspec custom matcher documentation, I decided to DRY up the tests using some of my own matchers. The code is here in my "power tools" gem. Enjoy!

The tests end up looking like this:

  MyWorker.should have_queued_job(2)
  Collector.should have_queued_job_at(Time.new(2012,1,23,14,00),2)