NYC Start Up CTO

I want to praise @SteveKuhn for giving fast and loose estimates.  He’s a busy guy @CrowdTwist and I can’t imagine he actually stands behind some of his estimates when he makes them based on the number of things he’s juggling in a given day.  But what he does is incredibly important and worth highlighting.

Often I’ll hear him say, “I think I can have that for you shortly after lunch.”  This is intriguing: what time is lunch and how much time does “shortly” mean?  Even with so much ambiguity, he’s still doing a few very important things for the people he works with and those that are dependent on him:

1. He’s created a forcing function for himself.  He’s made a commitment that he needs to honor.
2. He’s given the person waiting on him a rough sense of how much time they must wait before he delivers so they can go off and fill the time with other tasks.
3. He’s reinforcing that it’s important, as a team member, to respect other people’s time and give them some sense of what to expect.
4. When he can’t deliver he’s forced a touch point to communicate his status and a new deliverable.  This continues to help the people waiting on him plan their time more effectively.
5. He quickly time boxes his tasks.  This let’s him plan his own time better and forces him to finish tasks in the time he’s allotted.

Agile is not a software development methodology.  Being Agile, implementing Agile practices, requires cultural and personal changes that seem counter intuitive based on older software methodologies and even some conceptions that employees have about corporate culture.

Attempting to “implement Agile” as a process will fail.  It will fail, more likely than not, by creating what appears to be (to an astute observer) Agile waterfall.  Participants will feel compelled to work faster within the waterfall framework and even iterate over each of the steps of the process.  Ultimately, a team will feel more that Agile does not work or is unattainable.

If you haven’t read the Agile Manifesto or it’s supporting documentation you should:

Manifesto for Agile Software Development
Principles behind the Agile Manifesto

Here’s an excerpt:

    “Individuals and interactions over processes and tools
     Working software over comprehensive documentation
     Customer collaboration over contract negotiation
     Responding to change over following a plan”

When I read this, nothing sounds procedural.  These guidelines appear to be fundamental cultural traits that emphasize people-to-people (and decidely not people-to-computer) interactions.

Why?  Because using a computer is a very solitary activity and as such eliminates any opportunity for people to engage one another.  Here are some examples of what not being Agile looks like:

    “Can you tell me about feature XYZ?”
    “You can read about it on the Wiki.”

    “I think I found a bug.”
    “File a ticket in Jira.”

    “How does feature ABC work?”
    “I’ll write a document about it and email it to you.”

    “Can our software do DEF?”
    “Send me an email and I’ll get back to you with what we’re capable of.”

You cannot measure the loss from shutting down conversations by directing people to interact with you by first interacting with their computer.  The more this pervades day-to-day life the more quickly teams entrench themselves in building digital processes to replace personal interactions.

Over time this will devolve into spending time managing “the process” itself (and all of the sub-processes that each team has built) over building self-motivated teams that learn through constant practice how to excel at face-to-face communication.  Let’s face it, most of us sit within 20 feet of every other member of our team yet for many of us our preferred mechanisms of communication include email, Skype, wikis and requirements management software.

Nothing can require someone to change their personal beliefs more than this:

   ”Responding to change over following a plan”

Few people can exist without a plan and often find themselves agitated or terrified without one.  Traditional structures for blame and accountability fall away when your culture truly understands (particularly in software based start ups) that stuff changes all the time.  Building software involves far more unknowns and last minute changes than most people would like to believe. But it’s called “soft”ware because it can be molded and changed like clay. Hardware, on the hand, computers and phones and printers, are literally made of molded plastic and metal that cannot be changed without first destroying it.

Agile practices require Agile individuals, Agile physical spaces and a cultural desire to first get out from behind a computer, engage in face-to-face communications with your team before getting to work constantly course correcting, handling surprises and adapting to change.  The rewards, though, are far more meaningful than a computer telling you you successfully submitted the new documentation you wrote…all alone.

I have to think a lot about scale @CrowdTwist.  Investors and clients ask me all the time - “Does it scale?”  Yes, it does!  But I thought I’d shed some light on how I have to think about scale since my concerns go beyond whether or not servers can handle traffic if it increases.  So here’s what I’m constantly managing:

• Do our processes and tools scale we as add more members to the product development team?
• Does our quality assurance process scale such that quality remains as high as it is even if we’re producing more software faster?
• Does launching a new client instance of our software scale even though the product is highly configurable including the ability to customize any piece of text and translate it to 109 languages?
• Do our infrastructure automation systems scale as the number of servers increases?
• We’re required to manually QA several client configurations across an array of browsers and devices.  How do we continue to scale the QA process to handle this?
• How do we scale communication with Sales and Client Services as we add more features and introduce more functionality into the product?
• How do we scale communication to clients as we increase the number of releases of our software and the breadth of our product?
• We are getting a lot of RFP’s so I’m constantly thinking about how we scale documentation and communication with the people handling our responses.
• How do I scale?  I code, I manage, I go on sales calls, I architect solutions and manage infrastructure.  PFM?  No, I have an incredible team that makes all of this happen and makes something that is incredibly sophisticated and big scale look easy.

And yes, we can handle ALL of the data and ALL of the traffic.  Bring it!

While working as the CTO of CityRealty.com we received the documents below by mail.  It’s the “Nigerian scam” but modified slightly to suit a business.  Essentially someone is asking us to wire money directly to a Nigerian bank account as a loan so the writer can transfer in this case $30MM into their US bank account.  In this variation, though, they aren’t offering us a cut of the money for granting them the loan which I found unusual.

I also love that the writer prefers communication only by fax or email!  I found this document recently and thought I’d share.  Six pages follow:

After the Amazon Web Services (AWS) failure on Elastic Block Store (EBS) volumes in the us-east-1 availability zone (October 22, 2012), CrowdTwist suffered multiple component failures.  We lost four EBS volumes that were members of RAID and one server completely died.  These outages cost us no downtime.

We’ve spent a lot of time architecting our AWS infrastructure and working with very smart people from Amazon and Google to create systems that are resilient and built “according” to the way in which EC2 was intended to be used.  That paid off last week.

Recovering the server was easy: it was one of two servers configured for redundancy.  We launched a new instance with our custom Amazon Machine Image (AMI) and put the new server into rotation in the appropriate Elastic Load Balancers (ELB).

The EBS volumes that failed were all configured using md (software RAID) and were members of RAID 1 or RAID 10 groups.  These disks going missing did not affect the RAID, the server or the database at all.

We use Oracle 11g and have combined several EBS volumes into RAID configurations to boost the performance of reads and writes.  The Pythian Group manages our databases and helped us configure the disks properly to optimize performance.  Note, this was done before AWS started offering provisioned IOPs.

I’ll spare you the very low level technical details (but please direct message me on Twitter if you want them) but essentially the volumes in question were no longer visible at the OS level even though the AWS management console was seeing them as attached to the server they were originally mounted on.  We created matching volumes, mounted them on the server, and used mdadm to fail out the old disks and add the new ones.  The longest part of the operation was the recovery mdadm had to perform as a result of a new disk being introduced.

There is an interesting trend among my peers of moving out of AWS to managed or dedicated hosting.  The primary reason given is that AWS fails too often.  I don’t disagree that the failure rate is too high but we’ve managed to architect a solution that allows us to recover very quickly with no loss.

I’m proud of that and impressed with both the technology behind AWS and the people that have helped architect our stuff.

[If you’re curious, the AWS outage lasted 15 hours.  CrowdTwist was affected for one hour and one minute because we were able to fail over to redundant systems including a complete Oracle fail over to stand by.  We could have been up in under 20 minutes but took the time to make sure no data was lost.  None was!]

Here’s what not to do as a start up CTO: use a completely new set of technologies to build your product, stuff you’ve never used before or have little experience with.

Why?

You’re usually convinced that the idea you’re working on has a place in the market.  Therefore your number one goal should be to create value.  The more product you build that customers like and use, the more value you’re creating for the business.  Eventually you want to prove enough value that you attract investors.  But they need to be convinced that you have a product that cannot easily be copied and that, aside from just satisfying customers, has some intrinsic value that may lead to an acquisition.

From a software perspective, what is value?  It’s really good code that is high quality and well tested.  It’s used to create features that satisfy specific user needs while strengthening the overall product.  The more code I write, the more value I create.

There’s also value in creating a lot of product that doesn’t work out as expected because this allows the team to abandon features that would otherwise consume development cycles and not get used by customers.  But in both cases you need to move quickly, write a lot of code, try a lot of things out to figure out what works and what doesn’t.

As a developer, I’m far more capable of producing lots of top quality code using tools and systems that I’m familiar with and that I’ve built things with in the past.  There is an inherent level of bugginess that a programmer is going to introduce programming in a new language – things get done in a way that work but are not necessarily the correct or maintainable way.  A lot of time can be wasted figuring out how to best integrate the various pieces being brought together – what library is best to use to connect PHP to MongoDB?  Which gem should I use when incorporating payment processing into my Ruby project?

A start up CTO recently told me that he had decided to use Ruby, MongoDB and Node.js for his current project.  I asked him why and he remarked that he felt he needed to learn them and this was a great time to do it.  If you want to learn new languages or technologies, jump into the next hackathon and hang out with other experts that can quickly show you how to use the tools.  Or buy a few books and spend a weekend messing around generating tons of questions to post to message boards and mailing lists.

Two important points that I don’t want misconstrued: 1) the proper process for prioritizing features is absolutely essential.  I’m not advocating that you just build software until you stumble upon value.  Build value based on customer driven development.  2) You cannot build a web application on yesterday’s standards because you will have a hard time attracting the talent you need to lead.  It is very tricky to balance the need for practical technical solutions and bleeding edge implementations for the sake of experimentation.  So tread lightly here but don’t let your stuff get old.