A Different Paradigm for CTO’s

Throughout my career I’ve had CTO’s express their frustration with their developers being “at the bottom of the totem pole”.  Sales teams, product management, or founders generate ideas that need to make it into the product development process.  These ideas, regardless of the methodology used, are stepped through a process of specification where details are laid out and an attempt is made to communicate to developers exactly what should be built.

Many times these ideas come with hard dates attached at which point the features need to be in production.  Once development is underway, however, edge cases, error states, and critical functionality are found to be missing from the specification.  What does a developer do now that an estimate has been given and a date set for production release?

All of the ills in the planning process have hit the developer like a ton of bricks and they are left alone to figure out how to make things work, thus feeling “at the bottom of the totem pole.”  Some colleagues have described to me developers that cope well with ambiguity and those that do not.  In essence, when left without specifications a person considered a great developer will be able to fill in the blanks without missing the deadline.

Unfair?  A bit but not entirely a non-developer problem.

I’ve been meeting with CTO’s recently and many are working to define product development processes that attempt to account for this situation.  Some seem disillusioned with any process’s ability to get it right and the resultant culture established is one of defensiveness, missed dates and overall lack of confidence in what developers can deliver.  My feeling is that the main manifestation of this problem is simple: product doesn’t ship when it ought to.

In fact, I’m hearing more and more that this particular issue is second in accounting for missed deadlines and letting the product team down when a lackluster feature finally makes it into production.  The #1 most disruptive scenario: having to respond to off-release requests that come in from clients, sales, or impatient founders.

I believe, though, that a CTO has failed the second they have attempted to create a process to account for this dynamic.  They certainly have failed when their level of confidence has been diminished to the point that missing deadlines is not only acceptable but expected.

It’s my belief that CTO’s need to stop thinking of themselves as people at the bottom of a well trying to deal with everything that is being dumped into it.  Product development processes aren’t linear and to the extent developers are the last stop on the train it’s because perception or process have put them there.

Throughout my career I’ve managed half or more of the people in my start ups by virtue of being the guy that controls the resources building the company’s core asset.  What does it say if 50% of the people in an organization report to the CTO?  Furthermore, what does it say if 100% of the people in an organization need technology to get their jobs done?  It means that the CTO is literally beholden to everyone and can have significant impact on efficiency, culture, and process.

I would argue that a CTO owns at least one process with every single department in their company: sales has to know how to communicate recurring client requests into product development, marketing needs to coordinate messaging with product releases, IT controls desktops and networks, customer service is significantly more efficient if they have the proper tools to quickly handle service requests.

The CTO isn’t the buffer at the bottom of well between everyone dumping things in and the developers.  The CTO is the hub and everyone else is a spoke.

A CTO’s ability to effect culture or implement sweeping change is unparalleled and subtle in most organizations that I’ve been a part of or have interacted with throughout my career.  CTO’s need to be proactive, transparent, confident, and present.  When a problem arises, it is likely a result of her/his people or process or something that, with a bit of focused work, she/he can solve being the center of an extraordinary number of exchanges.

Make no mistake, these issues start early and can lead to significant frustration and breakdown.  I’m coaching more and more CTO’s to get up from their desks and create relationships and processes that get ahead of these deficiencies.  I’m helping them to craft processes that instill confidence in their ability to deliver and in the organization’s to get things done and ship product.  The first step, though, is getting them to realize just how core they are to everything that is happening around them and getting them comfortable with learning to see problems, rolling up their sleeves and getting in there to fix them.

VP of Engineering (VPE) vs. Chief Technology Officer (CTO)

The roles of VPE & CTO are important in terms of their distinctions & one or both may be required depending on the stage & make up of your business.  At least one is essential if technology or software are core assets of what you do.

My preference is to see a founding team have at least one (and preferably more) engineers on it with one CTO.  My hope is that the person will grow with the business & learn to understand the different roles & responsibilities they’ll need to master as they transition from founding developer to full fledged CTO.

Here’s how I see the differences in these critical roles.

VPE’s optimize process, hire people & implement tools.

  • They code with some frequency and are considered a developer.
  • They are often player/coaches.
  • They perform light management tasks like weekly meetings & 1-on-1’s.
  • They are strategic in the sense that they determine what roles should be hired to fill present day gaps.
  • They may perform light or all project management.
  • More often than not, they take direction over setting it.
  • Pervasive morale issues will often impact them in the same negative ways they impact others.

CTO’s scale people & process and create engineering-centric cultures.

  • They may perform all of the tasks of the VPE (depending on the stage of the company).
  • They think strategically about how to build scale such that adding more people is minimally disruptive.
  • They establish standards of quality & behavior.
  • They evangelize the business to engineers.
  • They connect non-technical business needs to day-to-day technical operations.
  • They create and scale processes where interfacing with non-engineering functions is required.
  • They represent a valuable asset (tech) to investors, customers & partners.
  • They foresee technological changes & internalize them.
  • They instigate changes through the use of new technology.

Experience, business acumen & inter-personal skills are key traits for a successful CTO that can provide mentoring & leadership.  I don’t believe it’s enough to be able to perform the tasks of CTO without first having experienced the role of VPE.  Put another way, the best CTO’s I’ve met started their careers as founding developers, grew into VPE’s & eventually stood as the CTO.

The key factor when considering a CTO is timing around when scale is necessary.  More often than not, thinking about scale earlier rather than later can be a key advantage.  When the business needs to be grown by virtue of adding more money, marketing & people, having strong technology, culture & process are required to make sure you don’t miss a beat.

Beware of the CTO that is actually a VPE & cannot transcend his or her own limitations to take on the larger role.  I keep my eye on what people like to do day-to-day over what they should be doing to determine what role is best for them.  For example, constantly going back to writing code or project managing tells me the candidate is likely a VPE because they love that type of work.

Want to discuss more?  Email me at mcmontero at gmail dot com.


Fast & Loose Estimates

I want to praise @SteveKuhn for giving fast and loose estimates.  He’s a busy guy @CrowdTwist and I can’t imagine he actually stands behind some of his estimates when he makes them based on the number of things he’s juggling in a given day.  But what he does is incredibly important and worth highlighting.

Often I’ll hear him say, “I think I can have that for you shortly after lunch.”  This is intriguing: what time is lunch and how much time does "shortly" mean?  Even with so much ambiguity, he’s still doing a few very important things for the people he works with and those that are dependent on him:

  1. He’s created a forcing function for himself.  He’s made a commitment that he needs to honor.
  2. He’s given the person waiting on him a rough sense of how much time they must wait before he delivers so they can go off and fill the time with other tasks.
  3. He’s reinforcing that it’s important, as a team member, to respect other people’s time and give them some sense of what to expect.
  4. When he can’t deliver he’s forced a touch point to communicate his status and a new deliverable.  This continues to help the people waiting on him plan their time more effectively.
  5. He quickly time boxes his tasks.  This let’s him plan his own time better and forces him to finish tasks in the time he’s allotted.

Being Agile

Agile is not a software development methodology.  Being Agile, implementing Agile practices, requires cultural and personal changes that seem counter intuitive based on older software methodologies and even some conceptions that employees have about corporate culture.

Attempting to “implement Agile” as a process will fail.  It will fail, more likely than not, by creating what appears to be (to an astute observer) Agile waterfall.  Participants will feel compelled to work faster within the waterfall framework and even iterate over each of the steps of the process.  Ultimately, a team will feel more that Agile does not work or is unattainable.

If you haven’t read the Agile Manifesto or it’s supporting documentation you should:

Manifesto for Agile Software Development
Principles behind the Agile Manifesto

Here’s an excerpt:

    “Individuals and interactions over processes and tools
     Working software over comprehensive documentation
     Customer collaboration over contract negotiation
     Responding to change over following a plan”

When I read this, nothing sounds procedural.  These guidelines appear to be fundamental cultural traits that emphasize people-to-people (and decidely not people-to-computer) interactions.

Why?  Because using a computer is a very solitary activity and as such eliminates any opportunity for people to engage one another.  Here are some examples of what not being Agile looks like:

    "Can you tell me about feature XYZ?"
    “You can read about it on the Wiki.”

    “I think I found a bug.”
    “File a ticket in Jira.”

    “How does feature ABC work?”
    “I’ll write a document about it and email it to you.”

    “Can our software do DEF?”
    “Send me an email and I’ll get back to you with what we’re capable of.”

You cannot measure the loss from shutting down conversations by directing people to interact with you by first interacting with their computer.  The more this pervades day-to-day life the more quickly teams entrench themselves in building digital processes to replace personal interactions.

Over time this will devolve into spending time managing “the process” itself (and all of the sub-processes that each team has built) over building self-motivated teams that learn through constant practice how to excel at face-to-face communication.  Let’s face it, most of us sit within 20 feet of every other member of our team yet for many of us our preferred mechanisms of communication include email, Skype, wikis and requirements management software.

Nothing can require someone to change their personal beliefs more than this:

   ”Responding to change over following a plan”

Few people can exist without a plan and often find themselves agitated or terrified without one.  Traditional structures for blame and accountability fall away when your culture truly understands (particularly in software based start ups) that stuff changes all the time.  Building software involves far more unknowns and last minute changes than most people would like to believe. But it’s called “soft”ware because it can be molded and changed like clay. Hardware, on the hand, computers and phones and printers, are literally made of molded plastic and metal that cannot be changed without first destroying it.

Agile practices require Agile individuals, Agile physical spaces and a cultural desire to first get out from behind a computer, engage in face-to-face communications with your team before getting to work constantly course correcting, handling surprises and adapting to change.  The rewards, though, are far more meaningful than a computer telling you you successfully submitted the new documentation you wrote…all alone.


How I Think About Scale

I have to think a lot about scale @CrowdTwist.  Investors and clients ask me all the time - “Does it scale?”  Yes, it does!  But I thought I’d shed some light on how I have to think about scale since my concerns go beyond whether or not servers can handle traffic if it increases.  So here’s what I’m constantly managing:

  • Do our processes and tools scale we as add more members to the product development team?
  • Does our quality assurance process scale such that quality remains as high as it is even if we’re producing more software faster?
  • Does launching a new client instance of our software scale even though the product is highly configurable including the ability to customize any piece of text and translate it to 109 languages?
  • Do our infrastructure automation systems scale as the number of servers increases?
  • We’re required to manually QA several client configurations across an array of browsers and devices.  How do we continue to scale the QA process to handle this?
  • How do we scale communication with Sales and Client Services as we add more features and introduce more functionality into the product?
  • How do we scale communication to clients as we increase the number of releases of our software and the breadth of our product?
  • We are getting a lot of RFP’s so I’m constantly thinking about how we scale documentation and communication with the people handling our responses.
  • How do I scale?  I code, I manage, I go on sales calls, I architect solutions and manage infrastructure.  PFM?  No, I have an incredible team that makes all of this happen and makes something that is incredibly sophisticated and big scale look easy.

And yes, we can handle ALL of the data and ALL of the traffic.  Bring it!

Nigerian Scam By Snail Mail

While working as the CTO of CityRealty.com we received the documents below by mail.  It’s the “Nigerian scam” but modified slightly to suit a business.  Essentially someone is asking us to wire money directly to a Nigerian bank account as a loan so the writer can transfer in this case $30MM into their US bank account.  In this variation, though, they aren’t offering us a cut of the money for granting them the loan which I found unusual.

I also love that the writer prefers communication only by fax or email!  I found this document recently and thought I’d share.  Six pages follow:

How We Recovered From Multiple AWS Failures

After the Amazon Web Services (AWS) failure on Elastic Block Store (EBS) volumes in the us-east-1 availability zone (October 22, 2012), CrowdTwist suffered multiple component failures.  We lost four EBS volumes that were members of RAID and one server completely died.  These outages cost us no downtime.

We’ve spent a lot of time architecting our AWS infrastructure and working with very smart people from Amazon and Google to create systems that are resilient and built “according” to the way in which EC2 was intended to be used.  That paid off last week.

Recovering the server was easy: it was one of two servers configured for redundancy.  We launched a new instance with our custom Amazon Machine Image (AMI) and put the new server into rotation in the appropriate Elastic Load Balancers (ELB).

The EBS volumes that failed were all configured using md (software RAID) and were members of RAID 1 or RAID 10 groups.  These disks going missing did not affect the RAID, the server or the database at all.

We use Oracle 11g and have combined several EBS volumes into RAID configurations to boost the performance of reads and writes.  The Pythian Group manages our databases and helped us configure the disks properly to optimize performance.  Note, this was done before AWS started offering provisioned IOPs.

I’ll spare you the very low level technical details (but please direct message me on Twitter if you want them) but essentially the volumes in question were no longer visible at the OS level even though the AWS management console was seeing them as attached to the server they were originally mounted on.  We created matching volumes, mounted them on the server, and used mdadm to fail out the old disks and add the new ones.  The longest part of the operation was the recovery mdadm had to perform as a result of a new disk being introduced.

There is an interesting trend among my peers of moving out of AWS to managed or dedicated hosting.  The primary reason given is that AWS fails too often.  I don’t disagree that the failure rate is too high but we’ve managed to architect a solution that allows us to recover very quickly with no loss.

I’m proud of that and impressed with both the technology behind AWS and the people that have helped architect our stuff.

[If you’re curious, the AWS outage lasted 15 hours.  CrowdTwist was affected for one hour and one minute because we were able to fail over to redundant systems including a complete Oracle fail over to stand by.  We could have been up in under 20 minutes but took the time to make sure no data was lost.  None was!]

How Technical Founders Create Value

Here’s what not to do as a start up CTO: use a completely new set of technologies to build your product, stuff you’ve never used before or have little experience with.


You’re usually convinced that the idea you’re working on has a place in the market.  Therefore your number one goal should be to create value.  The more product you build that customers like and use, the more value you’re creating for the business.  Eventually you want to prove enough value that you attract investors.  But they need to be convinced that you have a product that cannot easily be copied and that, aside from just satisfying customers, has some intrinsic value that may lead to an acquisition.

From a software perspective, what is value?  It’s really good code that is high quality and well tested.  It’s used to create features that satisfy specific user needs while strengthening the overall product.  The more code I write, the more value I create.

There’s also value in creating a lot of product that doesn’t work out as expected because this allows the team to abandon features that would otherwise consume development cycles and not get used by customers.  But in both cases you need to move quickly, write a lot of code, try a lot of things out to figure out what works and what doesn’t.

As a developer, I’m far more capable of producing lots of top quality code using tools and systems that I’m familiar with and that I’ve built things with in the past.  There is an inherent level of bugginess that a programmer is going to introduce programming in a new language – things get done in a way that work but are not necessarily the correct or maintainable way.  A lot of time can be wasted figuring out how to best integrate the various pieces being brought together – what library is best to use to connect PHP to MongoDB?  Which gem should I use when incorporating payment processing into my Ruby project?

A start up CTO recently told me that he had decided to use Ruby, MongoDB and Node.js for his current project.  I asked him why and he remarked that he felt he needed to learn them and this was a great time to do it.  If you want to learn new languages or technologies, jump into the next hackathon and hang out with other experts that can quickly show you how to use the tools.  Or buy a few books and spend a weekend messing around generating tons of questions to post to message boards and mailing lists.

Two important points that I don’t want misconstrued: 1) the proper process for prioritizing features is absolutely essential.  I’m not advocating that you just build software until you stumble upon value.  Build value based on customer driven development.  2) You cannot build a web application on yesterday’s standards because you will have a hard time attracting the talent you need to lead.  It is very tricky to balance the need for practical technical solutions and bleeding edge implementations for the sake of experimentation.  So tread lightly here but don’t let your stuff get old.

Calamity! Apparently my portable hard drive fell out of my bag and into a few inches of snow. It remained there overnight in below zero temperatures until my wife found it in the morning. Here’s to hoping it still works after a few hours of being warm and dry. Keep you posted.

Update: I plugged the drive in and it worked with no issues.

Four Questions I Asked My Staff for the New Year

They were:

  • What are your personal goals for 2012?
  • What are your professional goals for 2012?
  • What would you like to see the CrowdTwist technical team accomplish this year?
  • What would you like to see the company accomplish this year?

I need to know how to align the resources I can bring to bear for this incredible team with what they would like to accomplish both in advancing their professional careers as well as enriching their personal lives.

By asking, I’ve made a commitment to review what they’ve told me and figure out ways to structure training, compensation, days off, technical projects, reading materials, etc.  I want to craft a schedule that allows the team to take responsibility for implementing various parts of a plan that we’ll create to advance these goals.

What’s missing?  As @ArshadGC pointed out in a recent discussion we had, we need to make sure we didn’t get people’s New Year’s resolutions as those more often than not go unfulfilled.  So understanding where everyone wants to go in their careers and aligning the immediate goals to the long term will help to make sure we don’t waste people’s efforts.

Lastly, I’m really impressed with what the team wants to accomplish.  It’s very motivating for me to know I have a team with such varied interests and a high degree of concern about their accomplishments in 2012.  We’re on a mission!