Hiring for #devops - a primer

I've written about this previously as part of another post but I've had a few things on my mind recently about the topic and needed to do a brain dump.

As I mentioned in that previous post, I'm currently with a company where devops is part of the title of our team. I won't go into the how and why again for that use case. What I want to talk about is why organizations are using DevOps as title in both hiring and as an enumerated skillset.

We know that what makes up DevOps isn't anything new. I tend to agree with what John Willis wrote on the Opscode blog about CAMS as what it means to him. The problem is that even with such a clear cut definition, companies are still struggling with how to hire people who approach Operations with a DevOps "slant". Damon Edwards says "You wouldn't hire an Agile" but I don't think that's the case at all. While the title might not have Agile, it's definitely an enumerated skill set. A quick search on monster in a 10 mile radius from my house turned up 102 results with "Agile" in the description such as:

• experienced Project Manager with heavy Agile Scrum experience
• Agile development methodologies 
• Familiar with agile development techniques
• Agile Scrum development team 

Yes, it's something of a misuse of the word Agile in many situations but the fact of the matter is that when a company is looking for a specific type of person, they tend to list that as a skill or in the job description. Of course Agile development is something of a formal methodology whereas DevOps isn't really. I think that's why I like the term "Agile Operations" more in that regard. But in the end, you don't have your "Agile Development" team and so you really wouldn't have your "Agile Operations" team. You have development and you have operations.

So what's a company to do? They want someone who "does that devops thing". How do they find that person? Some places are listing "tools like puppet, chef and cfengine" as part of skill sets. That goes a long way to helping job seekers key off of the mindset of an organization but what about the organization? How do they determine if the person actually takes the message of DevOps to heart? I think CAMS provides that framework.

Culture and Sharing

What kind of culture are you trying to foster? Is it one where Operations and Development are silos or one where, as DevOps promotes, the destruction of artificial barriers between the groups? Ask questions of potential employees that attempt to draw that out of them. Relevance to each role is in parenthesis.

• Should developers have access to production? Why or why not? (for Operations staff)
• Should you have access to production? Why or why not? (for Development staff)
• Describe a typical release workflow at a previous company. What were the gaps? Where did it fail? (Both)
• Describe your optimal release workflow. (Both)
• Have you even been to a SCRUM? (Operations)
• Have you ever had operations staff in a SCRUM? (Development)
• At what point should your team start being involved/stop being involved in a product lifecycle? (Both)
• What are the boundaries between Development and Operations? (Both)
• Do you have any examples of documentation you've written? (Both)
• What constitutes a deployable product? (Both)
• Describe your process for troubleshooting an outage? What's the most important aspect of an outage? (Both)

Automation and Metrics

This is somewhat equivalent to a series of technical questions. The key is to deduce the thought process a person uses to approach a problem. Some of these aren't devops specific but have ties to it. Obviously these might be tailored to the specific environment you 

• Describe your process for troubleshooting an outage? What's the most important aspect of an outage? (Both)
• Do you code at all? What languages? Any examples? Github repo? (Operations)
• Do you code outside of work at all? Any examples? Github repo? (Development)
• Using psuedo-code, describe a server.  An environment. A deployable. (Operations)
• How might you "unit test" a server? (Operations)
• Have you ever exposed application metrics to operations staff? How would you go about doing that? (Development)
• What process would you use to recreate a server from bare metal to running in production? (Operations)
• How would you automate a process that does X in your application? How do you expose that automation? (Development)
• What does a Dashboard mean to you? (Both)
• How would you go about automating production deploys? (Both)

A few of these questions straddle both aspects. Some questions are "trick questions". I'm going to assume that these questions are also tailored to the specifics of your environment. I'm also assuming that basic vetting has been done.

So what are some answers I like to hear vice don't ever want to hear? Anything that sounds like an attitude of "pass the buck" is a red-flag. I really like seeing an operations person who has some sort of code they've written. I also like the same from developers outside of work. I don't expect everyone to live, breathe and eat code but I've known too many people who ONLY code at work and have no interest in keeping abreast of new technologies. They might as well be driving a forklift as opposed to writing code.

I think companies will benefit more from a "technologist" than someone who is only willing to put in 9to5 and never step outside of a predefined box of responsibilities. I'm not suggesting that someone forsake family life for the job. What I'm saying is that there are people who will drag your organization down because they have no aspirations or motivations to make things better. I love it when someone comes in the door and says "Hey I saw this cool project online and it might be useful around here". I love it from both developers and operations folks.

Do with these what you will. I'd love to hear other examples that people might have.