Latest Vogeler update - MongoDB, protobufs, Riak and war!

I wanted to take a minute to post an update about Vogeler to those who are following the project. Let's get the easy stuff out of the way - it's not abandoned. Far from it.

There have been several reasons why I haven't made any commits lately, the least of which is both kids have been sick recently and I haven't been able to get a good solid block of time to work on it.

Technical Hurdles

Another reason is that I almost went down a rabbit hole with regards to swappable persistence. In the process of refactoring the persistence backend, I realized it should be fairly easy, using the model I put into place, to go ahead and implement MongoDB and Riak support. I started with MongoDB when I hit a wall. MongoDB does not allow dots in key names. When I ran into that issue, I realized that I made some dangerous assumptions based on the fact that I started with Couchdbkit as the interface to CouchDB:

I was using an ORM when I should have used a lower level driver. You see couchdbkit does some nice stuff like translating native Python datatypes to the appropriate datatypes. If I define a row as having DictProperty(), couchdbkit converts that into the commensurate CouchDB JSON datatypes. If I use ListProperty(), the same thing. This is really evidenced in Futon and makes using Futon as your interface to Vogeler very appealing. However this is VERY couchdb specific.

The pymongo driver, however, didn't like my strategy of dumping execution results that way. You can see the "gist" of what I'm talking about here:

;

I brought the issue up on the MongoDB mailing list here. I opened an issue for myself to braindump my thoughts. One of my biggest goals (data transparency) was starting to fall apart for me. I decided to shelve MongoDB for a moment and look at Riak. I wanted to make sure that I at least thought about how a generic model would work across multiple document stores. That's when I ran into the biggest cockblock:

protobuf

I'm almost firmly convinced that protobuf is a piece of trash. Google has some smart people but protobuf is something that quite obviously came out of the mind of someone who was sent off to "solve the RPC problem". There are quite a few issues I have with protobuf:

• Despite being a "universal" format, it works well in exactly TWO languages - C and Java. Everything else is an afterthought. Don't get me started on Python support. The one guy at Google who supports protobuf on Python can't make it work on anything but Python 2.5 because that's all Google uses. He's unwilling to cut a new PyPi package just to fix all the 2.5 assumptions because he doesn't want to bump the version number. You can't even install it on anything higher than 2.5 without hacking setup.py.
• You have to precompile your protos before use. I understand what Google is trying to accomplish but seriously? So I have to build the protobuf compiler to compile protos to ship with my code. There's a reason why people like FFI folks.

There are alternatives like Apache Avro that have promise but they also have their own issues. However, Basho has committed to using protobuf which does make sense. Write your own serialization framework or use an existing one? Easy answer when Google wrote one for you.

So I started to noodle out what route I wanted to take when something else came out of left field.

Sgt first Class Lance Vogeler

I have a search setup in Tweetdeck on my Droid for Vogeler. It was nice to stay on top of people mentioning the project. The name for the project came out of me pretty much immersing myself in the latest S.M. Stirling Emberverse books. One of the characters was named Ingolf Vogeler. I really enjoyed the books and liked the name so I picked it. I'm also considering using Ritva for another project.

So one day my phone starts going nuts with Vogeler alerts. I was already getting the occasional history tweet about the real Ingolf Vogeler but it turns out a soldier from Georgia, of all places, was KIA in Afghanistan. He did 8 tours in Afghanistan and 4 tours in Iraq. Politics aside (I'm personally entirely against these campaigns), I didn't want to "pollute" the twitter stream. Regardless of what I think of the current military climate in my country, I have the utmost respect for most of the members in our armed forces.

However what struck me most is that SFC Vogeler left behind a wife. A wife carrying his unborn child. That pretty much did me in. As a father myself, I was pretty torn up thinking about this happening to my wife. Yes, it was a known risk but that doesn't make it any less sad. I decided, in addition to making a donation of my own to his family and holding off on Vogeler till it wasn't alerting on Twitter so much that I would think hard about what my software means.

Steve Jobs asked a guy who emailed him this question "What have you created lately?" Someone else recently said that entrepreneurs are busy creating the next social media app that means fuckall when they could be affecting change with the software they write. That got me thinking. Could I help affect this family somehow with my project that happened to share a name with them? The best I could come up with is this:

If you use Vogeler, are interested in it or just feel like a random act of kindness, please make a small donation to the Vogeler family. My wife and I agreed that should I ever make ANY money off of the project in any identifiable form that I would donate what I could to the family. Vogeler is just a small project. I have no grand aspirations of getting integrated into some mainstream project. I'm just trying to scratch an itch - a niche itch at that. One reason I'm so gung-ho about DevOps is that, as a family man, I don't WANT to be dealing with stupid shit taking time away from my family. I've done it and I'm done with it. If my phone goes off, it's not going to be from some stupid mistake that I made editing a config file or lack of metrics causing an "oh shit we're out of space" moment. I'm past that in my career and I'm past having to work places where that's the norm rather than the exception. My family is first and foremost and anything I can do to keep it that way, I'm going to do it.

So I'm taking the Vim approach to Vogeler. I'm not going to ask anyone to go against his conscience. If you feel like a small donation to this family implies consent to the stupidity of my government then I fully understand. But if you think that open source software and the broader open source community can make a difference in more than just writing software, throw a small donation their way.

I'm going to be starting back up on Vogeler now. I've decided that for now, I'm going to attempt to keep things as generic as possible but continue to code against CouchDB. I'll keep revisiting MongoDB and Riak support but the primary target is CouchDB. If any Basho folks are reading this, if you can remove the hard dep in setup.py on protobuf, that would be awesome. You can't even install from PyPi with it in there anyway. If the MongoDB folks take a gander at this, can you do something about the dots in key names? Thanks!

Why I hope BankSimple succeeds but fear they won't have a choice in the matter

My current political affiliation is not the most popular with people right now. That's understandable. I tend to approach things from a rational and unemotional perspective. Call me callous. Call me a dipshit. I'm consistent if nothing.

But the issue that's been bugging me lately transcends party or political affiliation. It's one that touches me as a technologist, an IT worker and a citizen.

DevOps and the changing IT landscape

I've blogged/tweeted/talked/rambled enough about DevOps to make people sick. In some regards we risk crossing the line with our enthusiasm and killing interest in the topic. Ignore for a minute that much of what makes up "DevOps" is not new ground. That's not to say that the "movement" hasn't introduced some amazing things.

The point is that aspects of DevOps as a philosophy are changing how people approach IT holistically. Automation. Rapid iteration. Frequent changes. Breaking down organizational silos.

So how does this play into BankSimple?

It actually plays into it in two ways. The first is that, like DevOps, BankSimple isn't doing anything "new" per se. If you're talking "old" business, banking ranks right up there with prostitution and in shares more in common with it than most would want to admit. What's different is how BankSimple is doing it. The BankSimple guys want to make money, no doubt but they're approaching it from a different focus. It's been formed, not by the traditional cadre but by "techies". It's almost "Agile Banking".

You can guarantee that because of that perspective and the history of the cofounders, the business and by extension the IT aspect will be run differently.

If you can't beat 'em, buy 'em.

I'm a hands-off kind of guy when it comes to my government. Many people thought that, based on concepts like "hope" and "change", the government was going to turn around and focus on its customers - the citizens. It hasn't yet turned out that way.

Someone asked an interesting question at the last DevOps meetup in Atlanta.

"Do you think traditional companies feel threatened by startups who use more agile methods to beat them to market?"

The answer:

"Not really. If they get to be too big of a threat, just buy them out.".

That, of course, assumes they're willing to sell. Some people might have that luxury but when the VCs start to chime in that choice may be stripped from you. On the flipside, some might believe that by selling what they've worked so hard to build up will be destroyed yet they feel an obligation to the people who started with them to sell.

But let's assume that, in the case of BankSimple they have the goals of REALLY changing banking and finance (which I think they do). Assume they don't have to sell (which I don't know).

If you can't buy em, buy a lobbyist

I was talking to an individual a few weeks back about the idea that traditional companies would get eaten alive by leaner more agile startups. We were talking specifically about the financial sector but his comment applies. So I, with a head full of steam raving about agile operations and time to market, was hit head on with this reality in one comment:

"Yeah but do they have lobbyists?"

Every company I've worked at, save one, has had lobbyists on the books. At one company we actually had different colored checks that we printed for lobbyists vice expense checks. Some people think lobbyists just "lobby" for things like deregulation and "leave my business" alone but that's not the case.

Without going into too much detail, at one company we actually LIKED a certain amount of regulation. You see a certain amount of regulation is just enough to restrict entry into the market by new players. We liked that each state had its own set of regulations. That in one state we were classified under one set of laws while in another state we were classified differently. This ARTIFICIALLY raises the barrier to entry for competitors. It keeps them small. It relegates them to one of two positions:

• Small enough for us not to care because they can't afford to expand into new markets
• Big enough to justify buying them to eliminate the competition.

Sure, sometimes a company will succeed and grow big enough to be REAL competition but that's fine too. Helps prevent antitrust investigations. It's a win-win all around.

Not all lobbyists are bad. I would argue that the ACLU is a "good" lobbyist but they can also exist to help preserve an existing business either buy demanding hands-off or demanding more regulation under the guise of "consumer protection".

Sometimes the tree of innovation must be watered with the blood of failed business models

I sent this out as a tweet earlier tonight. With no disrespect meant to the original quote that inspired it, I firmly believe this to be true. Without even discussing politics or regulation or monopolies, the fact of the matter is that to progress as a society some things have to die. Horse and buggy. Media distribution. Publishing. All of these changes help move us to greater things. Many a business is built on convenience and inefficiencies in the supply chain. FedEx and UPS would probably have not been needed had the post office not sucked so bad. In an age when information is passed slowly, traditional news organizations made sense. BankSimple is part of that progression just as DevOps is part of that transition in IT.

But I hope that Alex and the gang aren't as naive as some people (myself included) have been about DevOps. Just as DevOps working its way into traditional organizations will have to deal with the Boogymen of HIPPA, SOX and PCI DSS, so will BankSimple have to deal with an establishment that, if threatened, sadly has the power to essentially make them "illegal".

We need them to succeed so we can move on to better and brighter things.

 

Hiring for #devops - a primer

I've written about this previously as part of another post but I've had a few things on my mind recently about the topic and needed to do a brain dump.

As I mentioned in that previous post, I'm currently with a company where devops is part of the title of our team. I won't go into the how and why again for that use case. What I want to talk about is why organizations are using DevOps as title in both hiring and as an enumerated skillset.

We know that what makes up DevOps isn't anything new. I tend to agree with what John Willis wrote on the Opscode blog about CAMS as what it means to him. The problem is that even with such a clear cut definition, companies are still struggling with how to hire people who approach Operations with a DevOps "slant". Damon Edwards says "You wouldn't hire an Agile" but I don't think that's the case at all. While the title might not have Agile, it's definitely an enumerated skill set. A quick search on monster in a 10 mile radius from my house turned up 102 results with "Agile" in the description such as:

• experienced Project Manager with heavy Agile Scrum experience
• Agile development methodologies 
• Familiar with agile development techniques
• Agile Scrum development team 

Yes, it's something of a misuse of the word Agile in many situations but the fact of the matter is that when a company is looking for a specific type of person, they tend to list that as a skill or in the job description. Of course Agile development is something of a formal methodology whereas DevOps isn't really. I think that's why I like the term "Agile Operations" more in that regard. But in the end, you don't have your "Agile Development" team and so you really wouldn't have your "Agile Operations" team. You have development and you have operations.

So what's a company to do? They want someone who "does that devops thing". How do they find that person? Some places are listing "tools like puppet, chef and cfengine" as part of skill sets. That goes a long way to helping job seekers key off of the mindset of an organization but what about the organization? How do they determine if the person actually takes the message of DevOps to heart? I think CAMS provides that framework.

Culture and Sharing

What kind of culture are you trying to foster? Is it one where Operations and Development are silos or one where, as DevOps promotes, the destruction of artificial barriers between the groups? Ask questions of potential employees that attempt to draw that out of them. Relevance to each role is in parenthesis.

• Should developers have access to production? Why or why not? (for Operations staff)
• Should you have access to production? Why or why not? (for Development staff)
• Describe a typical release workflow at a previous company. What were the gaps? Where did it fail? (Both)
• Describe your optimal release workflow. (Both)
• Have you even been to a SCRUM? (Operations)
• Have you ever had operations staff in a SCRUM? (Development)
• At what point should your team start being involved/stop being involved in a product lifecycle? (Both)
• What are the boundaries between Development and Operations? (Both)
• Do you have any examples of documentation you've written? (Both)
• What constitutes a deployable product? (Both)
• Describe your process for troubleshooting an outage? What's the most important aspect of an outage? (Both)

Automation and Metrics

This is somewhat equivalent to a series of technical questions. The key is to deduce the thought process a person uses to approach a problem. Some of these aren't devops specific but have ties to it. Obviously these might be tailored to the specific environment you 

• Describe your process for troubleshooting an outage? What's the most important aspect of an outage? (Both)
• Do you code at all? What languages? Any examples? Github repo? (Operations)
• Do you code outside of work at all? Any examples? Github repo? (Development)
• Using psuedo-code, describe a server.  An environment. A deployable. (Operations)
• How might you "unit test" a server? (Operations)
• Have you ever exposed application metrics to operations staff? How would you go about doing that? (Development)
• What process would you use to recreate a server from bare metal to running in production? (Operations)
• How would you automate a process that does X in your application? How do you expose that automation? (Development)
• What does a Dashboard mean to you? (Both)
• How would you go about automating production deploys? (Both)

A few of these questions straddle both aspects. Some questions are "trick questions". I'm going to assume that these questions are also tailored to the specifics of your environment. I'm also assuming that basic vetting has been done.

So what are some answers I like to hear vice don't ever want to hear? Anything that sounds like an attitude of "pass the buck" is a red-flag. I really like seeing an operations person who has some sort of code they've written. I also like the same from developers outside of work. I don't expect everyone to live, breathe and eat code but I've known too many people who ONLY code at work and have no interest in keeping abreast of new technologies. They might as well be driving a forklift as opposed to writing code.

I think companies will benefit more from a "technologist" than someone who is only willing to put in 9to5 and never step outside of a predefined box of responsibilities. I'm not suggesting that someone forsake family life for the job. What I'm saying is that there are people who will drag your organization down because they have no aspirations or motivations to make things better. I love it when someone comes in the door and says "Hey I saw this cool project online and it might be useful around here". I love it from both developers and operations folks.

Do with these what you will. I'd love to hear other examples that people might have.