Who owns my availability?

Hey did you know EC2 had problems today? Yeah nothing major just a total effing collapse of the EBS system at US-EAST-1.

You know what that means....

"Hey guys, can anyone tell me who owns my availability?"

"Internet learns lesson of putting "all eggs in the EC2 basket". Buy your own machines, brothers."

I could go on....but I won't. I'm also going to stop short of posting a CeeLo video at this point.

Your stupid little comments mean nothing. I especially find it hilarious that someone from Twitter would make a comment availability. I also find the short-lived memory of some people hilarious (paraphrasing here):

"Thank god we're hosted on Joyent/Linode/My mom's basement"

Please. Your attempt to curry favor and free service with your provider are transparent and frankly, makes you look stupid.

Yo Netflix/SimpleGeo/JRandomDude I'm happy for you and and all. I'ma let you finish but....

So who DOES own my availability?
Here's a hint; it's not always that simple.

Yes, the ultimate responsibility for those impacted lies with those who were impacted but let's look at a few facts (or excuses - if you're being a dick about it):

Not everyone has the resources of a Netflix
Comparing anyone else's EC2 usage to Netflix is simply retarded. It's a lot like working with an ex-Google employee (I've worked with a few). They have some awesome ideas and learned some great stuff there but guess what? About 85% of it is USELESS to anyone except someone the size of Google. What works at Google doesn't work at my company.

It's not even a matter of scaling down the concept. It's simply NOT possible. Yeah let me just go buy a shipping container and build a datacenter in a box. Hardware failure? Replace the box with one off the shelf. Oh wait, not everyone has a warehouse of replacement servers. People have trouble getting a few spare hard drives to swap out.

Telling someone that they should just do what Netflix does makes you look stupid. Not them.

WE used Joyent/Linode/GoGrid/My mom's basement
Really? Really? I'm not being an AWS fanboy here but here is a simple fact: No other 'cloud' provider comes even REMOTELY close to the feature set of AWS. No one. Not only does no one come close but Amazon is CONSTANTLY iterating on new stuff to widen the gap even more.

It's not like your provider hasn't had a major outage in recent memory. And comparing an effing VPS provider to Amazon? You seriously just don't get it.

You should have designed around this possibility
Well no shit, sherlock. Guess what, it was rejected. Why? Who knows? Who cares? It's irrelevant. Sometimes the decision isn't ours to make. In the REAL world, people have to balance risk vs. reward.

Here's a tidbit of information. At EVERY single company I've been at where I was involved with architecting a solution from the ground up, we never had redundancy built in from the get go. Did I find it appalling. Absolutely but the choice wasn't mine. I did the best I could to prevent anything that would make adding it TOO difficult later on but we didn't have our DR site online from day one. We sometimes had to accrue a little technical debt. The best we could do was to minimize it as much as possible.

Designing around failure is not the same as designing for the worse case scenario. Sometimes you just have to accept that "if component X has Y number of failures, we're going to have an outage". If you have the ability to deal with it now (resources/money/whatever), then that's awesome. Sometimes you just have to accept that risk.

Oh sure I'd love to use (insert buzzword/concurrent/distributed language of the day) here. But I can't. It would be totally awesome if everything were designed from the ground up to handle that level of failure but it's not.

And another thing
The thing that bothers me most is the two-faced attitude around it all.

On one hand people are telling you it's stupid to host your own hardware. On the other hand they'll laugh at you when your provider has an outage and tell you that you should have built your own.

On one hand they'll tell you it's stupid to use some non-traditional new-fangled language and on the other hand laugh at you when you could have avoided all these problems if you had just used non-traditional new-fangled language.

On one hand they'll tell you that you should use insert-traditional-RDBMS here and on the other hand say that it's your fault for not rearchitecting your entire codebase around some NoSQL data store.

Not everyone has the same options. I hate the phrase "hindsight is 20/20". Why? Because it's all relevant. Sometimes you don't know that something is the wrong choice till it bites you in the ass. Hindsight in technology is only valuable for about a year. Maybe 6 months. Technology moves fast. It's easy to say that someone should have used X when you don't realize that they started working on things six months before X came along. If you have that kind of foresight, I'd love to hire you to play the stock market for me.

Not everyone has the luxury of switching midstream. You have to make the most what technology is available. If you keep chasing the latest and greatest, you'll never actually accomplish anything.

Are these excuses? Absolutely but there's nothing inherently wrong with excuses. You live and learn. So to those affected by the outage (still on-going mind you), take some comfort. Learn from your mistakes. The worst thing you could do at this point would be to NOT change anything. At a minimum, if you aren't the decision maker, you should document your recommendations and move on. If you are the decision maker, you need to..you know...decide if the risk of this happening again is acceptable.

Sinatra, Noah and CloudFoundry - the dirty details

So via some magical digital god, my signup for Cloud Foundry got processed. Obviously my first thought was to try and get Noah up and running. Cloud Foundry is a perfect fit for Noah because I have access to Redis natively. I have a working setup now but it took a little bit of effort.

Getting set up
As with everything these days, my first action was to create a gemset. I'll not bore you with that process but for the sake of this walkthrough, let's use a 1.9.2 gemset called 'cfdev'.

The VMC getting started guide has most of the information you'll need but I'm going to duplicate some of it here for completeness:

 gem install vmc
 vmc target api.cloudfoundry.com
 vmc login

And we're ready to rock. The VMC command line help is very good with the exception that the optional args aren't immediately visible.

vmc help options

will give you a boatload of optional flags you can pass in. One that was frequently used during the demos at launch was '-n'. I would suggest you NOT use that for now. The prompts are actually pretty valuable.

So in the case of Noah, we know we're going to need a Redis instance. Because everything is allocated dynamically CloudFoundry makes heavy use of environment variables to provide you with important settings you'll need.

First Attempt
If you watched the demo (or read the quickstart Sinatra example), there's a demo app called 'env' that they walk you through. You're going to want to use that when troubleshooting things. My first task was to duplicate the env demo so I could take a gander at the variables I would need for Redis. For the record, the steps I'm documenting here might appear out of order and result in some wasted time. I'm one of those guys who reads the instructions 2 days after I've broken something so you have an idea of what I did here:

 vmc help
 vmc services
 vmc create-service redis redis-noah
 vmc services

At this point, I now have a named instance of redis. The reason I felt safe enough doing this now is that I noticed in the help two service commands - 'bind-service' and 'unbind-service'. I figured it was easy enough to add the service to my app based on those options.

So go ahead and create the env app per the getting started documentation. If you followed my suggestion and DIDN'T disable prompts, you'll get the option to bind you app to a service when you push the first time. If you're running without prompts (using the '-n' option), you'll probably want to do something like this:

vmc push myenvapp --url ohai-env.cloudfoundry.com
vmc bind-service my_redis_service myenvapp

If you visit the url you provided (assuming it wasn't taken already?) at /env, you'll get a big dump of all the environment variables. The ones that you'll need be using most are probably going to be under `VCAP_SERVICES`. What you'll probably also notice is that `VCAP_SERVICES` is a giant JSON blob. Now you may also notice that there's a nice `VMC_REDIS` env variable there. It's pretty useless primarily because there's also a GIANT warning in the env output that all `VMC_` environment variables are deprecated but also because your redis instance requires a password to access which means you need to traverse the JSON blob ANYWAY.

So if we paste the blog into an IRB session we can get a better representation. I wish I had done that first. Instead, I reformatted it with jsonlint dutifully wrote the following madness:

which I spent a good 30 minutes troubleshooting before I realized that it's actually an array. It should have been this:


So now that I had all the variables in place, I went about converting my heroku Noah demo . That demo uses a Gemfile and a rackup file so I figured it would work just fine here. No such luck. This is where things get hairy.

Sinatra limitations
The short of it is that Sinatra application support on CF right now is a little bit of a CF. It's very basic and somewhat brute force. If you're running a single file sinatra application, it will probably work. However if you're running anything remotely complex, it's not going to work without considerable effort. Noah is even more of a special case because it's distributed as a gem. This actually has some benefit as I'll mention farther down. However it's not really "compatible" with the current setup on Cloud Foundry. Here's the deal:

If you look here, You'll see that the way your sinatra application is start is by calling ruby (with or without bundler depending) against what it detects as your main app file. This is done here which leads us all the way to this file:

`https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/manifests/sinatra.yml`

Essentially for sinatra applications, the first .rb file it comes across with 'require sinatra', is considered the main app file. Bummer. So config.ru is out. The next step is to rename it to a '.rb' file and try again. This is where I spent most of my troubleshooting. There's a gist of the things I tried (including local testing) here:

`https://gist.github.com/920552`

Don't jump to the solution just yet because it's actually incomplete. This troubleshooting led to another command you'll want to remember:

vmc files myapp logs/stderr.log

I found myself typing it a lot during this process. For whatever reason, possibly due to bundler or some other vcap magic I've not discovered yet what works at home does not work on Cloud Foundry exactly the same. That's fine, it's just a matter of knowing about it. It also didn't help that I wasn't getting any output at all for the entire time I was trying to figure out why config.ru didn't work.

Thanks to Konstantin Haase for his awesome suggestion in #sinatra. The trick here was to mimic what rackup does. Because the currently released Noah gem has a hard requirement on rack 1.2.1, his original suggestion wasn't an exact fit but I was able to get something working:

https://gist.github.com/921292

So what did we do?
Ensure that the wrapper file is picked up first by making sure it's the ONLY rb file uploaded with `require sinatra` at the top.
Because of a bug in rack 1.2.1 with Rack::Server.new, I HAD to create a file called config.ru. The fix in rack 1.2.2 actually honors passing all the options into the constructor without needing the config.ru file.
Explicitly connect to redis before we start the application up.

The last one was the almost as big of a pain in the ass as getting the application to start up.

I think (and I'm not 100% sure) that you are prohibited from setting environment variables inside your code. Because of the convoluted way I had to get the application started, I couldn't use my sinatra configuration block properly (`set :redis_url, blahblahblah`). I'm sure it's possible but I'm not an expert at rack and sinatra. I suppose I could have used Noah::App.set but at this point I was starting to get frustrated. Explicitly setting it via Ohm.connect worked.

I'm almost confident of this environment variable restriction because you can see options in 'vmc help' that allow you to pass environment variables into your application. That would work fine for most cases except that I don't know what the redis values are outside of the app and they're set dynamically anyway.

So where can things improve?
First off, this thing is in beta. I'm only adding this section because it'll serve as a punch list of bugs for me to fix in vcap ;)

• Sinatra support needs to be more robust.

You can see that the developers acknowledged that in the staging plugin code. There are TODOs listed. It's obvious that a sinatra application of any moderate complexity wasn't really tested and that's fine. The building blocks are there and the code is opensource. I'll fix it myself (hopefully) and submit a pull request.

• Allow override of the main app file from VMC.

It appears from the various comments that the node.js support suffers some of the same brute force detection routines. An option to pass in what the main applictation file is would solve some of that.

• Document the environment variable restrictions.

I didn't see any documentation anywhere about that restriction (should it exist). I could be doing something wrong too. It's worth clarifying.

• Better error reporting for failed startups

I'm not going to lie but I spent a LONG time troubleshooting the fact that the app simply wasn't starting up. The default output when a failure happens during deploy is the staging.log file. All this EVER contained was the output from bundler. It should include the output of stderr.log and stdout.log as well. Also an explicit message should be returned if the main app file can't be detected. That would have solved much of my frustration up front.

That's just the stuff I ran into to get things going. The first item is the biggest one. If you're writing a monolithic single-file sinatra app, the service will work GREAT. If you aren't, you'll have to jump through hoops and wrapper scripts for now. Supporting rackup files for Sinatra and Rack apps will go a long way to making things even more awesome.

One pleasant surprise I found was that, despite what I was told, I didn't need to include every gem in my Gemfile. Because Noah itself has its deps, Bundler pulls those in for me.

I've created a git repo with the code as well as a quickstart guide for getting your own instance running. You can find it here:

https://github.com/lusis/noah-cloudfoundry-demo

Operational Primitives

"Infrastructure as code". I love the phrase. Where devops is a word that is sadly open to so much (mis)interpretation, "Infrastructure as code" is pretty clear. Treat your infrastructure as code. Programmable. Testable. Deployable.

But when you start to really think about that concept, there's a deep dive you can take, navigating various programming and computer science constructs and applying those to your infrastructure.

I've been working pretty heavily on getting the first API stable release of Noah out the door. It's been a challenge with the schedule I have to work on it - which is essentially "when everyone else in the house is asleep and I'm awake'. Last night, I came to a fork in the road where I needed to make a decision. This decision would lock me into an API path that I was unwilling to change for a while. Nobody wants to use a service or tool with a constantly changing API. I needed to shit or get off the pot, to use a creative euphemism. With the announcements of both Doozer and riak_zab, it was clear that I wasn't the only person attempting to tackle the ZooKeeper space.

Since Github lacks any facility for soliciting project feedback (hint hint, @github), I decided to create a  Wufoo form and tweet it out. I don't have a very big audience but I was hoping it would at least get to the people who were likely to use Noah. The form was fairly simple with one question on something that I had pretty summarily dismissed early on - HATEOAS (hypermedia as the engine of application state).

A small HATEOAS diversion


The HATEOAS debate is a lot like Linux vs. GNU/Linux. It's fairly esoteric but there's some meat to the matter. My problem with it was simply that, despite what Roy Fielding and others intended, REST had taken on a new definition and it wasn't the strict HATEOAS one. Additionally, I found it VERY difficult to map HATEOAS concepts to JSON. JSON is a great format but a rich document structure is not (rightly so) part of the format. It's intended to be simple, easily read and cleanly mapped to machine readable format. It also felt like extra work on the part of the API consumer. The concepts that we use when reading a website (click this link, read this list, click this link) are simple not necessary when you have a contextually relevant (or descriptive) URL scheme. True, as a human I don't make changes in the URL bar to navigate a site (I use the links provided by the site) but when it comes to dealing with an API, I don't exhibit the same usage patterns as a web browser. I'm making distinct atomic transactions (DELETE this resource, PUT this resource) at a given endpoint. These simply aren't the same as filling out forms and are only tangentially related. I'm simply not willing to force someone to parse a JSON object to tell them how to create a new object in the system. The API for Noah is fairly simple as it is. Objects in the system have only two or three required attributes for a given operation and normally one of those attributes is directly inferable from the URL.

But based on the poll results thus far, I wanted to give the idea fair consideration which led me to think about what types of objects Noah had in its system.

Primitives


For those who aren't familiar or simple don't know, there's a term in computer science and programming called "Primitive". It essentially means a basic data type in a language from which other complex data types are created. A building block if you will. Some easily grokable examples of primitives are Characters and Integers. Some languages actually have ONE primitive like Object and everything is built on top of that. You could get into a semantic argument about a lot of this so I'm going to leave it at that.

But back to the phrase "Infrastucture as code". If we start looking at how we "program" our infrastructure, what are the "primitives" that our language supports. I inadvertently created some of these in Noah. I've been calling them the "opinionated models" but really in the infrastructure programming language of Noah, they're primitives.

When this hit me last night, I immediately pulled out the tablet and went to work on a mind map. I laid out what I had already implemented as primitives in Noah:

• Host
• Service
• Application
• Configuration

I then started to think about other concepts in Noah. Were Ephmerals really a primitive. Not really. If anything Ephemerals are more similar to ruby's BasicObject. The only real attribute Ephemerals have are a path (similar to the object_id).

So what else would be our modern operational primitives? Remember that we're talking about building blocks here. I don't want to abstract out too much. For instance you could simply say that a "Resource" is the only real operational primitive and that everything else is built on top of that.  Also consider that languages such as Python have some richer primitives built-in like tuples.

One interesting thought I had was the idea that "State" was a primitive. Again, in the world of operations and infrastructure, one of your basic building blocks is if something is available or not - up or down. At first glance it would appear that this maps pretty cleanly to a Boolean (which is a primitive in most languages) however I think it's a richer primitive than that.

In the world of operations, State is actually quaternary (if that's the right word) rather than binary. There are two distinct areas between up and down that have dramatically different implications on how you interact with it:

• Up
• Down
• Pending Up
• Pending Down

Currently in Noah, we simple have Up, Down and Pending but something that is in the State of shutting down is grossly different than something in the state of Starting up. Look at a database that is queiscing connections. It's in a state of "Pending Down". It's still servicing existing requests. However a database in the state of "Pending Up" is NOT servicing any requests.

So I'm curious what other thoughts people have. What else are the basic building blocks of modern operations when viewed through the lens of "infrastructure as code"?

For the record, I'm still pretty confident that Noah still has a place in the Doozer, riak_zab, ZooKeeper world. All three of those still rely on the persistent connection for signaling and broadcast whereas Noah is fundamentally about the disconnected and asynchronous world.

It does not follow and Wheaton's Law

"I'm not a smart guy".

I say this quite a bit. I don't say it to fish for compliments or as a chance to have my ego boosted. I say it because I realize that, out of the vast corpus of computer science knowledge that exists, the part that I DO know is a blade of grass on a football field.

"I'm not a developer"

I say this a lot too. This is not meant as a slight to developers. It's meant as a compliment. There are REAL developers out there and I'm just pretending (after a fashion). I have never worked a professional gig as a developer. I've had honest discussions with people who want to pay me lots of money to be a developer. The best way I can explain it to them is that it would be unfair to you, as an employer, to hire me for a developer position because you would be unhappy with the results. In general it takes me twice as long to solve a development problem as it takes a real developer.

There are lots of factors to this; education, regular skill use and a general affinity for picking up concepts. I never graduated college and I pretty much suck at math. That's not to say I couldn't learn it but there are some things I know I'll never be as good at as someone else and that's fine by me. I'm not settling for mediocrity I just know my limitations. I'll still take a stab at it.

There are, however, some REALLY smart people out there. I used to follow a bunch of them on Twitter because they would link to or drop ideas that really made me want to go research something. I noticed an interesting trend though about some of them. They had a tendency to be dicks. Not just the occasional "Only an idiot would do X" but outright vitriol. Was it trolling? In some cases, sure, but I honestly got the impression that they actually looked down on people who didn't who use a certain technology or chose any path different than they would have chosen.

At the other extreme, you have the folks who make snide remarks or drop a non sequitur about a given technology presumably in an attempt to make the in-crowd giggle and the rest of us poor saps wonder what the hell we're doing wrong. I mean these are smart people, right? If they know something I don't about a given technology, then by god, I'd love to know what it is. I'd love to learn why they feel that way. In the end, though, all you hear is giggling in the background and wonder what the big joke was.

When the hell did we, the people who were typically on the outside of the in-crowd, turn into the people who gave us the most shit growing up? It's like a fucking geek Stockholm Syndrome thing that's gone off the deep end but instead of just sympathizing with our abuser, we're the abuser and we relish it.

I'm guilty of this behavior. I'm the first in line to criticize MongoDB, for instance. The difference? I'll actually sit down with you and tell you WHY I don't like MongoDB and why I feel it's a bad choice in many situations.

What I'm asking is that, as one of the people on the outside, educate me. As much as I think Ted Dziuba is a big troll, at least he takes the time to write it down and trys to defend his position. Ben Bleything had an awesome tweet today:

I guess what I meant is, I don't have the experience to form that opinion, I'd like to learn from you.

That's my attitude exactly. "Put up or shut up" is a bit harsh but in the broadest terms, that's what needs to happen. If you think X is superior to Z then say why. There are some of us who could benefit from it.

Sidebar on Semantics


Additionally,  let's make sure we're also on the same page in terms of semantics. If we're talking about queues, clarify if you're talking about data structures versus a message queue because there's a big f'ing difference in my mind.

When I hear queue, I don't think data structure. I think of a message queue in the product sense. That's just my background. I think about things like guaranteed delivery and message durability.