Relocating again.

FYI, I'm relocating my blog back to my own domain. All the new posts will go there and eventually I'll get everything pulled in from here.

New Old Place is here: http://blog.lusis.org

Monitoring Sucks - Watch your language

The following post is a recap of what was discussed in the 07/21/11 #monitoringsucks irc meeting

Before I get any further, I just want to thank everyone who attended, either in virtual person or in spirit. There have been so many awesome discussions going around this topic since we started. I am truly priviledged to interact with each and every one of you. I struggle to count myself a peer and I can only hope that I provide something in return.

I mentioned to someone today that I’m literally sick of the current landscape. I consider the current crop of monitoring solutions to be technologically bankrupt. The situation is fairly untenable at this point.

I just installed (after having a total loss of an existing Zenoss setup) Nagios again. I’m not joking when I say that it depressed the everliving hell out of me. The monitoring landscape simply has not kept up with modern developments. At first it was mildly frustrating. Then it was annoying. Now it’s actually a detriment.

Now that we have that out of the way….

Darmok and Jalad at Tanagra

Communication is important. Like Picard and Dathon, we’ve been stranded on a planet with shitty monitoring tools and we’re unable to communicate about the invisibile threat of suck because we aren’t even speaking the same language. I say event, you hear trigger, I mean data point. So the first order of business was to try and agree on a set of terms. It was decided that we would consider these things primitives. Here they are:

Please read through this before jumping to any conclusions. I promise it will all become clear (as mud).

metric

a numeric or boolean data point

The data type of a metric was something of a sticking point. People were getting hung up on data points being various things (a log message, a “status”, a value). We needed something to describe the origin. The single “thing” that triggered it all. That thing is a metric.

So why numeric OR boolean? It was pretty clear that many people considered, and rightly so I would argue, that a state change is a metric. A good example given by Christopher Webber is that of a BGP route going away. Why is this a less valid data point than the amount of disk space in use or the latency from one host to another? Frankly, it’s not.

But here’s where it gets fuzzy. What about a log message. Surely that’s a data point and thus a metric.

Yes and No. The presence of a log message is a data point. But it’s a boolean. The log message itself?

context

metadata about a metric

Now metadata itself is a loaded term but in this scope, the “human readable” attributes are considered context. Going back to our log example. The presence of the log message is a metric. The log message itself is context. Here’s the thing. You want to know if there is an error message in a log file. The type of error, the error message text? That’s context for the metric to use in determining a course of action.

Plainly speaking, metrics are for machines. Context is for humans. This leads us to….

event

metric combined with context

This is still up in the air but the general consensus was that this was a passable definition. The biggest problem with a group of domain experts is that they are frequently unable to accept semantic approximation. Take the discussion of Erlang Spawned process:

• It’s sort of like a VM on a VM
• NO IT’S NOT.
• headdesk

The fact is that an Erlang spawned process has shades of a virtual machine is irrelevant to the domain expert. We found similar discussions around what we would call the combination of a metric and its context. But where do events come from?

resource

the source of a metric

Again, we could get into arguments around what a resource is. One thing that was painfully obvious is that we’re all sick and tired of being tied to the Host and Service model. It’s irrelevant. These constructs are part “legacy” and part “presentation”.

Any modern monitoring thought needs to realize that metrics no longer come from physical hosts or are service states. In the modern world, we’re taking a holistic view of monitoring that includes not only bare metal but business matters. The number of sales is a metric but it’s not tied to a server. It’s tied to the business as a whole. The source of your metrics is a resource. So now that we have this information - a metric, its context and who generated it - what do we do? We take….

action

a response to a given metric

What response? It doesn’t MATTER. Remember that these are primitives. The response is determined by components of your monitoring infrastructure. Humans note the context. Graphite graphs it. Nagios alerts on it. ESPER correlates it with other metrics. Don’t confuse scope here. From this point on, whatever happens has is all decided on by a given component. It’s all about perspective and aspects.

Temba, his arms wide

I’m sure through much of that, you were thinking “alerting! graphing! correlation!”. Yes, that was pretty much what happened during the meeting as well. Everyone has pretty much agreed (I think) at this point that any new monitoring systems should be modular in nature. As Jason Dixon put it - “Voltron”. No single system that attempts to do everything will meet everyone’s needs. However, with a common dictionary and open APIs you should be able to build a system that DOES meet your needs. So what are those components? Sadly this part is not as fleshed out. We simply ran out of time. However we did come up with a few basics:

Collection

getting the metrics

It doesn’t matter if it’s push or pull. It doesn’t matter what the transport is - async or point-to-point. Somehow, you have to get a metric from a resource.

Event Processing

taking action

Extract the metric and resource from an event. Do something with it. Maybe you send the metric to another component. Maybe you “present” it or send it somewhere to be presented. Maybe you perform a change on a resource (restarting a service). Essentially the decision engine.

Presentation

While you might be thinking of graphing here, that’s a type of presentation. You know what else is presentation? An email alert. Stick with me. I know what’s going through your head. No..not that…the other thing.

Analytics

This is pretty much correlation. We didn’t get a REAL solid defintion here but everyone was in agreement that some sort of analytics is a distinct component.

The “other” stuff

As I said, we had to kind of cut “official” things short. There was various discussion around Storage and Configuration. Storage I personally can see as a distinct component but Configuration not so much. Configuration is an aspect of a component but not a component itself.

Logical groupings

Remember when I said I know what you’re thinking? This is what I think it was.

You can look at the above items and from different angles they look similar. I mean sending an email feels more like event processing than presentation. You’d probably be right. By that token, drawing a point on a graph is technically processing an event. The fact is many components have a bit of a genetic bond. Not so much parent/child or sibling but more like cousins. In all honesty, if I were building an event processing component, I’d probably handle sending the email right there. Why send it to another component? That makes perfect sense. Graphing? Yeah I’ll let graphite handle that but I can do service restarts and send emails. Maybe you have an intelligent graphing component that can do complex correlation inband. That makes sense too.

I’m quite sure that we’ll have someone who writes a kickass event processor that happens to send email. I’m cool with that. I just don’t want to be bound to ONLY being able to send emails because that’s all your decision system supports.

Shaka, when the walls fell

Speaking personally, I really feel like today’s discussion was VERY productive. I know that you might not agree with everything here. Things are always up for debate. The only thing I ask is that at some point, we’re all willing to say “I know that this definition isn’t EXACTLY how I would describe something but it’s close enough to work”.

So what are the next steps? I think we’ve got enough information and consensus here for people to start moving forward with some things. One exercise, inspired by something Matt Ray said, that we agreed would be REALLY productive is to take an existing application and map what it does to our primitives and components. Matt plans on doing that with Zenoss since that’s what he knows best.

Let me give an example:

Out of the box, Nagios supports Hosts and Services which map pretty cleanly to resources. It is does not only collection but event processing and presentation. It not only supports metrics but also context (Host down is the boolean metric. “Response timeout” is the context. Through something like pnp4nagios, it can support different presentations. It has very basic set of Analytic functionality.

Meanwhile Graphite is, in my mind, strictly presentation and deals only with metrics. It does support both numeric and boolean metrics. It also has basic resource functionality but it’s not hardwired. It doesn’t really do event handling in the strict sense. Analytics is left to the human mind. It certainly doesn’t support context.

I’d love to see more of these evaluations.

Also, I know there are tons of “words” that we didn’t cover - thresholds for instance. While there wasn’t a total consensus, there was some agreement that somethings were attributes of a component but not a primitive itself. It was also accepted that components themselves would be primitives. You correlation engine might aggregate (another word) a group of metrics and generate an event. At that point, your correlation engine is now a resource with its own metrics (25 errors) and context (“number of errors across application servers exceeded acceptable limits”) which could be then sent to an event processor.

That’s the beauty of the Voltron approach and not binding a resource to a construct like a Host.

Special note to the Aussies

I’m very sorry that we couldn’t get everyone together. I’ve scheduled another meeting where we can start from scratch just like this one, or build on what was discussed already. I’m flexible and willing to wake up at whatever time works best for you guys

Thanks again to everyone who attended. If you couldn’t be there, I hope you can make the next one.

Monitoring sucks - Round 2 - FIGHT

Recently there’s been more and more attention to the whole ##monitoringsucks campaign. Peopel are writing blog posts. Code is being written. Discussions are being had. Smart people are doing smart things.

meetingssuck

I figure it’s time to get the band back together for another official irc session and stop flooding infra-talk with random rants about where things stand. As much as I hate to even consider it, I think having something of an agenda so that it doesn’t turn into a bitch fest.

I’m open to any ideas that people have.

Jordan Sissel brought up some things that might be nice to have:

• use cases
• primitives

I’d like to also love to get input from others before we settle on the details. This one is going to be a bit more “formal” than the first session. Don’t let that turn you off. Everyone has something valuable to contribute.

Deliverables

I can’t believe I’m actually suggesting this part but I’d also love to walk away with something that people can work with. It’s not like people will have homework or a task list.

This is more of a “this is what we would like to see, if you want to write some code it’s a decent place to start”.

International flavor

I’m also keen on finding a time when we can get as many folks as possible to contribute. This may not be possible with that whole rotation of the earth thing but we’ll see.

Suggestions, Comments and Rude Remarks

If you’ve got anything you’d like to see or contribute on the planning side, drop me a message on IRC, email or twitter and I’ll include it (if possible) when I draft up the final “notes” beforehand.

Why Monitoring Sucks

Why Monitoring Sucks (and what we're doing about it)

About two weeks ago someone made a tweet. At this point, I don't remember who said it but the gist was that "monitoring sucks". I happened to be knee-deep in frustrating bullshit around that topic and was currently evaluating the same effing tools I'd evaluated at every other company over the past 10 years or so. So I did what seems to be S.O.P for me these days. I started something.

But does monitoring REALLY suck?

Heck no! Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: - Having my hands tied with the model of host and service bindings. - Having to set up "fake" hosts just to group arbitrary metrics together - Having to either collect metrics twice - once for alerting and another for trending - Only being able to see my metrics in 5 minute intervals - Having to chose between shitty interface but great monitoring or shitty monitoring but great interface - Dealing with a monitoring system that thinks IT is the system of truth for my environment - Perl (I kid...sort of) - Not actually having any real choices

Yes, yes I know:

You can just combine Nagios + collectd + graphite + cacti + pnp4nagios and you have everything you need!

Seriously? Kiss my ass. I'm a huge fan of the Unix pipeline philosophy but, christ, have you ever heard the phrase "antipattern"?

So what the hell are you going to do about it?

I'm going to let smart people be smart and do smart things.

Step one was getting everyone who had similar complaints together on IRC. That went pretty damn well. Step two was creating a github repo. Seriously. Step two should ALWAYS be "create a github repo". Step three? Hell if I know.

Here's what I do know. There are plenty of frustrated system administrators, developers, engineers, "devops" and everything under the sun who don't want much. All they really want is for shit to work. When shit breaks, they want to be notified. They want pretty graphs. They want to see business metrics along side operational ones. They want to have a 52-inch monitor in the office that everyone can look at and say:

See that red dot? That's bad. Here's what was going on when we got that red dot. Let's fix that shit and go get beers

About the "repo"

So the plan I have in place for the repository is this. We don't really need code. What we need is an easy way for people to contribute ideas. The plan I have in place for this is partially underway. There's now a monitoringsucks organization on Github. Pretty much anyone who is willing to contribute can get added to the team. The idea is that, as smart people think of smart shit, we can create new repository under some unifying idea and put blog posts, submodules, reviews, ideas..whatever into that repository so people have an easy place to go get information. I'd like to assign someone per repository to be the owner. We're all busy but this is something we're all highly interested in. If we spread the work out and allow easy contribution, then we can get some real content up there.

I also want to keep the repos as light and cacheable as possible. The organization is under the github "free" plan right now and I'd like to keep it that way.

Blog Posts Repo

This repo serves as a place to collect general information about blog posts people come across. Think of it as hyper-local delicious in a DVCS.

Currently, by virtue of the first commit, Michael Conigliaro is the "owner". You can follow him on twitter and github as @mconigliaro

IRC Logs Repo

This repo is a log of any "scheduled" irc sessions. Personally, I don't think we need a distinct #monitoringsucks channel but people want to keep it around. The logs in this repo are not full logs. Just those from when someone says "Hey smart people. Let's think of smart shit at this date/time" on twitter.

Currently I appear to be the owner of this repo. I would love for someone who can actually make the logs look good to take this over.

Tools Repo

This repo is really more of a "curation" repo. The plan is that each directory is the name of some tool with two things it in:

• A README.md as a review of the tool
• A submodule link to the tool's repo (where appropriate)

Again, I think I'm running point on this one. Please note that the submodule links APPEAR to have some sort of UI issue on github. Every submodule appears to point to Dan DeLeo's 'critical' project.

Metrics Catalog Repo

This is our latest member and it already has an official manager! Jason Dixon (@obfuscurity on github/twitter - jdixon on irc) suggested it so he get's to run it ;) The idea here is that this will serves as a set of best practices around what metrics you might want to collect and why. I'm leaving the organization up to Jason but I suggested a per-app/service/protocol directory.

Wrap Up

So that's where we are. Where it goes, I have no idea. I just want to help where ever I can. If you have any ideas, hit me up on twitter/irc/github/email and let me know. It might help to know that if you suggest something, you'll probably be made the person reponsible for it ;)

Update!

It was our good friend Sean Porter (@portertech on twitter), that we have to thank for all of this ;)

From Public Photos

Update (again)

It was kindly pointed out that I never actually included a link to the repositories. Here they are:

https://github.com/monitoringsucks

On Noah - Part 4

This is the fourth part in a series on Noah. Part 1, Part 2 and Part 3 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. Part 3 was about the components underneath Noah that make it tick.

This post is about the "future" of Noah. Since I'm a fan of Fourcast podcast, I thought it would be nice to do an immediate, medium and long term set of goals.

Immediate Future - the road to 1.0

In the most immediate future there are a few things that need to happen. These are in no specific order.

• General
  
  • Better test coverage ESPECIALLY around the watch subsystem
  • Full code comment coverage
  • Chef cookbooks/Puppet manifests for doing a full install
  • "fatty" installers for a standalone server
  • Documentation around operational best practices
  • Documentation around clustering, redundancy and hadr
  • Documentation around integration best practices
  • Performance testing
• Noah Server
  
  • Expiry flags and reaping for Ephemerals
  • Convert mime-type in Configurations to make sense
  • Untag and Unlink support
  • Refactor how you specify Redis connection information
  • Integrated metrics for monitoring (failed callbacks, expired ephemeral count, that kind of stuff)
• Watcher callback daemon
  
  • Make the HTTP callback plugin more flexible
  • Finish binscript for the watcher daemon
• Other
  
  • Finish Boat
  • Finish NoahLite LWRP for Chef (using Boat)
  • A few more HTTP-based callback plugins (Rundeck, Jenkins)

Now that doesn't look like a very cool list but it's a lot of work for one person. I don't blame anyone for not getting excited about it. The goal now is to get a functional and stable application out the door that people can start using. Mind you I think it's usable now (and I'm already using it in "production").

Obviously if anyone has something else they'd like to see on the list, let me know.

Medium Rare

So beyond that 1.0 release, what's on tap? Most of the work will probably occur around the watcher subsystem and the callback daemon. However there are a few key server changes I need to implement.

• Server
  
  • Full ACL support on every object at every level
  • Token-based and SSH key based credentialing
  • Optional versioning on every object at every level
  • Accountability/Audit trail
  • Implement a long-polling interface for inband watchers
• Watcher callback daemon
  
  • Decouple the callback daemon from the Ruby API of the server. Instead the daemon itself needs to be a full REST client of the Noah server
  • Break out the "official" callback daemon into a distinct package
• Clients
  
  • Sinatra Helper

Also during this period, I want to spend time building up the ecosystem as a whole. You can see a general mindmap of that here.

Going into a bit more detail...

Tokens and keys

It's plainly clear that something which has the ability to make runtime environment changes needs to be secure. The first thing to roll off the line post-1.0 will be that functionality. Full ACL support for all entries will be enabled and in can be set at any level in the namespace just the same as Watches.

Versioning and Auditing

Again for all entires and levels in the namespace, versioning and auditing will be allowed. The intention is that the number of revisions and audit entries are configurable as well - not just an enable/disable bit.

In-band watches

While I've lamented the fact that watches were in-band only in Zookeeper, there's a real world need for that model. The idea of long-polling functionality is something I'd actually like to have by 1.0 but likely won't happen. The intent is simply that when you call say /some/path/watch, you can pass an optional flag in the message stating that you want to watch that endpoint for a fixed amount of time for any changes. Optionally a way to subscribe to all changes over long-polling for a fixed amount of time is cool too.

Agent changes

These two are pretty high on my list. As I said, there's a workable solution with minimal tech debt going into the 1.0 release but long term, this needs to be a distinct package. A few other ideas I'm kicking around are allowing configurable filtering on WHICH callback types an agent will handle. The idea is that you can specify that this invocation only handle http callbacks while this other one handles AMQP.

Sinatra Helper

One idea I'd REALLY like to come to fruition is the Sinatra Helper. I envision it working something like this:

    require 'sinatra/base'

class MyApp < Sinatra::Base
register Noah::Sinatra

noah_server "http://localhost:5678"
noah_node_name "myself"
noah_app_name "MyApp"
noah_token "somerandomlongstring"

dynamic_get :database_server
dynamic_set :some_other_variable, "foobar"
watch :this_other_node

end

The idea is that the helper allows you to register your application very easily with Noah for other components in your environment to be know. As a byproduct, you get the ability to get/set certain configuration parameters entirely in Noah. The watch setting is kind of cool as well. What will happen is if you decide to watch something this way, the helper will create a random (and yes, secure) route in your application that watch events can notify. In this way, your Sinatra application can be notified of any changes and will automatically "reconfigure" itself.

Obviously I'd love to see other implementations of this idea for other languages and frameworks.

Long term changes

There aren't so much specific list items here as general themes and ideas. While I list these as long term, I've already gotten an offer to help with some of them so they might actually get out sooner.

Making Noah itself distributed

This is something I'm VERY keen on getting accomplished and would really consider it the fruition of what Noah itself does. The idea is simply that multiple Noah servers themselves are clients of other Noah servers. I've got several ideas about how to accomplish this but I got an interesting follow up from someone on Github the other day. He asked what my plans were in this area and we had several lengthy emails back and forth including an offer to work on this particular issue.

Obviously there are a whole host of issues to consider. Race conditions in ordered delivery of Watch callbacks (getting a status "down" after a status "up" when it's supposed to be the other way around..) and eventual consistency spring to mind first.

The general architecture idea that was offered up is to use NATS as the mechanism for accomplishing this. In the same way that there would be AMQP callback support, there would be NATS support. Additional Noah servers would only need to know one other member to bootstrap and everything else happens using the natural flows within Noah.

The other part of that is how to handle the Redis part. The natural inclination is to use the upcoming Redis clustering but that's not something I want to do. I want each Noah server to actually include its OWN Redis instance "embedded" and not need to rely on any external mechanism for replication of the data. Again, the biggest validation of what Noah is designed to do is using only Noah itself to do it.

Move off Redis/Swappable persistence

If NATS says anything to me, it says "Why do you even need Redis?". If you recall, I went with Redis because it solved multiple problems. If I can find a persistence mechanism that I can use without any external service running, I'd love to use it.

ZeroMQ

If I were to end up moving off Redis, I'd need a cross platform and cross language way to handle the pubsub component. NATS would be the first idea but NATS is Ruby only (unless I've missed something). ZeroMQ appears to have broad language and platform support so writing custom agents in the same vein as the Redis PUBSUB method should be feasible.

Nanite-style agents

This is more of a command-and-control topic but a set of high-performance specialized agents on systems that can watch the PUBSUB backend or listen for callbacks would be awesome. This would allow you really integrate Noah into your infrastructure beyond the application level. Use it to trigger a puppet or chef run, reboot instances or do whatever. This is really about bringing what I wanted to accomplish with Vogeler into Noah.

The PAXOS question

A lot of people have asked me about this. I'll state right now that I can only make it through about 20-30% of any reading about Paxos before my brain starts to melt. However in the interest of proving myself the fool, I think it would be possible to implement some Paxos like functionality on top of Noah. Remember that Noah is fundamentally about fully disconnected nodes. What better example of a network of unreliable processors than ones that never actually talk to each other. The problem is that the use case for doing it in Noah is fairly limited so as not to be worth it.

The grand scheme is that Noah helps enable the construction of systems where you can say "This component is free to go off and operate in this way secure in the knowledge that if something it needs to know changes, someone will tell it". I did say "grand" didn't I? At some point, I may hit the limit of what I can do using only Ruby. Who knows.

Wrap up - Part 4

Again with the recap

• Get to 1.0 with a stable and fixed set of functionality
• Nurture the Noah ecosystem
• Make it easy for people to integrate Noah into thier applications
• Get all meta and make Noah itself distributed using Noah
• Minimize the dependencies even more
• Build skynet

I'm not kidding on that last one. Ask me about Parrot AR drones and Noah sometime

If you made it this far, I want to say thank you to anyone who read any or all of the parts. Please don't hesitate to contact me with any questions about the project.

On Noah - Part 3

This is the third part in a series on Noah. Part 1 and Part 2 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. This post is discussing the technology stack under Noah and the reasoning for it.

A little back story

I've told a few people this but my original intention was to use Noah as a way to learn Erlang. However this did not work out. I needed to get a proof of concept out much quicker than the ramp up time it would take to learn me some Erlang. I had this grandiose idea to slap mnesia, riak_core and webmachine into a tasty ball of Zookeeper clonage.

I am not a developer by trade. I don't have any formal education in computer science (or anything for that matter). The reason I mention this is to say that programming is hard work for me. This has two side effects:

• It takes me considerably longer than a working developer to code what's in my head
• I can only really learn a new language when I have an itch to scratch. A real world problem to model.

So in the interest of time, I fell back to a language I'm most comfortable with right now, Ruby.

Sinatra and Ruby

Noah isn't so much a web application as it is this 'api thing'. There's no proper front end and honestly, you guys don't want to see what my design deficient mind would create. I like to joke that in the world of MVC, I stick to the M and C. Sure, APIs have views but not in the "click the pretty button sense".

I had been doing quite a bit of glue code at the office using Sinatra (and EventMachine) so I went with that. Sinatra is, if you use sheer number of clones in other languages as an example, a success for writing API-only applications. I also figured that if I wanted to slap something proper on the front, I could easily integrate it with Padrino.

But now I had to address the data storage issue.

Redis

Previously, as a way to learn Python at another company, I wrote an application called Vogeler. That application had a lot of moving parts - CouchDB for storage and RabbitMQ for messaging.

I knew from dealing with CouchDB on CentOS5 that I wasn't going to use THAT again. Much of it would have been overkill for Noah anyway. I realized I really needed nothing more than a key/value store. That really left me with either Riak or Redis. I love Riak but it wasn't the right fit in this case. I needed something with a smaller dependency footprint. Mind you Riak is VERY easy to install but managing Erlang applications is still a bit edgy for some folks. I needed something simpler.

I also realized early on that I needed some sort of basic queuing functionality. That really sealed Redis for me. Not only did it have zero external dependencies, but it also met the needs for queuing. I could use lists as dedicated direct queues and I could use the built-in pubsub as a broadcast mechanism. Redis also has a fast atomic counter that could be used to approximate the ZK sequence primitive should I want to do that.

Additionally, Redis has master/slave (not my first choice) support for limited scaling as well as redundancy. One of my original design goals was that Noah behave like a traditional web application. This is a model ops folks understand very well at this point.

EventMachine

When you think asynchronous in the Ruby world, there's really only one tool that comes to mind, EventMachine. Noah is designed for asynchronous networks and is itself asynchronous in its design. The callback agent itself uses EventMachine to process watches. As I said previously, this is simply using an EM friendly Redis driver that can do PSUBSCRIBE (using em-hiredis) and send watch messages (using em-http-request since we only support HTTP by default).

Ohm

Finally I slapped Ohm on top as the abstraction layer for Redis access. Ohm, if you haven't used it, is simply one of if not the best Ruby library for working with Redis. It's easily extensible, very transparent and frankly, it just gets the hell out of your way. A good example of this is converting some result to a hash. By default, Ohm only returns the id of the record. Nothing more. It also makes it VERY easy to drop past the abstraction and operate on Redis directly. It even provides helpers to get the keys it uses to query Redis. A good example of this is in the Linking and Tagging code. The following is a method in the Tag model:

    def members=(member)
      self.key[:members].sadd(member.key)
      member.tag! self.name unless member.tags.member?(self)
    end

Because Links and Tags are a one-to-many across multiple models, I drop down to Redis and use sadd to add the object to a Redis set of objects sharing the same tag.

It also has a very handy feature which is how the core of Watches are done. You can define hooks at any phase of Redis interaction - before and after saves, creates, updates and deletes. the entire Watch system is nothing more than calling these post hooks to format the state of the object as JSON, add metadata and send the message using PUBLISH messages to Redis with the Noah namespace as the channel.

Distribution vectors

I've used this phrase with a few people. Essentially, I want as many people as possible to be able to use the Noah server component. I've kept the Ruby dependencies to a minimum and I've made sure that every single one works on MRI 1.8.7 up to 1.9.2 as well as JRuby. I already distribute the most current release as a war that can be deployed to a container or run standalone. I want the lowest barrier to entry to get the broadest install base possible. When a new PaaS offering comes out, I pester the hell out of anyone I can find associated with it so I can get deploy instructions written for Noah. So far you can run it on Heroku (using the various hosted Redis providers), CloudFoundry and dotcloud.

I'm a bit more lax on the callback daemon. Because it can be written in any language that can talk to the Redis pubsub system and because it has "stricter" performance needs, I'm willing to make the requirements for the "official" daemon more stringent. It currently ONLY works on MRI (mainly due to the em-hiredis requirement).

Doing things differently

Some people have asked me why I didn't use technology A or technology B. I think I addressed that mostly above but I'll tackle a couple of key ones.

ZeroMQ

The main reason for not using 0mq was that I wasn't really aware of it. Were I to start over and still be using Ruby, I'd probably give it a good strong look. The would still be the question of the storage component though. There's still a possible place for it that I'll address in part four.

NATS

This was something I simply had no idea about until I started poking around the CloudFoundry code base. I can almost guarantee that NATS will be a part of Noah in the future. Expect much more information about that in part four.

MongoDB

You have got to be kidding me, right? I don't trust my data (or anyone else's for that matter) to a product that doesn't understand what durability means when we're talking about databases.

Insert favorite data store here

As I said, Redis was the best way to get multiple required functionality into a single product. Why does a data storage engine have a pubsub messaging subsystem built in? I don't know off the top of my head but I'll take it.

Wrap up - Part 3

So again, because I evidently like recaps, here's the take away:

• The key components in Noah are Redis and Sinatra
• Noah is written in Ruby because of time constraints in learning a new language
• Noah strives for the server component to have the broadest set of distribution vectors as possible
• Ruby dependencies are kept to a minimum to ensure the previous point
• The lightest possible abstractions (Ohm) are used.
• Stricter requirements exist for non-server components because of flexibility in alternates
• I really should learn me some erlang
• I'm not a fan of MongoDB

If you haven't guessed, I'm doing one part a night in this series. Tomorrow is part four which will cover the future plans for Noah. I'm also planning on a bonus part five to cover things that didn't really fit into the first four.

On Noah - Part 2

This is the second part in a series on Noah. Part 1 is available here

In part one of this series, I went over a little background about ZooKeeper and how the basic Zookeeper concepts are implemented in Noah. In this post, I want to go over a little bit about a few things that Noah does differently.

Noah Primitives

As mentioned in the previous post, Noah has 5 essential data types, four of which are what I've interchangeably refered to as either Primitives and Opinionated models. The four primitives are Host, Service, Application and Configuration. The idea was to map some common use cases for Zookeeper and Noah onto a set of objects that users would find familiar.

You might detect a bit of Nagios inspiration in the first two.

Host

Analogous to a traditional host or server. The machine or instance running the operating system. Unique by name.

Service

Typically mapped to something like HTTP or HTTPS. Think of this as the listening port on a Host. Services must be bound to Hosts. Unique by service name and host name.

Application

Apache, your application (rails, php, java, whatever). There's a subtle difference here from Service. Unique by name.

Configuration

A distinct configuration element. Has a one-to-many relationship with Applications. Supports limited mime typing.

Hosts and Services have a unique attribute known as status. This is a required attribute and is one of up,down or pending. These primitives would work very well integrated into the OS init process. Since Noah is curl-friendly, you could add something globally to init scripts that updated Noah when your host is starting up or when some critical init script starts. If you were to imagine Noah primitives as part of the OSI model, these are analagous to Layers 2 and 3.

Applications and Configurations are intended to feel more like Layer 7 (again, using our OSI model analogy). The differentiation is that your application might be a Sinatra or Java application that has a set of Configurations associated with it. Interestingly enough, you might choose to have something like Tomcat act as both a Service AND an Application. The aspect of Tomcat as a Service is different than the Java applications running in the container or even Tomcat's own configurations (such as logging).

One thing I'm trying to pull off with Configurations is limited mime-type support. When creating a Configuration in Noah, you can assign a format attribute. Currently 3 formats or types are understood:

• string
• json
• yaml

The idea is that, if you provide a type, we will serve that content back to you in that format when you request it (assuming you request it that way via HTTP headers). This should allow you to skip parsing the JSON representation of the whole object and instead use it directly. Right now this list is hardcoded. I have a task to convert this.

Hosts and Services make a great "canned" structure for building a monitoring system on top of Noah. Applications and Configurations are a lightweight configuration management system. Obviously there are more uses than that but it's a good way to look at it.

Ephemerals

Ephemerals, as mentioned previously, are closer to what Zookeeper provides. The way I like to describe Ephemerals to people is a '512 byte key/value store with triggers' (via Watch callbacks). If none of the Primitives fit your use case, the Ephemerals make a good place to start. Simply send some data in the body of your post to the url and the data is stored there. No attempt is made to understand or interpret the data. The hierarchy of objects in the Ephemeral namespace is completely arbitrary. Data living at /ephemerals/foo has no relationship with data living at /ephemerals/foo/bar.

Ephemerals are also not browseable except via a Linking and Tagging.

Links and Tags

Links and Tags are, as far as I can tell, unique to Noah compared to Zookeeper. Because we namespace against Primitives and Ephemerals, there existed the need to visualize objects under a custom hierarchy. Currently Links and Tags are the only way to visualize Ephemerals in a JSON format.

Tags are pretty standard across the internet by now. You might choose to tag a bunch of items as production or perhaps group a set of Hosts and Services as out-of-service. Tagging an item is a simple process in the API. Simply PUT the name of the tag(s) to the url of a distinct named item appended by tag. For instance, the following JSON posted to /applications/my_kick_ass_app/tag with tag the Application my_kick_ass_app with the tags sinatra, production and foobar:

{"tags":["sinatra", "production", "foobar"]}

Links work similar to Tags (including the act of linking) except that the top level namespace is now replaced with the name of the Link. The top level namespace in Noah for the purposes of Watches is //noah. By linking a group of objects together, you will be able to (not yet implemented), perform operations such as Watches in bulk. For instance, if you wanted to be informed of all changes to your objects in Noah, you would create a Watch against //noah/*. This works fine for most people but imagine you wanted a more multi-tenant friendly system. By using links, you can group ONLY the objects you care about and create the watch against that link. So //noah/* becomes //my_organization/* and only those changes to items in that namespace will fire for that Watch.

The idea is also that other operations outside of setting Watches can be applied to the underlying object in the link as well. The name Link was inspired by the idea of symlinking.

Watches and Callbacks

In the first post, I mentioned that by nature of Noah being "disconnected", Watches were persistent as opposed to one-shot. Additionally, because of the pluggable nature of Noah Watches and because Noah has no opinion regarding the destination of a fired Watch, it becomes very easy to use Noah as a broadcast mechanism. You don't need to have watches for each interested party. Instead, you can create a callback plugin that could dump the messages on an ActiveMQ Fanout queue or AMQP broadcast exchange. You could even use multicast to notify multiple interested parties at once.

Again, the act of creating a watch and the destination for notifications is entirely disconnected from the final client that might use the information in that watch event.

Additionally, because of how changes are broadcast internally to Noah, you don't even have to use the "official" Watch method. All actions in Noah are published post-commit to a pubsub queue in Redis. Any language that supports Redis pubsub can attach directly to the queue and PSUBSCRIBE to the entire namespace or a subset. You can write your own engine for listening, filtering and notifying clients.

This is exactly how the Watcher daemon works. It attaches to the Redis pubsub queue, makes a few API calls for the current registered set of watches and then uses the watches to filter messages. When a new watch is created, that message is like any other change in Noah. The watcher daemon sees that and immediately adds it to its internal filter. This means that you can create a new watch, immediately change the watched object and the callback will be made.

Wrap up - Part Two

So to wrap up:

• Noah has 5 basic "objects" in the system. Four of those are opinionated and come with specific contracts. The other is a "dumb" key/value store of sorts.
• Noah provides Links and Tags as a way to perform logical grouping of these objects. Links replace the top-level hierarchy.
• Watches are persistent. The act of creating a watch and notifying on watched objects is disconnected from the final recipient of the message. System A can register a watch on behalf of System B.
• Watches are nothing more than a set of filters applied to a Redis pubsub queue listener. Any language that supports Redis and its pubsub queue can be a processor for watches.
• You don't even have to register any Watches in Noah if you choose to attach and filter yourself.

Part three in this series will discuss the technology stack under Noah and the reasoning behind it. A bit of that was touched on in this post. Part four is the discussion about long-term goals and roadmaps.

On Noah - Part 1

This is the first part in a series of posts going over Noah

As you may have heard (from my own mouth no less), I've got a smallish side project I've been working on called Noah.

It's a project I've been wanting to work on for a long time now and earlier this year I got off my ass and started hacking. The response has been nothing short of overwhelming. I've heard from so many people how they're excited for it and nothing could drive me harder to work on it than that feedback. To everyone who doesn't run away when I talk your ear off about it, thank you so much.

Since I never really wrote an "official" post about it, I thought this would be a good opportunity to talk about what it is, what my ideas are and where I'd like to see it go in the future.

So why Noah?

fair warning. much of the following may be duplicates of information in the Noah wiki

The inspiration for Noah came from a few places but the biggest inspiration is Apache Zookeeper. Zookeeper is one of those things that by virtue of its design is a BUNCH of different things. It's all about perspective. I'm going to (yet again) paste the description of Zookeeper straight from the project site:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Now that might be a bit confusing at first. Which is it? Is it a configuration management system? A naming system? It's all of them and, again, it's all about perspective.

Zookeeper, however, has a few problems for my standard use case.

• Limited client library support
• Requires persistent connections to the server for full benefit

By the first, I mean that the only official language bindings are C and Java. There's contributed Python support and Twitter maintains a Ruby library. However all of these bindings are "native" and must be compiled. There is also a command-line client that you can use for interacting as well - one in Java and two C flavors.

The second is more of a showstopper. Zookeeper uses the client connection to the server as in-band signaling. This is how watches (discussed in a moment) are communicated to clients. Persistent connections are simply not always an option. I can't deploy something to Heroku or AppEngine that requires that persistent connection. Even if I could, it would be cost-prohibitive and honestly wouldn't make sense.

Looking at the list of features I loved about ZK, I thought "How would I make that work in the disconnected world?". By that I mean what would it take to implement any or all of the Zookeeper functionality as a service that other applications could use?

From that thought process, I came up with Noah. The name is only a play on the concept of a zookeeper and holds no other real significance other than irritation at least two people named Noah when I talk about the project.

So working through the feature list, I came up with a few things I REALLY wanted. I wanted Znodes, Watches and I wanted to do it all over HTTP so that I could have the broadest set of client support. JSON is really the defacto standard for web "messaging" at this point so that's what I went with. Basically the goal was "If your language can make HTTP requests and parse JSON, you can write a Noah client"

Znodes and Noah primitives

Zookeeper has a shared hierarchical namespace similar to a UNIX filesystem. Points in the hierarchy are called znodes. Essentially these are arbitrary paths where you can store bits of data - up to 1MB in size. These znodes are unique absolute paths. For instance:

    /
/systems
/foo
/bar
/networks
/kansas
/router-1
/router-2

Each fully qualified path is a unique znode. Znodes can be ephemeral or persistent. Zookeeper also has some primitives that can be applied to znodes such as 'sequence`.

When I originally started working on Noah, so that I could work with a model, I created some base primitives that would help me demonstrate an example of some of the use cases:

• Host
• Service
• Application
• Configuration

These primitives were actual models in the Noah code base with a strict contract on them. As an example, Hosts must have a status and can have any number of services associated with them. Services MUST be tied explicity to a host. Applications can have Configurations (or not) and Configurations can belong to any number of Applications or not. Additionally, I had another "data type" that I was simply calling Ephemerals. This is similar to the Zookeeper znode model. Originally I intended for Ephemerals to be just that - ephemeral. But I've backed off that plan. In Noah, Ephemerals can be either persistent or truely ephemeral (not yet implemented).

So now I had a data model to work with. A place to store information and flexibility to allow people to use the predefined primitives or the ephemerals for storing arbitrary bits of information.

Living the disconnected life

As I said, the model for my implementation was "disconnected". When thinking about how to implement Watches in a disconnected model, the only thing that made sense to me was a callback system. Clients would register an interest on an object in the system and when that object changed, they would get notified by the method of their choosing.

One thing about Watches in Zookeeper that annoys me is that they're one-shot deals. If you register a watch on a znode, once that watch is triggered, you have to REREGISTER the watch. First off this creates, as documented by the ZK project, a window of opportunity where you could miss another change to that watch. Let's assume you aren't using a language where interacting with Zookeeper is a synchronous process:

• Connect to ZK
• Register watch on znode
• Wait
• Change happens
• Watch fires
• Process watch event
• Reregister watch on znode

In between those last two steps, you risk missing activity on the znode. In the Noah world, watches are persistent. This makes sense for two reasons. The first is that the latency between a watch callback being fired and proccessed could be much higher than the persistent connection in ZK. The window of missed messages is simply much greater. We could easily be talking 100's of milliseconds of latency just to get the message and more so to reregister the watch.

Secondly, the registration of Watches in Noah is, by nature of Noah's design and as a byproduct, disconnected from the consumer of those watches. This offers much greater flexibility in what watches can do. Let's look at a few examples.

First off, it's important to understand how Noah handles callbacks. The message format of a callback in Noah is simply a JSON representation of the changed state of an object and some metadata about the action taken (i.e. delete, create, update). Watches can be registered on distinct objects, a given path (and thus all the children under that path) and further refined down to a given action. Out of the box, Noah ships with one callback handler - http. This means that when you register a watch on a path or object, you provide an http endpoint where Noah can post the aforementioned JSON message. What you do with it from there is up to you.

By virtue of the above, the callback system is also designed to be 'pluggable' for lack of a better word. While the out of the box experience is an http post, you could easily write a callback handler that posted the message to an AMQP exchange or wrote the information to disk as a flat file. The only requirement is that you represent the callback location as a single string. The string will be parsed as a url and broken down into tokens that determine which plugin to call.

So this system allows for you to distribute watches to multiple systems with a single callback. Interestingly enough, this same watch callback system forms the basis of how Noah servers will share changes with each other in the future.

Wrap up - Part 1

So wrapping up what I've discussed, here are the key take aways:

• Noah is a 'port' of specific Zookeeper functionality to a disconnected and asynchronous world
• Noah uses HTTP and JSON as the interface to the server
• Noah has both traditional ZK-style Ephemerals as well as opinionated Primitives
• Noah uses a pluggable callback system to approximate the Watch functionality in Zookeeper
• Clients can be written in any language that can speak HTTP and understand JSON (yes, even a shell script)

Part 2 and beyond

In part two of this series we'll discuss some of the additions to Noah that aren't a part of Zookeeper such as Tags and Links. Part 3 will cover the underlying technology which I am intentionally not discussing at this point. Part 4 will be a roadmap of my future plans for Noah.

Who owns my availability?

Hey did you know EC2 had problems today? Yeah nothing major just a total effing collapse of the EBS system at US-EAST-1.

You know what that means....

"Hey guys, can anyone tell me who owns my availability?"

"Internet learns lesson of putting "all eggs in the EC2 basket". Buy your own machines, brothers."

I could go on....but I won't. I'm also going to stop short of posting a CeeLo video at this point.

Your stupid little comments mean nothing. I especially find it hilarious that someone from Twitter would make a comment availability. I also find the short-lived memory of some people hilarious (paraphrasing here):

"Thank god we're hosted on Joyent/Linode/My mom's basement"

Please. Your attempt to curry favor and free service with your provider are transparent and frankly, makes you look stupid.

Yo Netflix/SimpleGeo/JRandomDude I'm happy for you and and all. I'ma let you finish but....

So who DOES own my availability?
Here's a hint; it's not always that simple.

Yes, the ultimate responsibility for those impacted lies with those who were impacted but let's look at a few facts (or excuses - if you're being a dick about it):

Not everyone has the resources of a Netflix
Comparing anyone else's EC2 usage to Netflix is simply retarded. It's a lot like working with an ex-Google employee (I've worked with a few). They have some awesome ideas and learned some great stuff there but guess what? About 85% of it is USELESS to anyone except someone the size of Google. What works at Google doesn't work at my company.

It's not even a matter of scaling down the concept. It's simply NOT possible. Yeah let me just go buy a shipping container and build a datacenter in a box. Hardware failure? Replace the box with one off the shelf. Oh wait, not everyone has a warehouse of replacement servers. People have trouble getting a few spare hard drives to swap out.

Telling someone that they should just do what Netflix does makes you look stupid. Not them.

WE used Joyent/Linode/GoGrid/My mom's basement
Really? Really? I'm not being an AWS fanboy here but here is a simple fact: No other 'cloud' provider comes even REMOTELY close to the feature set of AWS. No one. Not only does no one come close but Amazon is CONSTANTLY iterating on new stuff to widen the gap even more.

It's not like your provider hasn't had a major outage in recent memory. And comparing an effing VPS provider to Amazon? You seriously just don't get it.

You should have designed around this possibility
Well no shit, sherlock. Guess what, it was rejected. Why? Who knows? Who cares? It's irrelevant. Sometimes the decision isn't ours to make. In the REAL world, people have to balance risk vs. reward.

Here's a tidbit of information. At EVERY single company I've been at where I was involved with architecting a solution from the ground up, we never had redundancy built in from the get go. Did I find it appalling. Absolutely but the choice wasn't mine. I did the best I could to prevent anything that would make adding it TOO difficult later on but we didn't have our DR site online from day one. We sometimes had to accrue a little technical debt. The best we could do was to minimize it as much as possible.

Designing around failure is not the same as designing for the worse case scenario. Sometimes you just have to accept that "if component X has Y number of failures, we're going to have an outage". If you have the ability to deal with it now (resources/money/whatever), then that's awesome. Sometimes you just have to accept that risk.

Oh sure I'd love to use (insert buzzword/concurrent/distributed language of the day) here. But I can't. It would be totally awesome if everything were designed from the ground up to handle that level of failure but it's not.

And another thing
The thing that bothers me most is the two-faced attitude around it all.

On one hand people are telling you it's stupid to host your own hardware. On the other hand they'll laugh at you when your provider has an outage and tell you that you should have built your own.

On one hand they'll tell you it's stupid to use some non-traditional new-fangled language and on the other hand laugh at you when you could have avoided all these problems if you had just used non-traditional new-fangled language.

On one hand they'll tell you that you should use insert-traditional-RDBMS here and on the other hand say that it's your fault for not rearchitecting your entire codebase around some NoSQL data store.

Not everyone has the same options. I hate the phrase "hindsight is 20/20". Why? Because it's all relevant. Sometimes you don't know that something is the wrong choice till it bites you in the ass. Hindsight in technology is only valuable for about a year. Maybe 6 months. Technology moves fast. It's easy to say that someone should have used X when you don't realize that they started working on things six months before X came along. If you have that kind of foresight, I'd love to hire you to play the stock market for me.

Not everyone has the luxury of switching midstream. You have to make the most what technology is available. If you keep chasing the latest and greatest, you'll never actually accomplish anything.

Are these excuses? Absolutely but there's nothing inherently wrong with excuses. You live and learn. So to those affected by the outage (still on-going mind you), take some comfort. Learn from your mistakes. The worst thing you could do at this point would be to NOT change anything. At a minimum, if you aren't the decision maker, you should document your recommendations and move on. If you are the decision maker, you need to..you know...decide if the risk of this happening again is acceptable.

Sinatra, Noah and CloudFoundry - the dirty details

So via some magical digital god, my signup for Cloud Foundry got processed. Obviously my first thought was to try and get Noah up and running. Cloud Foundry is a perfect fit for Noah because I have access to Redis natively. I have a working setup now but it took a little bit of effort.

Getting set up
As with everything these days, my first action was to create a gemset. I'll not bore you with that process but for the sake of this walkthrough, let's use a 1.9.2 gemset called 'cfdev'.

The VMC getting started guide has most of the information you'll need but I'm going to duplicate some of it here for completeness:

 gem install vmc
 vmc target api.cloudfoundry.com
 vmc login

And we're ready to rock. The VMC command line help is very good with the exception that the optional args aren't immediately visible.

vmc help options

will give you a boatload of optional flags you can pass in. One that was frequently used during the demos at launch was '-n'. I would suggest you NOT use that for now. The prompts are actually pretty valuable.

So in the case of Noah, we know we're going to need a Redis instance. Because everything is allocated dynamically CloudFoundry makes heavy use of environment variables to provide you with important settings you'll need.

First Attempt
If you watched the demo (or read the quickstart Sinatra example), there's a demo app called 'env' that they walk you through. You're going to want to use that when troubleshooting things. My first task was to duplicate the env demo so I could take a gander at the variables I would need for Redis. For the record, the steps I'm documenting here might appear out of order and result in some wasted time. I'm one of those guys who reads the instructions 2 days after I've broken something so you have an idea of what I did here:

 vmc help
 vmc services
 vmc create-service redis redis-noah
 vmc services

At this point, I now have a named instance of redis. The reason I felt safe enough doing this now is that I noticed in the help two service commands - 'bind-service' and 'unbind-service'. I figured it was easy enough to add the service to my app based on those options.

So go ahead and create the env app per the getting started documentation. If you followed my suggestion and DIDN'T disable prompts, you'll get the option to bind you app to a service when you push the first time. If you're running without prompts (using the '-n' option), you'll probably want to do something like this:

vmc push myenvapp --url ohai-env.cloudfoundry.com
vmc bind-service my_redis_service myenvapp

If you visit the url you provided (assuming it wasn't taken already?) at /env, you'll get a big dump of all the environment variables. The ones that you'll need be using most are probably going to be under `VCAP_SERVICES`. What you'll probably also notice is that `VCAP_SERVICES` is a giant JSON blob. Now you may also notice that there's a nice `VMC_REDIS` env variable there. It's pretty useless primarily because there's also a GIANT warning in the env output that all `VMC_` environment variables are deprecated but also because your redis instance requires a password to access which means you need to traverse the JSON blob ANYWAY.

So if we paste the blog into an IRB session we can get a better representation. I wish I had done that first. Instead, I reformatted it with jsonlint dutifully wrote the following madness:

which I spent a good 30 minutes troubleshooting before I realized that it's actually an array. It should have been this:


So now that I had all the variables in place, I went about converting my heroku Noah demo . That demo uses a Gemfile and a rackup file so I figured it would work just fine here. No such luck. This is where things get hairy.

Sinatra limitations
The short of it is that Sinatra application support on CF right now is a little bit of a CF. It's very basic and somewhat brute force. If you're running a single file sinatra application, it will probably work. However if you're running anything remotely complex, it's not going to work without considerable effort. Noah is even more of a special case because it's distributed as a gem. This actually has some benefit as I'll mention farther down. However it's not really "compatible" with the current setup on Cloud Foundry. Here's the deal:

If you look here, You'll see that the way your sinatra application is start is by calling ruby (with or without bundler depending) against what it detects as your main app file. This is done here which leads us all the way to this file:

`https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/manifests/sinatra.yml`

Essentially for sinatra applications, the first .rb file it comes across with 'require sinatra', is considered the main app file. Bummer. So config.ru is out. The next step is to rename it to a '.rb' file and try again. This is where I spent most of my troubleshooting. There's a gist of the things I tried (including local testing) here:

`https://gist.github.com/920552`

Don't jump to the solution just yet because it's actually incomplete. This troubleshooting led to another command you'll want to remember:

vmc files myapp logs/stderr.log

I found myself typing it a lot during this process. For whatever reason, possibly due to bundler or some other vcap magic I've not discovered yet what works at home does not work on Cloud Foundry exactly the same. That's fine, it's just a matter of knowing about it. It also didn't help that I wasn't getting any output at all for the entire time I was trying to figure out why config.ru didn't work.

Thanks to Konstantin Haase for his awesome suggestion in #sinatra. The trick here was to mimic what rackup does. Because the currently released Noah gem has a hard requirement on rack 1.2.1, his original suggestion wasn't an exact fit but I was able to get something working:

https://gist.github.com/921292

So what did we do?
Ensure that the wrapper file is picked up first by making sure it's the ONLY rb file uploaded with `require sinatra` at the top.
Because of a bug in rack 1.2.1 with Rack::Server.new, I HAD to create a file called config.ru. The fix in rack 1.2.2 actually honors passing all the options into the constructor without needing the config.ru file.
Explicitly connect to redis before we start the application up.

The last one was the almost as big of a pain in the ass as getting the application to start up.

I think (and I'm not 100% sure) that you are prohibited from setting environment variables inside your code. Because of the convoluted way I had to get the application started, I couldn't use my sinatra configuration block properly (`set :redis_url, blahblahblah`). I'm sure it's possible but I'm not an expert at rack and sinatra. I suppose I could have used Noah::App.set but at this point I was starting to get frustrated. Explicitly setting it via Ohm.connect worked.

I'm almost confident of this environment variable restriction because you can see options in 'vmc help' that allow you to pass environment variables into your application. That would work fine for most cases except that I don't know what the redis values are outside of the app and they're set dynamically anyway.

So where can things improve?
First off, this thing is in beta. I'm only adding this section because it'll serve as a punch list of bugs for me to fix in vcap ;)

• Sinatra support needs to be more robust.

You can see that the developers acknowledged that in the staging plugin code. There are TODOs listed. It's obvious that a sinatra application of any moderate complexity wasn't really tested and that's fine. The building blocks are there and the code is opensource. I'll fix it myself (hopefully) and submit a pull request.

• Allow override of the main app file from VMC.

It appears from the various comments that the node.js support suffers some of the same brute force detection routines. An option to pass in what the main applictation file is would solve some of that.

• Document the environment variable restrictions.

I didn't see any documentation anywhere about that restriction (should it exist). I could be doing something wrong too. It's worth clarifying.

• Better error reporting for failed startups

I'm not going to lie but I spent a LONG time troubleshooting the fact that the app simply wasn't starting up. The default output when a failure happens during deploy is the staging.log file. All this EVER contained was the output from bundler. It should include the output of stderr.log and stdout.log as well. Also an explicit message should be returned if the main app file can't be detected. That would have solved much of my frustration up front.

That's just the stuff I ran into to get things going. The first item is the biggest one. If you're writing a monolithic single-file sinatra app, the service will work GREAT. If you aren't, you'll have to jump through hoops and wrapper scripts for now. Supporting rackup files for Sinatra and Rack apps will go a long way to making things even more awesome.

One pleasant surprise I found was that, despite what I was told, I didn't need to include every gem in my Gemfile. Because Noah itself has its deps, Bundler pulls those in for me.

I've created a git repo with the code as well as a quickstart guide for getting your own instance running. You can find it here:

https://github.com/lusis/noah-cloudfoundry-demo

Operational Primitives

"Infrastructure as code". I love the phrase. Where devops is a word that is sadly open to so much (mis)interpretation, "Infrastructure as code" is pretty clear. Treat your infrastructure as code. Programmable. Testable. Deployable.

But when you start to really think about that concept, there's a deep dive you can take, navigating various programming and computer science constructs and applying those to your infrastructure.

I've been working pretty heavily on getting the first API stable release of Noah out the door. It's been a challenge with the schedule I have to work on it - which is essentially "when everyone else in the house is asleep and I'm awake'. Last night, I came to a fork in the road where I needed to make a decision. This decision would lock me into an API path that I was unwilling to change for a while. Nobody wants to use a service or tool with a constantly changing API. I needed to shit or get off the pot, to use a creative euphemism. With the announcements of both Doozer and riak_zab, it was clear that I wasn't the only person attempting to tackle the ZooKeeper space.

Since Github lacks any facility for soliciting project feedback (hint hint, @github), I decided to create a  Wufoo form and tweet it out. I don't have a very big audience but I was hoping it would at least get to the people who were likely to use Noah. The form was fairly simple with one question on something that I had pretty summarily dismissed early on - HATEOAS (hypermedia as the engine of application state).

A small HATEOAS diversion


The HATEOAS debate is a lot like Linux vs. GNU/Linux. It's fairly esoteric but there's some meat to the matter. My problem with it was simply that, despite what Roy Fielding and others intended, REST had taken on a new definition and it wasn't the strict HATEOAS one. Additionally, I found it VERY difficult to map HATEOAS concepts to JSON. JSON is a great format but a rich document structure is not (rightly so) part of the format. It's intended to be simple, easily read and cleanly mapped to machine readable format. It also felt like extra work on the part of the API consumer. The concepts that we use when reading a website (click this link, read this list, click this link) are simple not necessary when you have a contextually relevant (or descriptive) URL scheme. True, as a human I don't make changes in the URL bar to navigate a site (I use the links provided by the site) but when it comes to dealing with an API, I don't exhibit the same usage patterns as a web browser. I'm making distinct atomic transactions (DELETE this resource, PUT this resource) at a given endpoint. These simply aren't the same as filling out forms and are only tangentially related. I'm simply not willing to force someone to parse a JSON object to tell them how to create a new object in the system. The API for Noah is fairly simple as it is. Objects in the system have only two or three required attributes for a given operation and normally one of those attributes is directly inferable from the URL.

But based on the poll results thus far, I wanted to give the idea fair consideration which led me to think about what types of objects Noah had in its system.

Primitives


For those who aren't familiar or simple don't know, there's a term in computer science and programming called "Primitive". It essentially means a basic data type in a language from which other complex data types are created. A building block if you will. Some easily grokable examples of primitives are Characters and Integers. Some languages actually have ONE primitive like Object and everything is built on top of that. You could get into a semantic argument about a lot of this so I'm going to leave it at that.

But back to the phrase "Infrastucture as code". If we start looking at how we "program" our infrastructure, what are the "primitives" that our language supports. I inadvertently created some of these in Noah. I've been calling them the "opinionated models" but really in the infrastructure programming language of Noah, they're primitives.

When this hit me last night, I immediately pulled out the tablet and went to work on a mind map. I laid out what I had already implemented as primitives in Noah:

• Host
• Service
• Application
• Configuration

I then started to think about other concepts in Noah. Were Ephmerals really a primitive. Not really. If anything Ephemerals are more similar to ruby's BasicObject. The only real attribute Ephemerals have are a path (similar to the object_id).

So what else would be our modern operational primitives? Remember that we're talking about building blocks here. I don't want to abstract out too much. For instance you could simply say that a "Resource" is the only real operational primitive and that everything else is built on top of that.  Also consider that languages such as Python have some richer primitives built-in like tuples.

One interesting thought I had was the idea that "State" was a primitive. Again, in the world of operations and infrastructure, one of your basic building blocks is if something is available or not - up or down. At first glance it would appear that this maps pretty cleanly to a Boolean (which is a primitive in most languages) however I think it's a richer primitive than that.

In the world of operations, State is actually quaternary (if that's the right word) rather than binary. There are two distinct areas between up and down that have dramatically different implications on how you interact with it:

• Up
• Down
• Pending Up
• Pending Down

Currently in Noah, we simple have Up, Down and Pending but something that is in the State of shutting down is grossly different than something in the state of Starting up. Look at a database that is queiscing connections. It's in a state of "Pending Down". It's still servicing existing requests. However a database in the state of "Pending Up" is NOT servicing any requests.

So I'm curious what other thoughts people have. What else are the basic building blocks of modern operations when viewed through the lens of "infrastructure as code"?

For the record, I'm still pretty confident that Noah still has a place in the Doozer, riak_zab, ZooKeeper world. All three of those still rely on the persistent connection for signaling and broadcast whereas Noah is fundamentally about the disconnected and asynchronous world.

It does not follow and Wheaton's Law

"I'm not a smart guy".

I say this quite a bit. I don't say it to fish for compliments or as a chance to have my ego boosted. I say it because I realize that, out of the vast corpus of computer science knowledge that exists, the part that I DO know is a blade of grass on a football field.

"I'm not a developer"

I say this a lot too. This is not meant as a slight to developers. It's meant as a compliment. There are REAL developers out there and I'm just pretending (after a fashion). I have never worked a professional gig as a developer. I've had honest discussions with people who want to pay me lots of money to be a developer. The best way I can explain it to them is that it would be unfair to you, as an employer, to hire me for a developer position because you would be unhappy with the results. In general it takes me twice as long to solve a development problem as it takes a real developer.

There are lots of factors to this; education, regular skill use and a general affinity for picking up concepts. I never graduated college and I pretty much suck at math. That's not to say I couldn't learn it but there are some things I know I'll never be as good at as someone else and that's fine by me. I'm not settling for mediocrity I just know my limitations. I'll still take a stab at it.

There are, however, some REALLY smart people out there. I used to follow a bunch of them on Twitter because they would link to or drop ideas that really made me want to go research something. I noticed an interesting trend though about some of them. They had a tendency to be dicks. Not just the occasional "Only an idiot would do X" but outright vitriol. Was it trolling? In some cases, sure, but I honestly got the impression that they actually looked down on people who didn't who use a certain technology or chose any path different than they would have chosen.

At the other extreme, you have the folks who make snide remarks or drop a non sequitur about a given technology presumably in an attempt to make the in-crowd giggle and the rest of us poor saps wonder what the hell we're doing wrong. I mean these are smart people, right? If they know something I don't about a given technology, then by god, I'd love to know what it is. I'd love to learn why they feel that way. In the end, though, all you hear is giggling in the background and wonder what the big joke was.

When the hell did we, the people who were typically on the outside of the in-crowd, turn into the people who gave us the most shit growing up? It's like a fucking geek Stockholm Syndrome thing that's gone off the deep end but instead of just sympathizing with our abuser, we're the abuser and we relish it.

I'm guilty of this behavior. I'm the first in line to criticize MongoDB, for instance. The difference? I'll actually sit down with you and tell you WHY I don't like MongoDB and why I feel it's a bad choice in many situations.

What I'm asking is that, as one of the people on the outside, educate me. As much as I think Ted Dziuba is a big troll, at least he takes the time to write it down and trys to defend his position. Ben Bleything had an awesome tweet today:

I guess what I meant is, I don't have the experience to form that opinion, I'd like to learn from you.

That's my attitude exactly. "Put up or shut up" is a bit harsh but in the broadest terms, that's what needs to happen. If you think X is superior to Z then say why. There are some of us who could benefit from it.

Sidebar on Semantics


Additionally,  let's make sure we're also on the same page in terms of semantics. If we're talking about queues, clarify if you're talking about data structures versus a message queue because there's a big f'ing difference in my mind.

When I hear queue, I don't think data structure. I think of a message queue in the product sense. That's just my background. I think about things like guaranteed delivery and message durability.

Ad-Hoc Configuration, Coordination and the value of change

For those who don't know, I'm currently in Boston for DevOps Days. It's been amazing so far and I've met some wonderful people. One thing that was REALLY awesome was the open space program that Patrick set up. You won't believe it works until you've tried it. It's really powerful.

In one of our open spaces, the topic of ZooKeeper came up. At this point I made a few comments, and at the additional prodding of everyone went into a discussion about ZooKeeper and Noah. I have a tendency to monopolize discussions around topics I'm REALLY passionate about so many thanks for everyone who insisted I go on ;)

Slaughter the deviants!
The most interesting part of the discussion about ZooKeeper (or at least the part I found most revealing) was that people tended to have trouble really seeing the value in it. One of the things I've really wanted to do with Noah is provide (via the wiki) some really good use cases about where it makes sense.

I was really excited to get a chance to talk with  Alex Honor (one of the co-founders of DTO along with Damon Edwards) about his ideas after his really interesting blog post around ad-hoc configuration. If you haven't read it, I suggest you do so.

Something that often gets brought up and, oddly, overlooked at the same time is the where ad-hoc change fits into a properly managed environment (using a tool like puppet or chef).

At this point, many of you have gone crazy over the thought of polluting your beautifully organized environment with something so dirty as ad-hoc changes. I mean, here we've spent all this effort on describing our infrastructure as code and you want to come in and make a random, "undocumented" change? Perish the thought!

However, as with any process or philosophy, strict adherence with out understanding WHEN to deviate will only lead to frustration. Yes, there is a time to deviate and knowing when is the next level of maturity in configuration management.

So when do I deviate
Sadly, knowing when it's okay to deviate is as much a learning experience as it was getting everything properly configured in the first place. To make it even worse, that knowledge is most often specific to the environment in which you operate. The whole point of the phrase ad-hoc is that it's..well...ad-hoc. It's 1 part improvisation/.5 parts stumbling in the dark and the rest is backfilled with a corpus of experience. I don't say this to sound elitist.

So, really, when do I deviate. When/where/why and how do I deviate from this beautifully described environment? Let's go over some use cases and point out that you're probably ALREADY doing it to some degree.

Production troubleshooting
The most obvious example of acceptable deviation is troubleshooting. We pushed code, our metrics are all screwed up and we need to know what the hell just happened. Let's crank up our logging.

At this point, changing your log level, you've deviated from what your system of record (your CM tool) says you should be. Our manifests, our cookbooks, our templates all have us using a loglevel of ERROR but we just bumped up one server to DEBUG. so we could troubleshoot. That system is now a snowflake. Unless you change that log level back to ERROR, you know have one system that will, until you do a puppetrun of chef-client run is different than all the other servers of the class/role.

Would you codify that in the manifest? No. This is an exception. A (should be) short-lived exception to the rules you've defined.

Dynamic environments
Another area where you might deviate is in highly elastic environments. Let's say you've reached the holy grail of elasticity. You're growing and shrinking capacity based on some external trigger. You can't codify this. I might run 20 instances of my app server now but drop back down to 5 instances when the "event" has passed. In a highly elastic environment, are you running your convergence tool after every spin up? Not likely. In an "event" you don't want to have to take down your load balancer (and thus affect service to the existing intstances) just to add capacity. A bit of a contrived example but you get the idea.

So what's the answer?
I am by far not the smartest cookie in the tool shed but I'm opinionated so that has to count for something. These "exception" events are where I see additional tools like Zookeeper (or my pet project Noah) stepping in to handle things.

Distributed coordination, dynamically reconfigurable code, elasticity and environment-aware applications.
These are all terms I've used to describe this concept to people. Damon Edwards provided me with the last one and I really like it.

Enough jibber-jabber, hook a brother up!
So before I give you the ability to shoot yourself in the foot, you should be aware of a few things:

• It's not a system of record

Your DDCS (dynamic distributed coordination service as I'll call it because I can't ever use enough buzzwords) is NOT your system of record. It can be but it shouldn't be. Existing tools provide that service very well and they do it in an idempotent manner.

• Know your configuration

This is VERY important. As I said before, much of this is environment specific. The category of information you're changing in this way is more "transient" or "point-in-time". Any given atom of configuration information has a specific value associated with it. Different levels of volatility. Your JDBC connection string is probably NOT going to change that often. However, the number of application servers might be at different amounts of capacity based on some dynamic external factor.

• Your environment is dynamic and so should be your response

This is where I probably get some pushback. Just as one of the goals of "devops" was to deal with, what Jesse Robbins described to day as misalignment of incentive, there's an internal struggle where some values are simply fluctuating in near real time. This is what we're trying to address.

• It is not plug and play

One thing that Chef and Puppet do very well is that you can, with next to no change to your systems, predefine how something should look or behave and have those tools "make it so".

With these realtime/dynamic configuration atoms your application needs to be aware of them and react to them intelligently.

Okay seriously. Get to the point
So let's take walk through a scenario where we might implement this ad-hoc philosophy in a way that gives us the power we're seeking.

The base configuration

•  application server (fooapp) uses memcached, two internal services called "lookup" and "evaluate" and a data store of somekind.
• "lookup" and "evaluate" are internally developed applications that provide private REST endpoints for providing a dictionary service (lookup) and a business rule parser of some kind (evaluate).
• Every component's base configuration (including the data source that "lookup" and "evaluation" use) is managed, configured and controlled by puppet/chef.

In a standard world, we store the ip/port mappings for "lookup" and "evaluate" in our CM tool and tags those. When we do a puppet/chef client run, the values for those servers are populated based on the ip/port information our EXISTING "lookup"/"evaluate" servers.

This works. It's being done right now.

So where's the misalignment?
What do you do when you want to spin up another "lookup"/"evaluate" server? Well you would probably use a bootstrap of some kind and apply, via the CM tool, the changes to those values. However this now means that for this to take effect across your "fooapp" servers you need to do a manual run of your CM client. Based on the feedback I've seen across various lists, this is where the point of contention exists.

What about any untested CM changes (a new recipe for instance). I don't want to apply that but if I run my CM tool, I've now not only pulled those unintentional changes but also forced a bounce of all of my fooapp servers. So as a side product of scaling capacity to meet demand, I've now reduced my capacity at another point just to make my application aware of the new settings.

Enter Noah
This is where the making your application aware of its environment and allowing it to dynamically reconfigure itself pays off.

Looking at our base example now, let's do a bit of architectural work around this new model.

• My application no longer hardcodes a base list of servers prodviding "lookup" and "evaluate" services.
• My application understands the value of a given configuration atom
• Instead of the hardcoded list, we convert those configuration atoms akin to something like a singleton pattern that points to a bootstrap endpoint.
• FooApp provides some sot of "endpoint" where it can be notified of changes to the number/ip addresses or urls available a a given of our services. This can also be proxied via another endpoint.
• The "bootstrap" location is managed by our CM tool based on some more concrete configuration - the location of the bootstrap server.

Inside our application, we're now:

• Pulling a list of "lookup"/"evaluate" servers from the bootstrap url (i.e. http://noahserver/s/evaluate)
• Registering a "watch" on the above "path" and providing an in-application endpoint to be notified when they change.
• validating at startup if the results of the bootstrap call provide valid information (i.e. doing a quick connection test to each of the servers provided by the bootstrap lookup or a subset thereof)

If we dynamically add a new transient "lookup" server, Noah fires a notification to the provided endpoint with the details of the change. The application will receive a message saying "I have a new 'lookup' server available". It will run through some sanity checks to make sure that the new "lookup" server really does exist and works. It then appends the new server to the list of existing (permanent servers) and start taking advantage of the increase in capacity.

That's it. How you implement the "refresh" and "validation" mechanisms is entirely language specific. This also doesn't, despite my statements previously, have to apply to transient resources. The new "lookup" server could be a permanent addition to my infra. Of course this would have been captured as part of the bootstrapping process if that were the case.

Nutshell
And that's it in a nutshell. All of this is availalbe in Noah and Zookeeer right now. Noah is currently restricted to http POST endpoints but that will be expanded. Zookeeper treats watches as ephemeral. Once the event has fired, you must register that same watch. With Noah, watches are permanent.

Takeaway
I hope the above has made sense. This was just a basic introduction to some of the concepts and design goals. There are plenty of OTHER use cases for ZooKeeper alone. So the key take aways are:

• Know the value of your configuration data
• Know when and where to use that data
• Don't supplant your existing CM tool but instead enhance it.

Links
Noah
ZooKeeper
Hadoop Book (which has some AMAZING detail around ZooKeeper, the technology and use cases