Lusis

Relocating again.

2011-09-13T22:19:00.001-04:00

FYI, I'm relocating my blog back to my own domain. All the new posts will go there and eventually I'll get everything pulled in from here.

New Old Place is here: http://blog.lusis.org

Monitoring Sucks - Watch your language

2011-07-21T22:19:00.001-04:00

The following post is a recap of what was discussed in the 07/21/11 #monitoringsucks irc meeting

Before I get any further, I just want to thank everyone who attended, either in virtual person or in spirit. There have been so many awesome discussions going around this topic since we started. I am truly priviledged to interact with each and every one of you. I struggle to count myself a peer and I can only hope that I provide something in return.

I mentioned to someone today that I’m literally sick of the current landscape. I consider the current crop of monitoring solutions to be technologically bankrupt. The situation is fairly untenable at this point.

I just installed (after having a total loss of an existing Zenoss setup) Nagios again. I’m not joking when I say that it depressed the everliving hell out of me. The monitoring landscape simply has not kept up with modern developments. At first it was mildly frustrating. Then it was annoying. Now it’s actually a detriment.

Now that we have that out of the way….

Darmok and Jalad at Tanagra

Communication is important. Like Picard and Dathon, we’ve been stranded on a planet with shitty monitoring tools and we’re unable to communicate about the invisibile threat of suck because we aren’t even speaking the same language. I say event, you hear trigger, I mean data point. So the first order of business was to try and agree on a set of terms. It was decided that we would consider these things primitives. Here they are:

Please read through this before jumping to any conclusions. I promise it will all become clear (as mud).

metric

a numeric or boolean data point

The data type of a metric was something of a sticking point. People were getting hung up on data points being various things (a log message, a “status”, a value). We needed something to describe the origin. The single “thing” that triggered it all. That thing is a metric.

So why numeric OR boolean? It was pretty clear that many people considered, and rightly so I would argue, that a state change is a metric. A good example given by Christopher Webber is that of a BGP route going away. Why is this a less valid data point than the amount of disk space in use or the latency from one host to another? Frankly, it’s not.

But here’s where it gets fuzzy. What about a log message. Surely that’s a data point and thus a metric.

Yes and No. The presence of a log message is a data point. But it’s a boolean. The log message itself?

context

metadata about a metric

Now metadata itself is a loaded term but in this scope, the “human readable” attributes are considered context. Going back to our log example. The presence of the log message is a metric. The log message itself is context. Here’s the thing. You want to know if there is an error message in a log file. The type of error, the error message text? That’s context for the metric to use in determining a course of action.

Plainly speaking, metrics are for machines. Context is for humans. This leads us to….

event

metric combined with context

This is still up in the air but the general consensus was that this was a passable definition. The biggest problem with a group of domain experts is that they are frequently unable to accept semantic approximation. Take the discussion of Erlang Spawned process:

It’s sort of like a VM on a VM
NO IT’S NOT.
headdesk

The fact is that an Erlang spawned process has shades of a virtual machine is irrelevant to the domain expert. We found similar discussions around what we would call the combination of a metric and its context. But where do events come from?

resource

the source of a metric

Again, we could get into arguments around what a resource is. One thing that was painfully obvious is that we’re all sick and tired of being tied to the Host and Service model. It’s irrelevant. These constructs are part “legacy” and part “presentation”.

Any modern monitoring thought needs to realize that metrics no longer come from physical hosts or are service states. In the modern world, we’re taking a holistic view of monitoring that includes not only bare metal but business matters. The number of sales is a metric but it’s not tied to a server. It’s tied to the business as a whole. The source of your metrics is a resource. So now that we have this information - a metric, its context and who generated it - what do we do? We take….

action

a response to a given metric

What response? It doesn’t MATTER. Remember that these are primitives. The response is determined by components of your monitoring infrastructure. Humans note the context. Graphite graphs it. Nagios alerts on it. ESPER correlates it with other metrics. Don’t confuse scope here. From this point on, whatever happens has is all decided on by a given component. It’s all about perspective and aspects.

Temba, his arms wide

I’m sure through much of that, you were thinking “alerting! graphing! correlation!”. Yes, that was pretty much what happened during the meeting as well. Everyone has pretty much agreed (I think) at this point that any new monitoring systems should be modular in nature. As Jason Dixon put it - “Voltron”. No single system that attempts to do everything will meet everyone’s needs. However, with a common dictionary and open APIs you should be able to build a system that DOES meet your needs. So what are those components? Sadly this part is not as fleshed out. We simply ran out of time. However we did come up with a few basics:

Collection

getting the metrics

It doesn’t matter if it’s push or pull. It doesn’t matter what the transport is - async or point-to-point. Somehow, you have to get a metric from a resource.

Event Processing

taking action

Extract the metric and resource from an event. Do something with it. Maybe you send the metric to another component. Maybe you “present” it or send it somewhere to be presented. Maybe you perform a change on a resource (restarting a service). Essentially the decision engine.

Presentation

While you might be thinking of graphing here, that’s a type of presentation. You know what else is presentation? An email alert. Stick with me. I know what’s going through your head. No..not that…the other thing.

Analytics

This is pretty much correlation. We didn’t get a REAL solid defintion here but everyone was in agreement that some sort of analytics is a distinct component.

The “other” stuff

As I said, we had to kind of cut “official” things short. There was various discussion around Storage and Configuration. Storage I personally can see as a distinct component but Configuration not so much. Configuration is an aspect of a component but not a component itself.

Logical groupings

Remember when I said I know what you’re thinking? This is what I think it was.

You can look at the above items and from different angles they look similar. I mean sending an email feels more like event processing than presentation. You’d probably be right. By that token, drawing a point on a graph is technically processing an event. The fact is many components have a bit of a genetic bond. Not so much parent/child or sibling but more like cousins. In all honesty, if I were building an event processing component, I’d probably handle sending the email right there. Why send it to another component? That makes perfect sense. Graphing? Yeah I’ll let graphite handle that but I can do service restarts and send emails. Maybe you have an intelligent graphing component that can do complex correlation inband. That makes sense too.

I’m quite sure that we’ll have someone who writes a kickass event processor that happens to send email. I’m cool with that. I just don’t want to be bound to ONLY being able to send emails because that’s all your decision system supports.

Shaka, when the walls fell

Speaking personally, I really feel like today’s discussion was VERY productive. I know that you might not agree with everything here. Things are always up for debate. The only thing I ask is that at some point, we’re all willing to say “I know that this definition isn’t EXACTLY how I would describe something but it’s close enough to work”.

So what are the next steps? I think we’ve got enough information and consensus here for people to start moving forward with some things. One exercise, inspired by something Matt Ray said, that we agreed would be REALLY productive is to take an existing application and map what it does to our primitives and components. Matt plans on doing that with Zenoss since that’s what he knows best.

Let me give an example:

Out of the box, Nagios supports Hosts and Services which map pretty cleanly to resources. It is does not only collection but event processing and presentation. It not only supports metrics but also context (Host down is the boolean metric. “Response timeout” is the context. Through something like pnp4nagios, it can support different presentations. It has very basic set of Analytic functionality.

Meanwhile Graphite is, in my mind, strictly presentation and deals only with metrics. It does support both numeric and boolean metrics. It also has basic resource functionality but it’s not hardwired. It doesn’t really do event handling in the strict sense. Analytics is left to the human mind. It certainly doesn’t support context.

I’d love to see more of these evaluations.

Also, I know there are tons of “words” that we didn’t cover - thresholds for instance. While there wasn’t a total consensus, there was some agreement that somethings were attributes of a component but not a primitive itself. It was also accepted that components themselves would be primitives. You correlation engine might aggregate (another word) a group of metrics and generate an event. At that point, your correlation engine is now a resource with its own metrics (25 errors) and context (“number of errors across application servers exceeded acceptable limits”) which could be then sent to an event processor.

That’s the beauty of the Voltron approach and not binding a resource to a construct like a Host.

Special note to the Aussies

I’m very sorry that we couldn’t get everyone together. I’ve scheduled another meeting where we can start from scratch just like this one, or build on what was discussed already. I’m flexible and willing to wake up at whatever time works best for you guys

Thanks again to everyone who attended. If you couldn’t be there, I hope you can make the next one.

Monitoring Sucks - Round 2 - FIGHT

2011-07-12T02:26:00.000-04:00

Monitoring sucks - Round 2 - FIGHT

Recently there’s been more and more attention to the whole ##monitoringsucks campaign. Peopel are writing blog posts. Code is being written. Discussions are being had. Smart people are doing smart things.

meetingssuck

I figure it’s time to get the band back together for another official irc session and stop flooding infra-talk with random rants about where things stand. As much as I hate to even consider it, I think having something of an agenda so that it doesn’t turn into a bitch fest.

I’m open to any ideas that people have.

Jordan Sissel brought up some things that might be nice to have:

use cases
primitives

I’d like to also love to get input from others before we settle on the details. This one is going to be a bit more “formal” than the first session. Don’t let that turn you off. Everyone has something valuable to contribute.

Deliverables

I can’t believe I’m actually suggesting this part but I’d also love to walk away with something that people can work with. It’s not like people will have homework or a task list.

This is more of a “this is what we would like to see, if you want to write some code it’s a decent place to start”.

International flavor

I’m also keen on finding a time when we can get as many folks as possible to contribute. This may not be possible with that whole rotation of the earth thing but we’ll see.

Suggestions, Comments and Rude Remarks

If you’ve got anything you’d like to see or contribute on the planning side, drop me a message on IRC, email or twitter and I’ll include it (if possible) when I draft up the final “notes” beforehand.

Why Monitoring Sucks

2011-06-05T23:30:00.002-04:00

Why Monitoring Sucks (and what we're doing about it)

About two weeks ago someone made a tweet. At this point, I don't remember who said it but the gist was that "monitoring sucks". I happened to be knee-deep in frustrating bullshit around that topic and was currently evaluating the same effing tools I'd evaluated at every other company over the past 10 years or so. So I did what seems to be S.O.P for me these days. I started something.

But does monitoring REALLY suck?

Heck no! Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: - Having my hands tied with the model of host and service bindings. - Having to set up "fake" hosts just to group arbitrary metrics together - Having to either collect metrics twice - once for alerting and another for trending - Only being able to see my metrics in 5 minute intervals - Having to chose between shitty interface but great monitoring or shitty monitoring but great interface - Dealing with a monitoring system that thinks IT is the system of truth for my environment - Perl (I kid...sort of) - Not actually having any real choices

Yes, yes I know:

You can just combine Nagios + collectd + graphite + cacti + pnp4nagios and you have everything you need!

Seriously? Kiss my ass. I'm a huge fan of the Unix pipeline philosophy but, christ, have you ever heard the phrase "antipattern"?

So what the hell are you going to do about it?

I'm going to let smart people be smart and do smart things.

Step one was getting everyone who had similar complaints together on IRC. That went pretty damn well. Step two was creating a github repo. Seriously. Step two should ALWAYS be "create a github repo". Step three? Hell if I know.

Here's what I do know. There are plenty of frustrated system administrators, developers, engineers, "devops" and everything under the sun who don't want much. All they really want is for shit to work. When shit breaks, they want to be notified. They want pretty graphs. They want to see business metrics along side operational ones. They want to have a 52-inch monitor in the office that everyone can look at and say:

See that red dot? That's bad. Here's what was going on when we got that red dot. Let's fix that shit and go get beers

About the "repo"

So the plan I have in place for the repository is this. We don't really need code. What we need is an easy way for people to contribute ideas. The plan I have in place for this is partially underway. There's now a monitoringsucks organization on Github. Pretty much anyone who is willing to contribute can get added to the team. The idea is that, as smart people think of smart shit, we can create new repository under some unifying idea and put blog posts, submodules, reviews, ideas..whatever into that repository so people have an easy place to go get information. I'd like to assign someone per repository to be the owner. We're all busy but this is something we're all highly interested in. If we spread the work out and allow easy contribution, then we can get some real content up there.

I also want to keep the repos as light and cacheable as possible. The organization is under the github "free" plan right now and I'd like to keep it that way.

Blog Posts Repo

This repo serves as a place to collect general information about blog posts people come across. Think of it as hyper-local delicious in a DVCS.

Currently, by virtue of the first commit, Michael Conigliaro is the "owner". You can follow him on twitter and github as @mconigliaro

IRC Logs Repo

This repo is a log of any "scheduled" irc sessions. Personally, I don't think we need a distinct #monitoringsucks channel but people want to keep it around. The logs in this repo are not full logs. Just those from when someone says "Hey smart people. Let's think of smart shit at this date/time" on twitter.

Currently I appear to be the owner of this repo. I would love for someone who can actually make the logs look good to take this over.

Tools Repo

This repo is really more of a "curation" repo. The plan is that each directory is the name of some tool with two things it in:

A README.md as a review of the tool
A submodule link to the tool's repo (where appropriate)

Again, I think I'm running point on this one. Please note that the submodule links APPEAR to have some sort of UI issue on github. Every submodule appears to point to Dan DeLeo's 'critical' project.

Metrics Catalog Repo

This is our latest member and it already has an official manager! Jason Dixon (@obfuscurity on github/twitter - jdixon on irc) suggested it so he get's to run it ;) The idea here is that this will serves as a set of best practices around what metrics you might want to collect and why. I'm leaving the organization up to Jason but I suggested a per-app/service/protocol directory.

Wrap Up

So that's where we are. Where it goes, I have no idea. I just want to help where ever I can. If you have any ideas, hit me up on twitter/irc/github/email and let me know. It might help to know that if you suggest something, you'll probably be made the person reponsible for it ;)

Update!

It was our good friend Sean Porter (@portertech on twitter), that we have to thank for all of this ;)

From Public Photos

Update (again)

It was kindly pointed out that I never actually included a link to the repositories. Here they are:

https://github.com/monitoringsucks

On Noah - Part 4

2011-05-20T02:01:00.001-04:00

On Noah - Part 4

This is the fourth part in a series on Noah. Part 1, Part 2 and Part 3 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. Part 3 was about the components underneath Noah that make it tick.

This post is about the "future" of Noah. Since I'm a fan of Fourcast podcast, I thought it would be nice to do an immediate, medium and long term set of goals.

Immediate Future - the road to 1.0

In the most immediate future there are a few things that need to happen. These are in no specific order.

General
- Better test coverage ESPECIALLY around the watch subsystem
- Full code comment coverage
- Chef cookbooks/Puppet manifests for doing a full install
- "fatty" installers for a standalone server
- Documentation around operational best practices
- Documentation around clustering, redundancy and hadr
- Documentation around integration best practices
- Performance testing
Noah Server
- Expiry flags and reaping for Ephemerals
- Convert mime-type in Configurations to make sense
- Untag and Unlink support
- Refactor how you specify Redis connection information
- Integrated metrics for monitoring (failed callbacks, expired ephemeral count, that kind of stuff)
Watcher callback daemon
- Make the HTTP callback plugin more flexible
- Finish binscript for the watcher daemon
Other
- Finish Boat
- Finish NoahLite LWRP for Chef (using Boat)
- A few more HTTP-based callback plugins (Rundeck, Jenkins)

Now that doesn't look like a very cool list but it's a lot of work for one person. I don't blame anyone for not getting excited about it. The goal now is to get a functional and stable application out the door that people can start using. Mind you I think it's usable now (and I'm already using it in "production").

Obviously if anyone has something else they'd like to see on the list, let me know.

Medium Rare

So beyond that 1.0 release, what's on tap? Most of the work will probably occur around the watcher subsystem and the callback daemon. However there are a few key server changes I need to implement.

Server
- Full ACL support on every object at every level
- Token-based and SSH key based credentialing
- Optional versioning on every object at every level
- Accountability/Audit trail
- Implement a long-polling interface for inband watchers
Watcher callback daemon
- Decouple the callback daemon from the Ruby API of the server. Instead the daemon itself needs to be a full REST client of the Noah server
- Break out the "official" callback daemon into a distinct package
Clients
- Sinatra Helper

Also during this period, I want to spend time building up the ecosystem as a whole. You can see a general mindmap of that here.

Going into a bit more detail...

Tokens and keys

It's plainly clear that something which has the ability to make runtime environment changes needs to be secure. The first thing to roll off the line post-1.0 will be that functionality. Full ACL support for all entries will be enabled and in can be set at any level in the namespace just the same as Watches.

Versioning and Auditing

Again for all entires and levels in the namespace, versioning and auditing will be allowed. The intention is that the number of revisions and audit entries are configurable as well - not just an enable/disable bit.

In-band watches

While I've lamented the fact that watches were in-band only in Zookeeper, there's a real world need for that model. The idea of long-polling functionality is something I'd actually like to have by 1.0 but likely won't happen. The intent is simply that when you call say /some/path/watch, you can pass an optional flag in the message stating that you want to watch that endpoint for a fixed amount of time for any changes. Optionally a way to subscribe to all changes over long-polling for a fixed amount of time is cool too.

Agent changes

These two are pretty high on my list. As I said, there's a workable solution with minimal tech debt going into the 1.0 release but long term, this needs to be a distinct package. A few other ideas I'm kicking around are allowing configurable filtering on WHICH callback types an agent will handle. The idea is that you can specify that this invocation only handle http callbacks while this other one handles AMQP.

Sinatra Helper

One idea I'd REALLY like to come to fruition is the Sinatra Helper. I envision it working something like this:

    require 'sinatra/base'

class MyApp < Sinatra::Base
register Noah::Sinatra

noah_server "http://localhost:5678"
noah_node_name "myself"
noah_app_name "MyApp"
noah_token "somerandomlongstring"

dynamic_get :database_server
dynamic_set :some_other_variable, "foobar"
watch :this_other_node

end

The idea is that the helper allows you to register your application very easily with Noah for other components in your environment to be know. As a byproduct, you get the ability to get/set certain configuration parameters entirely in Noah. The watch setting is kind of cool as well. What will happen is if you decide to watch something this way, the helper will create a random (and yes, secure) route in your application that watch events can notify. In this way, your Sinatra application can be notified of any changes and will automatically "reconfigure" itself.

Obviously I'd love to see other implementations of this idea for other languages and frameworks.

Long term changes

There aren't so much specific list items here as general themes and ideas. While I list these as long term, I've already gotten an offer to help with some of them so they might actually get out sooner.

Making Noah itself distributed

This is something I'm VERY keen on getting accomplished and would really consider it the fruition of what Noah itself does. The idea is simply that multiple Noah servers themselves are clients of other Noah servers. I've got several ideas about how to accomplish this but I got an interesting follow up from someone on Github the other day. He asked what my plans were in this area and we had several lengthy emails back and forth including an offer to work on this particular issue.

Obviously there are a whole host of issues to consider. Race conditions in ordered delivery of Watch callbacks (getting a status "down" after a status "up" when it's supposed to be the other way around..) and eventual consistency spring to mind first.

The general architecture idea that was offered up is to use NATS as the mechanism for accomplishing this. In the same way that there would be AMQP callback support, there would be NATS support. Additional Noah servers would only need to know one other member to bootstrap and everything else happens using the natural flows within Noah.

The other part of that is how to handle the Redis part. The natural inclination is to use the upcoming Redis clustering but that's not something I want to do. I want each Noah server to actually include its OWN Redis instance "embedded" and not need to rely on any external mechanism for replication of the data. Again, the biggest validation of what Noah is designed to do is using only Noah itself to do it.

Move off Redis/Swappable persistence

If NATS says anything to me, it says "Why do you even need Redis?". If you recall, I went with Redis because it solved multiple problems. If I can find a persistence mechanism that I can use without any external service running, I'd love to use it.

ZeroMQ

If I were to end up moving off Redis, I'd need a cross platform and cross language way to handle the pubsub component. NATS would be the first idea but NATS is Ruby only (unless I've missed something). ZeroMQ appears to have broad language and platform support so writing custom agents in the same vein as the Redis PUBSUB method should be feasible.

Nanite-style agents

This is more of a command-and-control topic but a set of high-performance specialized agents on systems that can watch the PUBSUB backend or listen for callbacks would be awesome. This would allow you really integrate Noah into your infrastructure beyond the application level. Use it to trigger a puppet or chef run, reboot instances or do whatever. This is really about bringing what I wanted to accomplish with Vogeler into Noah.

The PAXOS question

A lot of people have asked me about this. I'll state right now that I can only make it through about 20-30% of any reading about Paxos before my brain starts to melt. However in the interest of proving myself the fool, I think it would be possible to implement some Paxos like functionality on top of Noah. Remember that Noah is fundamentally about fully disconnected nodes. What better example of a network of unreliable processors than ones that never actually talk to each other. The problem is that the use case for doing it in Noah is fairly limited so as not to be worth it.

The grand scheme is that Noah helps enable the construction of systems where you can say "This component is free to go off and operate in this way secure in the knowledge that if something it needs to know changes, someone will tell it". I did say "grand" didn't I? At some point, I may hit the limit of what I can do using only Ruby. Who knows.

Wrap up - Part 4

Again with the recap

Get to 1.0 with a stable and fixed set of functionality
Nurture the Noah ecosystem
Make it easy for people to integrate Noah into thier applications
Get all meta and make Noah itself distributed using Noah
Minimize the dependencies even more
Build skynet

I'm not kidding on that last one. Ask me about Parrot AR drones and Noah sometime

If you made it this far, I want to say thank you to anyone who read any or all of the parts. Please don't hesitate to contact me with any questions about the project.

On Noah - Part 3

2011-05-18T22:14:00.000-04:00

On Noah - Part 3

This is the third part in a series on Noah. Part 1 and Part 2 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. This post is discussing the technology stack under Noah and the reasoning for it.

A little back story

I've told a few people this but my original intention was to use Noah as a way to learn Erlang. However this did not work out. I needed to get a proof of concept out much quicker than the ramp up time it would take to learn me some Erlang. I had this grandiose idea to slap mnesia, riak_core and webmachine into a tasty ball of Zookeeper clonage.

I am not a developer by trade. I don't have any formal education in computer science (or anything for that matter). The reason I mention this is to say that programming is hard work for me. This has two side effects:

It takes me considerably longer than a working developer to code what's in my head
I can only really learn a new language when I have an itch to scratch. A real world problem to model.

So in the interest of time, I fell back to a language I'm most comfortable with right now, Ruby.

Sinatra and Ruby

Noah isn't so much a web application as it is this 'api thing'. There's no proper front end and honestly, you guys don't want to see what my design deficient mind would create. I like to joke that in the world of MVC, I stick to the M and C. Sure, APIs have views but not in the "click the pretty button sense".

I had been doing quite a bit of glue code at the office using Sinatra (and EventMachine) so I went with that. Sinatra is, if you use sheer number of clones in other languages as an example, a success for writing API-only applications. I also figured that if I wanted to slap something proper on the front, I could easily integrate it with Padrino.

But now I had to address the data storage issue.

Redis

Previously, as a way to learn Python at another company, I wrote an application called Vogeler. That application had a lot of moving parts - CouchDB for storage and RabbitMQ for messaging.

I knew from dealing with CouchDB on CentOS5 that I wasn't going to use THAT again. Much of it would have been overkill for Noah anyway. I realized I really needed nothing more than a key/value store. That really left me with either Riak or Redis. I love Riak but it wasn't the right fit in this case. I needed something with a smaller dependency footprint. Mind you Riak is VERY easy to install but managing Erlang applications is still a bit edgy for some folks. I needed something simpler.

I also realized early on that I needed some sort of basic queuing functionality. That really sealed Redis for me. Not only did it have zero external dependencies, but it also met the needs for queuing. I could use lists as dedicated direct queues and I could use the built-in pubsub as a broadcast mechanism. Redis also has a fast atomic counter that could be used to approximate the ZK sequence primitive should I want to do that.

Additionally, Redis has master/slave (not my first choice) support for limited scaling as well as redundancy. One of my original design goals was that Noah behave like a traditional web application. This is a model ops folks understand very well at this point.

EventMachine

When you think asynchronous in the Ruby world, there's really only one tool that comes to mind, EventMachine. Noah is designed for asynchronous networks and is itself asynchronous in its design. The callback agent itself uses EventMachine to process watches. As I said previously, this is simply using an EM friendly Redis driver that can do PSUBSCRIBE (using em-hiredis) and send watch messages (using em-http-request since we only support HTTP by default).

Ohm

Finally I slapped Ohm on top as the abstraction layer for Redis access. Ohm, if you haven't used it, is simply one of if not the best Ruby library for working with Redis. It's easily extensible, very transparent and frankly, it just gets the hell out of your way. A good example of this is converting some result to a hash. By default, Ohm only returns the id of the record. Nothing more. It also makes it VERY easy to drop past the abstraction and operate on Redis directly. It even provides helpers to get the keys it uses to query Redis. A good example of this is in the Linking and Tagging code. The following is a method in the Tag model:

    def members=(member)
      self.key[:members].sadd(member.key)
      member.tag! self.name unless member.tags.member?(self)
    end

Because Links and Tags are a one-to-many across multiple models, I drop down to Redis and use sadd to add the object to a Redis set of objects sharing the same tag.

It also has a very handy feature which is how the core of Watches are done. You can define hooks at any phase of Redis interaction - before and after saves, creates, updates and deletes. the entire Watch system is nothing more than calling these post hooks to format the state of the object as JSON, add metadata and send the message using PUBLISH messages to Redis with the Noah namespace as the channel.

Distribution vectors

I've used this phrase with a few people. Essentially, I want as many people as possible to be able to use the Noah server component. I've kept the Ruby dependencies to a minimum and I've made sure that every single one works on MRI 1.8.7 up to 1.9.2 as well as JRuby. I already distribute the most current release as a war that can be deployed to a container or run standalone. I want the lowest barrier to entry to get the broadest install base possible. When a new PaaS offering comes out, I pester the hell out of anyone I can find associated with it so I can get deploy instructions written for Noah. So far you can run it on Heroku (using the various hosted Redis providers), CloudFoundry and dotcloud.

I'm a bit more lax on the callback daemon. Because it can be written in any language that can talk to the Redis pubsub system and because it has "stricter" performance needs, I'm willing to make the requirements for the "official" daemon more stringent. It currently ONLY works on MRI (mainly due to the em-hiredis requirement).

Doing things differently

Some people have asked me why I didn't use technology A or technology B. I think I addressed that mostly above but I'll tackle a couple of key ones.

ZeroMQ

The main reason for not using 0mq was that I wasn't really aware of it. Were I to start over and still be using Ruby, I'd probably give it a good strong look. The would still be the question of the storage component though. There's still a possible place for it that I'll address in part four.

NATS

This was something I simply had no idea about until I started poking around the CloudFoundry code base. I can almost guarantee that NATS will be a part of Noah in the future. Expect much more information about that in part four.

MongoDB

You have got to be kidding me, right? I don't trust my data (or anyone else's for that matter) to a product that doesn't understand what durability means when we're talking about databases.

Insert favorite data store here

As I said, Redis was the best way to get multiple required functionality into a single product. Why does a data storage engine have a pubsub messaging subsystem built in? I don't know off the top of my head but I'll take it.

Wrap up - Part 3

So again, because I evidently like recaps, here's the take away:

The key components in Noah are Redis and Sinatra
Noah is written in Ruby because of time constraints in learning a new language
Noah strives for the server component to have the broadest set of distribution vectors as possible
Ruby dependencies are kept to a minimum to ensure the previous point
The lightest possible abstractions (Ohm) are used.
Stricter requirements exist for non-server components because of flexibility in alternates
I really should learn me some erlang
I'm not a fan of MongoDB

If you haven't guessed, I'm doing one part a night in this series. Tomorrow is part four which will cover the future plans for Noah. I'm also planning on a bonus part five to cover things that didn't really fit into the first four.

On Noah - Part 2

2011-05-17T22:38:00.000-04:00

On Noah - Part 2

This is the second part in a series on Noah. Part 1 is available here

In part one of this series, I went over a little background about ZooKeeper and how the basic Zookeeper concepts are implemented in Noah. In this post, I want to go over a little bit about a few things that Noah does differently.

Noah Primitives

As mentioned in the previous post, Noah has 5 essential data types, four of which are what I've interchangeably refered to as either Primitives and Opinionated models. The four primitives are Host, Service, Application and Configuration. The idea was to map some common use cases for Zookeeper and Noah onto a set of objects that users would find familiar.

You might detect a bit of Nagios inspiration in the first two.

Host: Analogous to a traditional host or server. The machine or instance running the operating system. Unique by name.
Service: Typically mapped to something like HTTP or HTTPS. Think of this as the listening port on a Host. Services must be bound to Hosts. Unique by service name and host name.
Application: Apache, your application (rails, php, java, whatever). There's a subtle difference here from Service. Unique by name.
Configuration: A distinct configuration element. Has a one-to-many relationship with Applications. Supports limited mime typing.

Hosts and Services have a unique attribute known as status. This is a required attribute and is one of up,down or pending. These primitives would work very well integrated into the OS init process. Since Noah is curl-friendly, you could add something globally to init scripts that updated Noah when your host is starting up or when some critical init script starts. If you were to imagine Noah primitives as part of the OSI model, these are analagous to Layers 2 and 3.

Applications and Configurations are intended to feel more like Layer 7 (again, using our OSI model analogy). The differentiation is that your application might be a Sinatra or Java application that has a set of Configurations associated with it. Interestingly enough, you might choose to have something like Tomcat act as both a Service AND an Application. The aspect of Tomcat as a Service is different than the Java applications running in the container or even Tomcat's own configurations (such as logging).

One thing I'm trying to pull off with Configurations is limited mime-type support. When creating a Configuration in Noah, you can assign a format attribute. Currently 3 formats or types are understood:

string
json
yaml

The idea is that, if you provide a type, we will serve that content back to you in that format when you request it (assuming you request it that way via HTTP headers). This should allow you to skip parsing the JSON representation of the whole object and instead use it directly. Right now this list is hardcoded. I have a task to convert this.

Hosts and Services make a great "canned" structure for building a monitoring system on top of Noah. Applications and Configurations are a lightweight configuration management system. Obviously there are more uses than that but it's a good way to look at it.

Ephemerals

Ephemerals, as mentioned previously, are closer to what Zookeeper provides. The way I like to describe Ephemerals to people is a '512 byte key/value store with triggers' (via Watch callbacks). If none of the Primitives fit your use case, the Ephemerals make a good place to start. Simply send some data in the body of your post to the url and the data is stored there. No attempt is made to understand or interpret the data. The hierarchy of objects in the Ephemeral namespace is completely arbitrary. Data living at /ephemerals/foo has no relationship with data living at /ephemerals/foo/bar.

Ephemerals are also not browseable except via a Linking and Tagging.

Links and Tags

Links and Tags are, as far as I can tell, unique to Noah compared to Zookeeper. Because we namespace against Primitives and Ephemerals, there existed the need to visualize objects under a custom hierarchy. Currently Links and Tags are the only way to visualize Ephemerals in a JSON format.

Tags are pretty standard across the internet by now. You might choose to tag a bunch of items as production or perhaps group a set of Hosts and Services as out-of-service. Tagging an item is a simple process in the API. Simply PUT the name of the tag(s) to the url of a distinct named item appended by tag. For instance, the following JSON posted to /applications/my_kick_ass_app/tag with tag the Application my_kick_ass_app with the tags sinatra, production and foobar:

{"tags":["sinatra", "production", "foobar"]}

Links work similar to Tags (including the act of linking) except that the top level namespace is now replaced with the name of the Link. The top level namespace in Noah for the purposes of Watches is //noah. By linking a group of objects together, you will be able to (not yet implemented), perform operations such as Watches in bulk. For instance, if you wanted to be informed of all changes to your objects in Noah, you would create a Watch against //noah/*. This works fine for most people but imagine you wanted a more multi-tenant friendly system. By using links, you can group ONLY the objects you care about and create the watch against that link. So //noah/* becomes //my_organization/* and only those changes to items in that namespace will fire for that Watch.

The idea is also that other operations outside of setting Watches can be applied to the underlying object in the link as well. The name Link was inspired by the idea of symlinking.

Watches and Callbacks

In the first post, I mentioned that by nature of Noah being "disconnected", Watches were persistent as opposed to one-shot. Additionally, because of the pluggable nature of Noah Watches and because Noah has no opinion regarding the destination of a fired Watch, it becomes very easy to use Noah as a broadcast mechanism. You don't need to have watches for each interested party. Instead, you can create a callback plugin that could dump the messages on an ActiveMQ Fanout queue or AMQP broadcast exchange. You could even use multicast to notify multiple interested parties at once.

Again, the act of creating a watch and the destination for notifications is entirely disconnected from the final client that might use the information in that watch event.

Additionally, because of how changes are broadcast internally to Noah, you don't even have to use the "official" Watch method. All actions in Noah are published post-commit to a pubsub queue in Redis. Any language that supports Redis pubsub can attach directly to the queue and PSUBSCRIBE to the entire namespace or a subset. You can write your own engine for listening, filtering and notifying clients.

This is exactly how the Watcher daemon works. It attaches to the Redis pubsub queue, makes a few API calls for the current registered set of watches and then uses the watches to filter messages. When a new watch is created, that message is like any other change in Noah. The watcher daemon sees that and immediately adds it to its internal filter. This means that you can create a new watch, immediately change the watched object and the callback will be made.

Wrap up - Part Two

So to wrap up:

Noah has 5 basic "objects" in the system. Four of those are opinionated and come with specific contracts. The other is a "dumb" key/value store of sorts.
Noah provides Links and Tags as a way to perform logical grouping of these objects. Links replace the top-level hierarchy.
Watches are persistent. The act of creating a watch and notifying on watched objects is disconnected from the final recipient of the message. System A can register a watch on behalf of System B.
Watches are nothing more than a set of filters applied to a Redis pubsub queue listener. Any language that supports Redis and its pubsub queue can be a processor for watches.
You don't even have to register any Watches in Noah if you choose to attach and filter yourself.

Part three in this series will discuss the technology stack under Noah and the reasoning behind it. A bit of that was touched on in this post. Part four is the discussion about long-term goals and roadmaps.

On Noah - Part 1

2011-05-16T23:16:00.001-04:00

On Noah - Part 1

This is the first part in a series of posts going over Noah

As you may have heard (from my own mouth no less), I've got a smallish side project I've been working on called Noah.

It's a project I've been wanting to work on for a long time now and earlier this year I got off my ass and started hacking. The response has been nothing short of overwhelming. I've heard from so many people how they're excited for it and nothing could drive me harder to work on it than that feedback. To everyone who doesn't run away when I talk your ear off about it, thank you so much.

Since I never really wrote an "official" post about it, I thought this would be a good opportunity to talk about what it is, what my ideas are and where I'd like to see it go in the future.

So why Noah?

fair warning. much of the following may be duplicates of information in the Noah wiki

The inspiration for Noah came from a few places but the biggest inspiration is Apache Zookeeper. Zookeeper is one of those things that by virtue of its design is a BUNCH of different things. It's all about perspective. I'm going to (yet again) paste the description of Zookeeper straight from the project site:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Now that might be a bit confusing at first. Which is it? Is it a configuration management system? A naming system? It's all of them and, again, it's all about perspective.

Zookeeper, however, has a few problems for my standard use case.

Limited client library support
Requires persistent connections to the server for full benefit

By the first, I mean that the only official language bindings are C and Java. There's contributed Python support and Twitter maintains a Ruby library. However all of these bindings are "native" and must be compiled. There is also a command-line client that you can use for interacting as well - one in Java and two C flavors.

The second is more of a showstopper. Zookeeper uses the client connection to the server as in-band signaling. This is how watches (discussed in a moment) are communicated to clients. Persistent connections are simply not always an option. I can't deploy something to Heroku or AppEngine that requires that persistent connection. Even if I could, it would be cost-prohibitive and honestly wouldn't make sense.

Looking at the list of features I loved about ZK, I thought "How would I make that work in the disconnected world?". By that I mean what would it take to implement any or all of the Zookeeper functionality as a service that other applications could use?

From that thought process, I came up with Noah. The name is only a play on the concept of a zookeeper and holds no other real significance other than irritation at least two people named Noah when I talk about the project.

So working through the feature list, I came up with a few things I REALLY wanted. I wanted Znodes, Watches and I wanted to do it all over HTTP so that I could have the broadest set of client support. JSON is really the defacto standard for web "messaging" at this point so that's what I went with. Basically the goal was "If your language can make HTTP requests and parse JSON, you can write a Noah client"

Znodes and Noah primitives

Zookeeper has a shared hierarchical namespace similar to a UNIX filesystem. Points in the hierarchy are called znodes. Essentially these are arbitrary paths where you can store bits of data - up to 1MB in size. These znodes are unique absolute paths. For instance:

    /
/systems
/foo
/bar
/networks
/kansas
/router-1
/router-2

Each fully qualified path is a unique znode. Znodes can be ephemeral or persistent. Zookeeper also has some primitives that can be applied to znodes such as 'sequence`.

When I originally started working on Noah, so that I could work with a model, I created some base primitives that would help me demonstrate an example of some of the use cases:

Host
Service
Application
Configuration

These primitives were actual models in the Noah code base with a strict contract on them. As an example, Hosts must have a status and can have any number of services associated with them. Services MUST be tied explicity to a host. Applications can have Configurations (or not) and Configurations can belong to any number of Applications or not. Additionally, I had another "data type" that I was simply calling Ephemerals. This is similar to the Zookeeper znode model. Originally I intended for Ephemerals to be just that - ephemeral. But I've backed off that plan. In Noah, Ephemerals can be either persistent or truely ephemeral (not yet implemented).

So now I had a data model to work with. A place to store information and flexibility to allow people to use the predefined primitives or the ephemerals for storing arbitrary bits of information.

Living the disconnected life

As I said, the model for my implementation was "disconnected". When thinking about how to implement Watches in a disconnected model, the only thing that made sense to me was a callback system. Clients would register an interest on an object in the system and when that object changed, they would get notified by the method of their choosing.

One thing about Watches in Zookeeper that annoys me is that they're one-shot deals. If you register a watch on a znode, once that watch is triggered, you have to REREGISTER the watch. First off this creates, as documented by the ZK project, a window of opportunity where you could miss another change to that watch. Let's assume you aren't using a language where interacting with Zookeeper is a synchronous process:

Connect to ZK
Register watch on znode
Wait
Change happens
Watch fires
Process watch event
Reregister watch on znode

In between those last two steps, you risk missing activity on the znode. In the Noah world, watches are persistent. This makes sense for two reasons. The first is that the latency between a watch callback being fired and proccessed could be much higher than the persistent connection in ZK. The window of missed messages is simply much greater. We could easily be talking 100's of milliseconds of latency just to get the message and more so to reregister the watch.

Secondly, the registration of Watches in Noah is, by nature of Noah's design and as a byproduct, disconnected from the consumer of those watches. This offers much greater flexibility in what watches can do. Let's look at a few examples.

First off, it's important to understand how Noah handles callbacks. The message format of a callback in Noah is simply a JSON representation of the changed state of an object and some metadata about the action taken (i.e. delete, create, update). Watches can be registered on distinct objects, a given path (and thus all the children under that path) and further refined down to a given action. Out of the box, Noah ships with one callback handler - http. This means that when you register a watch on a path or object, you provide an http endpoint where Noah can post the aforementioned JSON message. What you do with it from there is up to you.

By virtue of the above, the callback system is also designed to be 'pluggable' for lack of a better word. While the out of the box experience is an http post, you could easily write a callback handler that posted the message to an AMQP exchange or wrote the information to disk as a flat file. The only requirement is that you represent the callback location as a single string. The string will be parsed as a url and broken down into tokens that determine which plugin to call.

So this system allows for you to distribute watches to multiple systems with a single callback. Interestingly enough, this same watch callback system forms the basis of how Noah servers will share changes with each other in the future.

Wrap up - Part 1

So wrapping up what I've discussed, here are the key take aways:

Noah is a 'port' of specific Zookeeper functionality to a disconnected and asynchronous world
Noah uses HTTP and JSON as the interface to the server
Noah has both traditional ZK-style Ephemerals as well as opinionated Primitives
Noah uses a pluggable callback system to approximate the Watch functionality in Zookeeper
Clients can be written in any language that can speak HTTP and understand JSON (yes, even a shell script)

Part 2 and beyond

In part two of this series we'll discuss some of the additions to Noah that aren't a part of Zookeeper such as Tags and Links. Part 3 will cover the underlying technology which I am intentionally not discussing at this point. Part 4 will be a roadmap of my future plans for Noah.

Who owns my availability?

2011-04-22T02:51:00.001-04:00

Hey did you know EC2 had problems today? Yeah nothing major just a total effing collapse of the EBS system at US-EAST-1.

You know what that means....

"Hey guys, can anyone tell me who owns my availability?"

"Internet learns lesson of putting "all eggs in the EC2 basket". Buy your own machines, brothers."

I could go on....but I won't. I'm also going to stop short of posting a CeeLo video at this point.

Your stupid little comments mean nothing. I especially find it hilarious that someone from Twitter would make a comment availability. I also find the short-lived memory of some people hilarious (paraphrasing here):

"Thank god we're hosted on Joyent/Linode/My mom's basement"

Please. Your attempt to curry favor and free service with your provider are transparent and frankly, makes you look stupid.

Yo Netflix/SimpleGeo/JRandomDude I'm happy for you and and all. I'ma let you finish but....

So who DOES own my availability?
Here's a hint; it's not always that simple.

Yes, the ultimate responsibility for those impacted lies with those who were impacted but let's look at a few facts (or excuses - if you're being a dick about it):

Not everyone has the resources of a Netflix
Comparing anyone else's EC2 usage to Netflix is simply retarded. It's a lot like working with an ex-Google employee (I've worked with a few). They have some awesome ideas and learned some great stuff there but guess what? About 85% of it is USELESS to anyone except someone the size of Google. What works at Google doesn't work at my company.

It's not even a matter of scaling down the concept. It's simply NOT possible. Yeah let me just go buy a shipping container and build a datacenter in a box. Hardware failure? Replace the box with one off the shelf. Oh wait, not everyone has a warehouse of replacement servers. People have trouble getting a few spare hard drives to swap out.

Telling someone that they should just do what Netflix does makes you look stupid. Not them.

WE used Joyent/Linode/GoGrid/My mom's basement
Really? Really? I'm not being an AWS fanboy here but here is a simple fact: No other 'cloud' provider comes even REMOTELY close to the feature set of AWS. No one. Not only does no one come close but Amazon is CONSTANTLY iterating on new stuff to widen the gap even more.

It's not like your provider hasn't had a major outage in recent memory. And comparing an effing VPS provider to Amazon? You seriously just don't get it.

You should have designed around this possibility
Well no shit, sherlock. Guess what, it was rejected. Why? Who knows? Who cares? It's irrelevant. Sometimes the decision isn't ours to make. In the REAL world, people have to balance risk vs. reward.

Here's a tidbit of information. At EVERY single company I've been at where I was involved with architecting a solution from the ground up, we never had redundancy built in from the get go. Did I find it appalling. Absolutely but the choice wasn't mine. I did the best I could to prevent anything that would make adding it TOO difficult later on but we didn't have our DR site online from day one. We sometimes had to accrue a little technical debt. The best we could do was to minimize it as much as possible.

Designing around failure is not the same as designing for the worse case scenario. Sometimes you just have to accept that "if component X has Y number of failures, we're going to have an outage". If you have the ability to deal with it now (resources/money/whatever), then that's awesome. Sometimes you just have to accept that risk.

Oh sure I'd love to use (insert buzzword/concurrent/distributed language of the day) here. But I can't. It would be totally awesome if everything were designed from the ground up to handle that level of failure but it's not.

And another thing
The thing that bothers me most is the two-faced attitude around it all.

On one hand people are telling you it's stupid to host your own hardware. On the other hand they'll laugh at you when your provider has an outage and tell you that you should have built your own.

On one hand they'll tell you it's stupid to use some non-traditional new-fangled language and on the other hand laugh at you when you could have avoided all these problems if you had just used non-traditional new-fangled language.

On one hand they'll tell you that you should use insert-traditional-RDBMS here and on the other hand say that it's your fault for not rearchitecting your entire codebase around some NoSQL data store.

Not everyone has the same options. I hate the phrase "hindsight is 20/20". Why? Because it's all relevant. Sometimes you don't know that something is the wrong choice till it bites you in the ass. Hindsight in technology is only valuable for about a year. Maybe 6 months. Technology moves fast. It's easy to say that someone should have used X when you don't realize that they started working on things six months before X came along. If you have that kind of foresight, I'd love to hire you to play the stock market for me.

Not everyone has the luxury of switching midstream. You have to make the most what technology is available. If you keep chasing the latest and greatest, you'll never actually accomplish anything.

Are these excuses? Absolutely but there's nothing inherently wrong with excuses. You live and learn. So to those affected by the outage (still on-going mind you), take some comfort. Learn from your mistakes. The worst thing you could do at this point would be to NOT change anything. At a minimum, if you aren't the decision maker, you should document your recommendations and move on. If you are the decision maker, you need to..you know...decide if the risk of this happening again is acceptable.

Sinatra, Noah and CloudFoundry - the dirty details

2011-04-15T04:06:00.001-04:00

So via some magical digital god, my signup for Cloud Foundry got processed. Obviously my first thought was to try and get Noah up and running. Cloud Foundry is a perfect fit for Noah because I have access to Redis natively. I have a working setup now but it took a little bit of effort.

Getting set up
As with everything these days, my first action was to create a gemset. I'll not bore you with that process but for the sake of this walkthrough, let's use a 1.9.2 gemset called 'cfdev'.

The VMC getting started guide has most of the information you'll need but I'm going to duplicate some of it here for completeness:

 gem install vmc
 vmc target api.cloudfoundry.com
 vmc login

And we're ready to rock. The VMC command line help is very good with the exception that the optional args aren't immediately visible.

vmc help options

will give you a boatload of optional flags you can pass in. One that was frequently used during the demos at launch was '-n'. I would suggest you NOT use that for now. The prompts are actually pretty valuable.

So in the case of Noah, we know we're going to need a Redis instance. Because everything is allocated dynamically CloudFoundry makes heavy use of environment variables to provide you with important settings you'll need.

First Attempt
If you watched the demo (or read the quickstart Sinatra example), there's a demo app called 'env' that they walk you through. You're going to want to use that when troubleshooting things. My first task was to duplicate the env demo so I could take a gander at the variables I would need for Redis. For the record, the steps I'm documenting here might appear out of order and result in some wasted time. I'm one of those guys who reads the instructions 2 days after I've broken something so you have an idea of what I did here:

 vmc help
 vmc services
 vmc create-service redis redis-noah
 vmc services

At this point, I now have a named instance of redis. The reason I felt safe enough doing this now is that I noticed in the help two service commands - 'bind-service' and 'unbind-service'. I figured it was easy enough to add the service to my app based on those options.

So go ahead and create the env app per the getting started documentation. If you followed my suggestion and DIDN'T disable prompts, you'll get the option to bind you app to a service when you push the first time. If you're running without prompts (using the '-n' option), you'll probably want to do something like this:

vmc push myenvapp --url ohai-env.cloudfoundry.com
vmc bind-service my_redis_service myenvapp

If you visit the url you provided (assuming it wasn't taken already?) at /env, you'll get a big dump of all the environment variables. The ones that you'll need be using most are probably going to be under `VCAP_SERVICES`. What you'll probably also notice is that `VCAP_SERVICES` is a giant JSON blob. Now you may also notice that there's a nice `VMC_REDIS` env variable there. It's pretty useless primarily because there's also a GIANT warning in the env output that all `VMC_` environment variables are deprecated but also because your redis instance requires a password to access which means you need to traverse the JSON blob ANYWAY.

So if we paste the blog into an IRB session we can get a better representation. I wish I had done that first. Instead, I reformatted it with jsonlint dutifully wrote the following madness:

which I spent a good 30 minutes troubleshooting before I realized that it's actually an array. It should have been this:

So now that I had all the variables in place, I went about converting my heroku Noah demo . That demo uses a Gemfile and a rackup file so I figured it would work just fine here. No such luck. This is where things get hairy.

Sinatra limitations
The short of it is that Sinatra application support on CF right now is a little bit of a CF. It's very basic and somewhat brute force. If you're running a single file sinatra application, it will probably work. However if you're running anything remotely complex, it's not going to work without considerable effort. Noah is even more of a special case because it's distributed as a gem. This actually has some benefit as I'll mention farther down. However it's not really "compatible" with the current setup on Cloud Foundry. Here's the deal:

If you look here, You'll see that the way your sinatra application is start is by calling ruby (with or without bundler depending) against what it detects as your main app file. This is done here which leads us all the way to this file:

`https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/manifests/sinatra.yml`

Essentially for sinatra applications, the first .rb file it comes across with 'require sinatra', is considered the main app file. Bummer. So config.ru is out. The next step is to rename it to a '.rb' file and try again. This is where I spent most of my troubleshooting. There's a gist of the things I tried (including local testing) here:

`https://gist.github.com/920552`

Don't jump to the solution just yet because it's actually incomplete. This troubleshooting led to another command you'll want to remember:

vmc files myapp logs/stderr.log

I found myself typing it a lot during this process. For whatever reason, possibly due to bundler or some other vcap magic I've not discovered yet what works at home does not work on Cloud Foundry exactly the same. That's fine, it's just a matter of knowing about it. It also didn't help that I wasn't getting any output at all for the entire time I was trying to figure out why config.ru didn't work.

Thanks to Konstantin Haase for his awesome suggestion in #sinatra. The trick here was to mimic what rackup does. Because the currently released Noah gem has a hard requirement on rack 1.2.1, his original suggestion wasn't an exact fit but I was able to get something working:

https://gist.github.com/921292

So what did we do?
Ensure that the wrapper file is picked up first by making sure it's the ONLY rb file uploaded with `require sinatra` at the top.
Because of a bug in rack 1.2.1 with Rack::Server.new, I HAD to create a file called config.ru. The fix in rack 1.2.2 actually honors passing all the options into the constructor without needing the config.ru file.
Explicitly connect to redis before we start the application up.

The last one was the almost as big of a pain in the ass as getting the application to start up.

I think (and I'm not 100% sure) that you are prohibited from setting environment variables inside your code. Because of the convoluted way I had to get the application started, I couldn't use my sinatra configuration block properly (`set :redis_url, blahblahblah`). I'm sure it's possible but I'm not an expert at rack and sinatra. I suppose I could have used Noah::App.set but at this point I was starting to get frustrated. Explicitly setting it via Ohm.connect worked.

I'm almost confident of this environment variable restriction because you can see options in 'vmc help' that allow you to pass environment variables into your application. That would work fine for most cases except that I don't know what the redis values are outside of the app and they're set dynamically anyway.

So where can things improve?
First off, this thing is in beta. I'm only adding this section because it'll serve as a punch list of bugs for me to fix in vcap ;)

Sinatra support needs to be more robust.

You can see that the developers acknowledged that in the staging plugin code. There are TODOs listed. It's obvious that a sinatra application of any moderate complexity wasn't really tested and that's fine. The building blocks are there and the code is opensource. I'll fix it myself (hopefully) and submit a pull request.

Allow override of the main app file from VMC.

It appears from the various comments that the node.js support suffers some of the same brute force detection routines. An option to pass in what the main applictation file is would solve some of that.

Document the environment variable restrictions.

I didn't see any documentation anywhere about that restriction (should it exist). I could be doing something wrong too. It's worth clarifying.

Better error reporting for failed startups

I'm not going to lie but I spent a LONG time troubleshooting the fact that the app simply wasn't starting up. The default output when a failure happens during deploy is the staging.log file. All this EVER contained was the output from bundler. It should include the output of stderr.log and stdout.log as well. Also an explicit message should be returned if the main app file can't be detected. That would have solved much of my frustration up front.

That's just the stuff I ran into to get things going. The first item is the biggest one. If you're writing a monolithic single-file sinatra app, the service will work GREAT. If you aren't, you'll have to jump through hoops and wrapper scripts for now. Supporting rackup files for Sinatra and Rack apps will go a long way to making things even more awesome.

One pleasant surprise I found was that, despite what I was told, I didn't need to include every gem in my Gemfile. Because Noah itself has its deps, Bundler pulls those in for me.

I've created a git repo with the code as well as a quickstart guide for getting your own instance running. You can find it here:

https://github.com/lusis/noah-cloudfoundry-demo

Operational Primitives

2011-04-14T07:36:00.002-04:00

"Infrastructure as code". I love the phrase. Where devops is a word that is sadly open to so much (mis)interpretation, "Infrastructure as code" is pretty clear. Treat your infrastructure as code. Programmable. Testable. Deployable.

But when you start to really think about that concept, there's a deep dive you can take, navigating various programming and computer science constructs and applying those to your infrastructure.

I've been working pretty heavily on getting the first API stable release of Noah out the door. It's been a challenge with the schedule I have to work on it - which is essentially "when everyone else in the house is asleep and I'm awake'. Last night, I came to a fork in the road where I needed to make a decision. This decision would lock me into an API path that I was unwilling to change for a while. Nobody wants to use a service or tool with a constantly changing API. I needed to shit or get off the pot, to use a creative euphemism. With the announcements of both Doozer and riak_zab, it was clear that I wasn't the only person attempting to tackle the ZooKeeper space.

Since Github lacks any facility for soliciting project feedback (hint hint, @github), I decided to create a Wufoo form and tweet it out. I don't have a very big audience but I was hoping it would at least get to the people who were likely to use Noah. The form was fairly simple with one question on something that I had pretty summarily dismissed early on - HATEOAS (hypermedia as the engine of application state).

A small HATEOAS diversion

The HATEOAS debate is a lot like Linux vs. GNU/Linux. It's fairly esoteric but there's some meat to the matter. My problem with it was simply that, despite what Roy Fielding and others intended, REST had taken on a new definition and it wasn't the strict HATEOAS one. Additionally, I found it VERY difficult to map HATEOAS concepts to JSON. JSON is a great format but a rich document structure is not (rightly so) part of the format. It's intended to be simple, easily read and cleanly mapped to machine readable format. It also felt like extra work on the part of the API consumer. The concepts that we use when reading a website (click this link, read this list, click this link) are simple not necessary when you have a contextually relevant (or descriptive) URL scheme. True, as a human I don't make changes in the URL bar to navigate a site (I use the links provided by the site) but when it comes to dealing with an API, I don't exhibit the same usage patterns as a web browser. I'm making distinct atomic transactions (DELETE this resource, PUT this resource) at a given endpoint. These simply aren't the same as filling out forms and are only tangentially related. I'm simply not willing to force someone to parse a JSON object to tell them how to create a new object in the system. The API for Noah is fairly simple as it is. Objects in the system have only two or three required attributes for a given operation and normally one of those attributes is directly inferable from the URL.

But based on the poll results thus far, I wanted to give the idea fair consideration which led me to think about what types of objects Noah had in its system.

Primitives

For those who aren't familiar or simple don't know, there's a term in computer science and programming called "Primitive". It essentially means a basic data type in a language from which other complex data types are created. A building block if you will. Some easily grokable examples of primitives are Characters and Integers. Some languages actually have ONE primitive like Object and everything is built on top of that. You could get into a semantic argument about a lot of this so I'm going to leave it at that.

But back to the phrase "Infrastucture as code". If we start looking at how we "program" our infrastructure, what are the "primitives" that our language supports. I inadvertently created some of these in Noah. I've been calling them the "opinionated models" but really in the infrastructure programming language of Noah, they're primitives.

When this hit me last night, I immediately pulled out the tablet and went to work on a mind map. I laid out what I had already implemented as primitives in Noah:

Host
Service
Application
Configuration

I then started to think about other concepts in Noah. Were Ephmerals really a primitive. Not really. If anything Ephemerals are more similar to ruby's BasicObject. The only real attribute Ephemerals have are a path (similar to the object_id).

So what else would be our modern operational primitives? Remember that we're talking about building blocks here. I don't want to abstract out too much. For instance you could simply say that a "Resource" is the only real operational primitive and that everything else is built on top of that. Also consider that languages such as Python have some richer primitives built-in like tuples.

One interesting thought I had was the idea that "State" was a primitive. Again, in the world of operations and infrastructure, one of your basic building blocks is if something is available or not - up or down. At first glance it would appear that this maps pretty cleanly to a Boolean (which is a primitive in most languages) however I think it's a richer primitive than that.

In the world of operations, State is actually quaternary (if that's the right word) rather than binary. There are two distinct areas between up and down that have dramatically different implications on how you interact with it:

Up
Down
Pending Up
Pending Down

Currently in Noah, we simple have Up, Down and Pending but something that is in the State of shutting down is grossly different than something in the state of Starting up. Look at a database that is queiscing connections. It's in a state of "Pending Down". It's still servicing existing requests. However a database in the state of "Pending Up" is NOT servicing any requests.

So I'm curious what other thoughts people have. What else are the basic building blocks of modern operations when viewed through the lens of "infrastructure as code"?

For the record, I'm still pretty confident that Noah still has a place in the Doozer, riak_zab, ZooKeeper world. All three of those still rely on the persistent connection for signaling and broadcast whereas Noah is fundamentally about the disconnected and asynchronous world.

It does not follow and Wheaton's Law

2011-04-05T05:24:00.000-04:00

"I'm not a smart guy".

I say this quite a bit. I don't say it to fish for compliments or as a chance to have my ego boosted. I say it because I realize that, out of the vast corpus of computer science knowledge that exists, the part that I DO know is a blade of grass on a football field.

"I'm not a developer"

I say this a lot too. This is not meant as a slight to developers. It's meant as a compliment. There are REAL developers out there and I'm just pretending (after a fashion). I have never worked a professional gig as a developer. I've had honest discussions with people who want to pay me lots of money to be a developer. The best way I can explain it to them is that it would be unfair to you, as an employer, to hire me for a developer position because you would be unhappy with the results. In general it takes me twice as long to solve a development problem as it takes a real developer.

There are lots of factors to this; education, regular skill use and a general affinity for picking up concepts. I never graduated college and I pretty much suck at math. That's not to say I couldn't learn it but there are some things I know I'll never be as good at as someone else and that's fine by me. I'm not settling for mediocrity I just know my limitations. I'll still take a stab at it.

There are, however, some REALLY smart people out there. I used to follow a bunch of them on Twitter because they would link to or drop ideas that really made me want to go research something. I noticed an interesting trend though about some of them. They had a tendency to be dicks. Not just the occasional "Only an idiot would do X" but outright vitriol. Was it trolling? In some cases, sure, but I honestly got the impression that they actually looked down on people who didn't who use a certain technology or chose any path different than they would have chosen.

At the other extreme, you have the folks who make snide remarks or drop a non sequitur about a given technology presumably in an attempt to make the in-crowd giggle and the rest of us poor saps wonder what the hell we're doing wrong. I mean these are smart people, right? If they know something I don't about a given technology, then by god, I'd love to know what it is. I'd love to learn why they feel that way. In the end, though, all you hear is giggling in the background and wonder what the big joke was.

When the hell did we, the people who were typically on the outside of the in-crowd, turn into the people who gave us the most shit growing up? It's like a fucking geek Stockholm Syndrome thing that's gone off the deep end but instead of just sympathizing with our abuser, we're the abuser and we relish it.

I'm guilty of this behavior. I'm the first in line to criticize MongoDB, for instance. The difference? I'll actually sit down with you and tell you WHY I don't like MongoDB and why I feel it's a bad choice in many situations.

What I'm asking is that, as one of the people on the outside, educate me. As much as I think Ted Dziuba is a big troll, at least he takes the time to write it down and trys to defend his position. Ben Bleything had an awesome tweet today:

I guess what I meant is, I don't have the experience to form that opinion, I'd like to learn from you.

That's my attitude exactly. "Put up or shut up" is a bit harsh but in the broadest terms, that's what needs to happen. If you think X is superior to Z then say why. There are some of us who could benefit from it.

Sidebar on Semantics

Additionally, let's make sure we're also on the same page in terms of semantics. If we're talking about queues, clarify if you're talking about data structures versus a message queue because there's a big f'ing difference in my mind.

When I hear queue, I don't think data structure. I think of a message queue in the product sense. That's just my background. I think about things like guaranteed delivery and message durability.

Ad-Hoc Configuration, Coordination and the value of change

2011-03-08T02:01:00.000-05:00

For those who don't know, I'm currently in Boston for DevOps Days. It's been amazing so far and I've met some wonderful people. One thing that was REALLY awesome was the open space program that Patrick set up. You won't believe it works until you've tried it. It's really powerful.

In one of our open spaces, the topic of ZooKeeper came up. At this point I made a few comments, and at the additional prodding of everyone went into a discussion about ZooKeeper and Noah. I have a tendency to monopolize discussions around topics I'm REALLY passionate about so many thanks for everyone who insisted I go on ;)

Slaughter the deviants!
The most interesting part of the discussion about ZooKeeper (or at least the part I found most revealing) was that people tended to have trouble really seeing the value in it. One of the things I've really wanted to do with Noah is provide (via the wiki) some really good use cases about where it makes sense.

I was really excited to get a chance to talk with Alex Honor (one of the co-founders of DTO along with Damon Edwards) about his ideas after his really interesting blog post around ad-hoc configuration. If you haven't read it, I suggest you do so.

Something that often gets brought up and, oddly, overlooked at the same time is the where ad-hoc change fits into a properly managed environment (using a tool like puppet or chef).

At this point, many of you have gone crazy over the thought of polluting your beautifully organized environment with something so dirty as ad-hoc changes. I mean, here we've spent all this effort on describing our infrastructure as code and you want to come in and make a random, "undocumented" change? Perish the thought!

However, as with any process or philosophy, strict adherence with out understanding WHEN to deviate will only lead to frustration. Yes, there is a time to deviate and knowing when is the next level of maturity in configuration management.

So when do I deviate
Sadly, knowing when it's okay to deviate is as much a learning experience as it was getting everything properly configured in the first place. To make it even worse, that knowledge is most often specific to the environment in which you operate. The whole point of the phrase ad-hoc is that it's..well...ad-hoc. It's 1 part improvisation/.5 parts stumbling in the dark and the rest is backfilled with a corpus of experience. I don't say this to sound elitist.

So, really, when do I deviate. When/where/why and how do I deviate from this beautifully described environment? Let's go over some use cases and point out that you're probably ALREADY doing it to some degree.

Production troubleshooting
The most obvious example of acceptable deviation is troubleshooting. We pushed code, our metrics are all screwed up and we need to know what the hell just happened. Let's crank up our logging.

At this point, changing your log level, you've deviated from what your system of record (your CM tool) says you should be. Our manifests, our cookbooks, our templates all have us using a loglevel of ERROR but we just bumped up one server to DEBUG. so we could troubleshoot. That system is now a snowflake. Unless you change that log level back to ERROR, you know have one system that will, until you do a puppetrun of chef-client run is different than all the other servers of the class/role.

Would you codify that in the manifest? No. This is an exception. A (should be) short-lived exception to the rules you've defined.

Dynamic environments
Another area where you might deviate is in highly elastic environments. Let's say you've reached the holy grail of elasticity. You're growing and shrinking capacity based on some external trigger. You can't codify this. I might run 20 instances of my app server now but drop back down to 5 instances when the "event" has passed. In a highly elastic environment, are you running your convergence tool after every spin up? Not likely. In an "event" you don't want to have to take down your load balancer (and thus affect service to the existing intstances) just to add capacity. A bit of a contrived example but you get the idea.

So what's the answer?
I am by far not the smartest cookie in the tool shed but I'm opinionated so that has to count for something. These "exception" events are where I see additional tools like Zookeeper (or my pet project Noah) stepping in to handle things.

Distributed coordination, dynamically reconfigurable code, elasticity and environment-aware applications.
These are all terms I've used to describe this concept to people. Damon Edwards provided me with the last one and I really like it.

Enough jibber-jabber, hook a brother up!
So before I give you the ability to shoot yourself in the foot, you should be aware of a few things:

It's not a system of record

Your DDCS (dynamic distributed coordination service as I'll call it because I can't ever use enough buzzwords) is NOT your system of record. It can be but it shouldn't be. Existing tools provide that service very well and they do it in an idempotent manner.

Know your configuration

This is VERY important. As I said before, much of this is environment specific. The category of information you're changing in this way is more "transient" or "point-in-time". Any given atom of configuration information has a specific value associated with it. Different levels of volatility. Your JDBC connection string is probably NOT going to change that often. However, the number of application servers might be at different amounts of capacity based on some dynamic external factor.

Your environment is dynamic and so should be your response

This is where I probably get some pushback. Just as one of the goals of "devops" was to deal with, what Jesse Robbins described to day as misalignment of incentive, there's an internal struggle where some values are simply fluctuating in near real time. This is what we're trying to address.

It is not plug and play

One thing that Chef and Puppet do very well is that you can, with next to no change to your systems, predefine how something should look or behave and have those tools "make it so".

With these realtime/dynamic configuration atoms your application needs to be aware of them and react to them intelligently.

Okay seriously. Get to the point
So let's take walk through a scenario where we might implement this ad-hoc philosophy in a way that gives us the power we're seeking.

The base configuration

application server (fooapp) uses memcached, two internal services called "lookup" and "evaluate" and a data store of somekind.
"lookup" and "evaluate" are internally developed applications that provide private REST endpoints for providing a dictionary service (lookup) and a business rule parser of some kind (evaluate).
Every component's base configuration (including the data source that "lookup" and "evaluation" use) is managed, configured and controlled by puppet/chef.

In a standard world, we store the ip/port mappings for "lookup" and "evaluate" in our CM tool and tags those. When we do a puppet/chef client run, the values for those servers are populated based on the ip/port information our EXISTING "lookup"/"evaluate" servers.

This works. It's being done right now.

So where's the misalignment?
What do you do when you want to spin up another "lookup"/"evaluate" server? Well you would probably use a bootstrap of some kind and apply, via the CM tool, the changes to those values. However this now means that for this to take effect across your "fooapp" servers you need to do a manual run of your CM client. Based on the feedback I've seen across various lists, this is where the point of contention exists.

What about any untested CM changes (a new recipe for instance). I don't want to apply that but if I run my CM tool, I've now not only pulled those unintentional changes but also forced a bounce of all of my fooapp servers. So as a side product of scaling capacity to meet demand, I've now reduced my capacity at another point just to make my application aware of the new settings.

Enter Noah
This is where the making your application aware of its environment and allowing it to dynamically reconfigure itself pays off.

Looking at our base example now, let's do a bit of architectural work around this new model.

My application no longer hardcodes a base list of servers prodviding "lookup" and "evaluate" services.
My application understands the value of a given configuration atom
Instead of the hardcoded list, we convert those configuration atoms akin to something like a singleton pattern that points to a bootstrap endpoint.
FooApp provides some sot of "endpoint" where it can be notified of changes to the number/ip addresses or urls available a a given of our services. This can also be proxied via another endpoint.
The "bootstrap" location is managed by our CM tool based on some more concrete configuration - the location of the bootstrap server.

Inside our application, we're now:

Pulling a list of "lookup"/"evaluate" servers from the bootstrap url (i.e. http://noahserver/s/evaluate)
Registering a "watch" on the above "path" and providing an in-application endpoint to be notified when they change.
validating at startup if the results of the bootstrap call provide valid information (i.e. doing a quick connection test to each of the servers provided by the bootstrap lookup or a subset thereof)

If we dynamically add a new transient "lookup" server, Noah fires a notification to the provided endpoint with the details of the change. The application will receive a message saying "I have a new 'lookup' server available". It will run through some sanity checks to make sure that the new "lookup" server really does exist and works. It then appends the new server to the list of existing (permanent servers) and start taking advantage of the increase in capacity.

That's it. How you implement the "refresh" and "validation" mechanisms is entirely language specific. This also doesn't, despite my statements previously, have to apply to transient resources. The new "lookup" server could be a permanent addition to my infra. Of course this would have been captured as part of the bootstrapping process if that were the case.

Nutshell
And that's it in a nutshell. All of this is availalbe in Noah and Zookeeer right now. Noah is currently restricted to http POST endpoints but that will be expanded. Zookeeper treats watches as ephemeral. Once the event has fired, you must register that same watch. With Noah, watches are permanent.

Takeaway
I hope the above has made sense. This was just a basic introduction to some of the concepts and design goals. There are plenty of OTHER use cases for ZooKeeper alone. So the key take aways are:

Know the value of your configuration data
Know when and where to use that data
Don't supplant your existing CM tool but instead enhance it.

Links
Noah
ZooKeeper
Hadoop Book (which has some AMAZING detail around ZooKeeper, the technology and use cases

Thank You

2011-02-25T23:28:00.000-05:00

In case you hadn't heard, today Amazon went all Top Gun today and gave the world Cloud Formation. This, of course, gave rise to tweets and one-offs from pundits all over the world stating that it was the death knell of tools like Chef and Puppet.

Amazon had usurped yet another business model with the stroke of its mighty hand!

Let's ignore for a moment the fact that:

Amazon had the Chef and Puppet folks in beta
Chef and Puppet are on the block to be supported as part of CloudFormation
CloudFormation is actually nothing like Chef and Puppet and serves an entirely different purpose

I was pretty heads down at the office today (as was everyone else) so I didn't get a chance to catch up a bit until tonight. That's when I saw some of the most ignorant tweets from some supposedly smart people that I've ever seen. I ended up having to prune quite a bit from my Twitter list.

These were obviously inspired by the CloudFormation announcement and discussions around how it relates to existing CM tools. There were gems like this:

"process of orchestration, policy, governance, stacks, cross clouds, billback, etc. way too complex for some scripts"

"Scripts also wouldn't cover complexity of trying to mng a variety of clouds, all w/differing APIs & Front ends"

"You heard it here first. All you need for cloud automation, orchestration and provisioning is some Perl and you're golden! #DevFlOps"

Now maybe I'm taking these a bit out of context. Maybe I was just being a pissy bastard but these really got me riled up. Mind you not so riled up that I ran downstairs because "someone was wrong on the internet". I put my son to bed, fell asleep and when I woke up, I was still pissed off about it. I figured an hour of sleeping on it was enough justification so here I am.

Thank You

Before I get into the bitching and moaning though, I want to say "Thank you" to some people.

To Mark Burgess, Luke Kanies, Adam Jacob, Damon Edwards and any other system administrator who got so fed up with the bullshit to write the tools that we're using today, THANK YOU.

Thank you for not accepting that we had to manage systems the way we always had. Thank you for stepping outside the comfort zone and writing amazing code. Thank you for taking 20 minutes to actually think about it when you we're only given 10 minutes to get it done. Thank you.

To Patrick Debois, John Allspaw, Andrew Clay Shafer and everyone who has promoted the idea of what we call "devops" today, THANK YOU.

Thank you for pushing the idea into the mainstream with a phrase that so accurately captures what is trying to be accomplished. Thank you for being innovative and being open and sharing about it.

To everyone else who's blog posts, newsgroup postings, tweets, emails, books, irc discussions that I've had the extreme pleasure of learning from over these past 17 years in this business, THANK YOU.

Thank you for sharing. Thank you for saying it even if you thought no one was reading or listening. Thank you for challenging me to learn more and inspiring me to grow as a person and as, what I'll always be at heart, a system administrator.

To everyone above and those who I didn't mention, thank you. I thank you because it's ideas like "opensource" and "devops" and "configuration management" that free us up as individuals to think and achieve more as individuals personally and professionally. It frees us up to spend time with our families instead of answering a page at 2AM troubleshooting a stupid issue that should have never happened in the first place.

These things are more valuable than gold.

And to the haters...

Seriously.

To the vendors who write stupid applications that require me to have X11 installed on a freaking server against ALL best practices forcing me to click through a goddamn powerpoint to install your "enterprise" software, FU.

I don't need your shit and I'm luckily at a point in my career where I don't have to put up with it anymore.

To the virtualization companies who require me to have a goddamn Windows VM to manage my Linux servers because, after how many f'ing years?, you can't write a Linux port even though your product IS BASED ON LINUX? FU.

Don't worry. I can Xen and KVM like mofo. I can go to Amazon or GoGrid or Rackspace or any other provider if I don't need to host it in house. And guess what? I can do it all from the same platform I'm deploying without jumping through any hoops.

To the networking vendors who give me a choice between telnet or some overpriced java gui to do configuration of your gear, FU.

"Oh sorry about the downtime. Because we have to drop and recreate rule sets just to add a new rule, we used copy/paste from Wordpad into HyperTerminal durdurdur".

To the pundits who think that "devops" is just a bunch of perl scripts that can't "cover the complexity of blah blah blah"...I think you know the drill by now.

Really? A bunch of scripts can't cover the complexity of the various cloud providers? Interesting. I guess fog or jclouds or libcloud are just toys then.

Oh wait, what's this? You mean I can use the same commands in my CM tool regardless of where my systems are hosted? I mean Chef's command-line tool uses Fog. Any provider Fog supports, Chef will support.

But really I feel for you all. I do. You're in a losing battle. Here's the thing. People like me. People like those I mentioned above. The up and coming decision makers? We won't settle for your shitty mediocrity anymore. We won't be beholden to doing it your way. When we pick a vendor or a product or a provider, we're going to go with the ones that provide us the flexibility to manage our infrastructure in the way that's best for our company. Not for you.

We've tasted what's it like to do things the "right way" and we won't take anything less.

Follow up to "No Operations Team Left Behind"

2011-01-14T12:42:00.000-05:00

Jim Bird over at the swreflections blog, recently posted an article entitled "What I like (and don't like) about DevOps". I've attempted to post a comment but something about my comment is making Blogger go batshit so I'm posting it here instead along with some additional notes. Jim, for the record I don't think it's anything on the Blogger side. My comment is triggering an HTTP post too large error.

Here's my original comment:

As the author of one of your links, I should probably qualify a few things that weren't originally clear. I don't think that DevOps and ITIL are mutually exclusive and I don't think that anything about DevOps inherently subverts any existing policy. The point of my original post was that the enthusiasm that so many of us have can cause a negative reaction. I've often told people that you can get to the point where you can do things like continuous deployment without actually "flipping the switch". I clarified some of this in a presentation I made to the local Atlanta devops user group:

http://devops-culture-hurdles.heroku.com/

One thing that's not clear in the slides regarding "boogeymen" is that very little of the regulation from things like HIPPA and SOX impose specific technical requirements. Much of the policy is around auditability and accountability. The problem is that companies use a checklist approach to addressing those regulations because it's most cost-effective. If,for instance, the requirement is that all user access and actions are logged why is it not acceptable to simply eliminate that user access altogether and use an automated tool instead?

Auditor: "Show me who logged on to the server and what they did"

Me: "I can do you one better. No one logs onto the servers. Here's an exact list of every single configuration change applied to the server and when."

In fact, Tools like puppet, chef, mcollective, run-deck and the like actually encourage MORE security, auditability and accountability. By approaching your infrastructure as code, using configuration management tools and automation you can eliminate most if not all of the cases where, for instance, a person needs to physically log in to a server. You get disaster recovery built in because you've now codified in "code" how to define your infrastructure and you can "compile" that infrastructure into a finished product. You attack the root cause and not just bandaid it.

I think companies like WealthFront (originally Kaching) are a good example of what's possible in a regulated industry. It will be interesting to see how Facebook deals with the additional regulation should they ever go public.

Sadly my original post has been used as "See? DevOps isn't for REAL enterprises" fodder. That was not my intention. The intention was simply this:

Do not let the "cool" factor of DevOps cloud the practical factor of DevOps.

Yes, continuous deployment and fully automated environments are freaking awesome and they are truly laudable goals but they aren't the only reason to adopt these practices. Using configuration management is a no-brainer. Automated testing is a no-brainer. Having teams work more closely together SHOULD be a no-brainer. You can implement 100% of the capabilities that allow you to do those things and never actually do them. If you do flip that switch, don't belittle another person who can't flip that switch for whatever reason.

THAT was the point of my original post.

Chef and Encrypted Data Bags - Revisted

2011-01-05T06:04:00.000-05:00

In my previous post here I described the logic behind wanting to store data in an encrypted form in our Chef data bags. I also described some general encryption techniques and gotchas for making that happen.

I've since done quite a bit of work in that regard and implemented this at our company. I wanted to go over a bit of detail about how to use my solution. Fair warning, this is a long post. Lot's of scrolling.

A little recap

As I mentioned in my previous post, the only reliable way to do the encryption of data bag items in an automated fashion is to handle key management yourself outside of Chef. I mentioned two techniques:

storing the decryption key on the server in a flat file
calling a remote resource to grab the key

Essentially the biggest problem of this issue is key management and, in an optimal world, how to automate it reliably. For this demonstration, I've gone with storing a flat text file on the server. As I also said in my previous post, this assumes you tightly control access to that server. We're going with the original assumption that if a malicious person gets on your box, you're screwed no matter what.

Creating the key file

I used the knife command to handle my key creation for now:

knife ssh '*:*' interactive
echo "somedecryptionstringblahblahblah" > /tmp/.chef_decrypt.key
chmod 0640 /tmp/.chef_decrypt.key

Setting up the databags and the rake tasks

One of the previous things I mentioned is knowing when and what to encrypt. Be sensible and keep it simple. We don't want to throw out the baby with the bath water. The Chef platform has lots of neat search capabilities that we'd like to keep. In this vein, I've created a fairly opinionated method for storing the encrypted data bag items.

We're going to want to create a new databag called "passwords". The format of the data bag is VERY simple:

We have an "id" that we want to use and the plaintext value that we want to encrypt.

Rake tasks

In my local chef-repo, I've created a 'tasks' folder. In that folder, I've added the following file:

As you can see, this requires a rubygem called encrypted_strings. I've done a cursory glance over the code and I can't see anything immediately unsafe about it. It only provides an abstraction to the native OpenSSL support in Ruby with an additional String helper. However I'm not a cryptographer by any stretch so you should do your own due diligence.

At the end of your existing Rakefile, add the following:

load File.join(TOPDIR, 'tasks','encrypt_databag_item.rake')

If you now run rake -T you should see the new task listed:

rake encrypt_databag[databag_item]  # Encrypt a databag item in the passwords databag

If you didn't already create a sample data bag and item, do so now:

mkdir data_bags/passwords/
echo '{"id":"supersecretpassword","data":"mysupersecretpassword"}' > data_bags/passwords/supersecretpassword.json

Now we run the rake task:

rake encrypt_databag[supersecretpassword]

Found item: supersecretpassword. Encrypting
Encrypted data is <some ugly string>
Uploading to Chef server
INFO: Updated data_bag_item[supersecretpassword_crypted.json]

You can test that the data was uploaded successfully:

knife data bag show passwords supersecretpassword

{
"data": "<some really ugly string>",
"id": "supersecretpassword"
}

Additionally, you should have in your 'data_bags/passwords' directory a new file called 'supersecretpassword_crypted.json'. The reason for keeping both files around is for key management. Should you need to change your passphrase/key, you'll need the original file around to reencrypt with the new key. You can decided to remove the unencrypted file if you want as long as you have a way of recreating it.

Using the encrypted data

So now that we have a data bag item uploaded that we need to use, how do we get it on the client?
That will require two cookbooks:

databag_decrypt
A cookbook which needs the decrypted data. example

The general idea is that, in any cookbook you need decrypted data, you essentially do three things:

include the decryption recipe

include_recipe "databag_decrypt::default"

assign the crypted data to a value via databag search

password = search(:passwords, "id:supersecretpassword").first

assign the decrypted data to a value for use in the rest of the recipe
```
decrypted_password = item_decrypt(password[:data])
```

From there, it's no different that any other recipe. Here's an example of how I use it to securely store Amazon S3 credentials as databag items:

include_recipe "databag_decrypt::default"
s3_access_key = item_decrypt(search(:passwords, "id:s3_access_key").first[:data])
s3_secret_key = item_decrypt(search(:passwords, "id:s3_secret_key").first[:data])
s3_file erlang_tar_gz do
  bucket "our-packages"
  object_name erlang_file_name
  aws_access_key_id s3_access_key
  aws_secret_access_key s3_secret_key
  checksum erl_checksum
end

Changing the key

Should you need to change the key, you'll need to jump through a few hoops:

Update the passphrase on each client. Ease depends on your method of key distribution
Update the passphrase in the rake task
Reencypt all your data bag items.

The last one can be a pain in the ass. Since Chef currently doesn't support multiple items in a data bag json file, I created a small helper script in my chef-repo called 'split-em.rb'.
I store all of my data bag items in large json files and use split-em.rb to break them into individual files. Those file I upload with knife:

bin/split-em.rb -f data_bags/passwords/passwords.json -d passwords -o

Parsing data for svnpass into file data_bags/passwords/svnpass.json
Parsing data for s3_access_key into file data_bags/passwords/s3_access_key.json
Parsing data for s3_secret_key into file data_bags/passwords/s3_secret_key.json
#Run the following command to load the split bags into the passwords in chef
for i in svnpass s3_access_key s3_secret_key; do knife data bag from file passwords $i.json; done

You could then run that through the rake task to reupload the encrypted data:

for i in svnpass s3_access_key s3_secret_key; do rake encrypt_databag[$i]; done

Limitations/Gotchas/Additional Tips

Take note of the following, please.

Key management

The current method of key management is somewhat cumbersome. Ideally, the passphrase should be moved outside of the rake task. Additionally, the rekey process should be made a distinct rake task. I imagine a workflow similar to this:

rake accepts a path to the encryption key
additional rake task to change the encryption key in the form of oldpassfile/newpassfile.
Existing data is decrypted using oldpassfile, reencrypted using new passfile and sent back to the chef server.

Optimally, the rake task would understand the same attributes that the decryption cookbook does so it can handle key managment on the client for you. I'd also like to make the cipher selection configurable as well an integrate it into the above steps.

Duplicate work

Seth Falcon at Opscode is already in the process of adding official support for encrypted data bags to Chef. His method involves converting the entire databag sans "id" to YAML and encrypting it. I wholeheartedly support that effort but that would obviously require a universal upgrade to Chef as well. The purpose of my cookbook and tasks is to work with the existing version.

AWS IAM

If you're an Amazon EC2 user, you should start using IAM NOW. Stop putting your master credentials in to recipes and limit your risk. I've created a 'chef' user who I give limited access to certain AWS operations. You can see the policy file here. It gives the chef user read-only access to 'my_bucket' and 'my_other_bucket'.
If you wanted to get REALLY sneaky, you could use fake two-factor authentication to store your key in S3:

Encrypt data bag items with "crediential B" password except for one item "s3_credentials"
s3_credentials (crendential A) is encrypted with a passphrase and managed similar to this article
Use transient credentials to access S3 and grab a passphrase file (credential B)
Decrypt data with secondary credentials

You would have to heavily modify the cookbook to do this. I think the current implementation is fine.

File-based passphrases

I'm not a big fan of the file-based passphrase method. While we agreed that you should consider yourself screwed if someone gets on the box, that still leaves poorly coded applications running as an attack vector. Imagine you have an application that must run as root. Now it can read the passphrase. Should that application become remotely exploitable, the passphrase file is vulnerable. I'm leaning to the method of a private server that allows RESTful access to grab the key. I've already added support in the cookbook for a passphrase type of 'url'.

Wrapup

I think that covers anything. I'd love some feedback on what people think. We've already implemented this in a limited scope for using IAM credentials in our cookbooks. I can easily revoke those should they get compromised without having to generate all new master keys.

Chef and encrypted data bags.

2010-12-15T00:47:00.000-05:00

As part of rolling out Chef at the new gig, we had a choice - stand up our own Chef server and maintain it or use the Opscode platform. From a cost perspective, the 50 node platform cost was pretty much break even with standing up another EC2 instance of our own. The upshot was that I didn't have to maintain it.

However, part of due diligence was making sure everything was covered from a security perspective. We use quite a few hosted/SaaS tools but this one had the biggest possible security risk. The biggest concern is dealing with sensitive data such as database passwords and AWS credentials. The Opscode platform as a whole is secure. It makes heavy use of SSL not only for transport layer encryption but also for authentication and authorization. That wasn't a concern. What was a concern was what should happen if a copy of our CouchDB database fell into the wrong hands or a "site reliability engineer" situation happened. That's where the concept of "encrypted data bags" came from for me.

Atlanta Chef Hack Day

I had the awesome opportunity to stop by the Atlanta Chef Hack day this past weekend. I couldn't stay long and came in fairly late in the afternoon. However I happened to come in right at the time that @botchagalupe (John Willis) and @schisamo (Seth Chisamore) brought up encrypted data bags. Of course, Willis proceeded to turn around and put me on the spot. After explaining the above use case, we all threw out some ideas but I think everyone came to the conclusion that it's a tough nut to crack with a shitload of gotchas.

Before I left, I got a chance to talk with @sfalcon (Seth Falcon) about his ideas. While he totally understood the use cases and mentioned that other people had asked about it as well, he had a few ideas but nothing that stood out as the best way.

So what are the options? I'm going to list a few here but I wanted to discuss a little bit about the security domain we're dealing with and what inherent holes exist.

Reality Checks

Nothing is totally secure.

Deal with it. Even though it's a remote chance in hell, your keys and/or data are going to be decrypted somewhere at some point in time. The type of information we need to read, unfortunately, can't use a one-way encryption algo like MD5 or SHA because we NEED to know what the data actually is. I need that MySQL password to provide to my application server to talk to the database. That means it has to be decrypted and during that process and during usage of that data, it's going to exist in a possible place that it can be snagged.

You don't need to encrypt everything

You need to understand what exactly needs to be encrypted and why. Yes, there's the "200k winter coats to troops" scenario and every bit of information you expose provides additional material for an attack vector but really think about what you need to encrypt. Application database account usernames? Probably not. The passwords for those accounts? Yes. Consider the "value" of the data you're considering encrypting.

Don't forget the "human" factor

So you've got this amazing library worked out, added it to your cookbooks and you're only encrypting what you need to really encrypt. Then some idiot puts the decryption key on the wiki or the master password is 5 alphabetical characters. As we often said when I was a kid, "Smooth move, exlax"

There might be another way

There might be another way to approach the issue. Make sure you've looked at all the options.

Our Use Case

So understanding that, we can narrow down our focus a bit. Let's use the use case of our applications database password because it's a simple enough case. It's a single string.

Now in a perfect world, Opscode would encrypt each CouchDB database with customer specific credentials (like say an organizational level client cert) and discards the credentials once you've downloaded them.

That's our first gotcha - What happens when the customer loses the key? All that data is now lost to the world.

But let's assume you were smart and kept a backup copy of the key in a secure location. There's another gotcha inherent in the platform itself - Chef Solr. If that entire database is encrypted, unless Opscode HAS the key, they can't index the data with Solr and all those handy searches you're using in your recipes to pull in all your users is gone. Now you'll have to manage the map/reduce views yourself and deal with the performance impact where you don't have one of those views in place.

So that option is out. The Chef server has to be able to see the data to actually work.

What about a master key? That has several problems.

You have to store the key somewhere accessible to the client (i.e. the client chef.rb or in an external file that your recipes can read to decrypt those data bag items).

How do you distribute the master key to the clients?
How do you revoke the master key to the clients and how does that affect future runs? See the previous line - how do you then distribute the updated key?

I'm sure someone just said "I'll put it in a data bag" and then promptly smacked themselves in the head. Chicken - meet Egg. Or is it the other way around?

You could have the Chef client ASK you for the key (remember apache SSL startups where the startup script required a password? Yeah, that sucked.

Going the Master Key Route

So let's assume that we want to go this route and use a master key. We know we can't store in with Opscode because that defeats the purpose. We need a way to distribute the master key to the clients so they can decrypt the data so how do we do it?

If you're using Amazon, you might say "I'll store it in S3 or on an EBS volume". That's great! Where do you store the AWS credentials? "In a data ba...oh wait. I've seen this movie before, haven't I?"

So we've come to the conclusion that we must store the master key somewhere ourselves locally available to the client. Depending on your platforming, you have a few options:

Make it part of the base AMI
Make it part of your kickstart script
Make it part of your vmware image

All of those are acceptable but they don't deal with updating/revocation. Creating new AMIs is a pain in the ass and you have to update all your scripts with new AMI ids when you do that. Golden images are never golden. Do you really want to rekick a box just to update the key?

Now we realize we have to make it dynamic. You could make it a part of a startup script in the AMI, first boot of the image or the like. Essentially, "when you startup, go here and grab this key". Of course now you've got to maintain a server to distribute the information and you probably want two of them just to be safe, right? Now we're spreading our key around again.

This is starting to look like an antipattern.

But let's just say we got ALL of that worked out. We have a simple easy way for clients to get and maintain the key. It works and your data is stored "securely" and you feel comfortable with it.

Then your master key gets compromised. No problem, you think. I'll just use my handy update mechanism to update the keys on all the clients and...shit...now I've got to re-encrypt EVERYTHING and re-upload my data bags. Where the hell is the plaintext of those passwords again? This is getting complicated, no?

So what's the answer? Is there one? Obviously, if you were that hypersensitive to the security implications you'd just run your own server anyway. You still have the human factor and backups can still be stolen but that's an issue outside of Chef as a tool. You just move the security up the stack a bit. You've got to secure the Chef server itself. But can you still use the Opscode platform? I think so. With careful deliberation and structure, you can reach a happy point that allows you to still automate your infrastructure with Chef (or some other tool) and host the data off-site.

Some options

Certmaster

Certmaster spun out of the Func project. It's essentially an SSL certificate server at the base. It's another thing you have to manage but it can handle all the revocation and distribution issues.

Riak

This is one idea I came up with tonight. The idea is that you run a very small Riak instance on all the nodes that require the ability to decrypt the data. Every node is a part of the same cluster and this can all be easily managed with Chef. It will probably have a single bucket containing the master key. You get the fault tolerance built in and you can pull the keys as part of your recipe using basic Chef resources. Resource utilization on the box should be VERY low for the erlang processes. You'll have a bit more network chatter as the intra-cluster gossip goes on though. Revocation is still an issue but that's VERY easily managed since it's a simple HTTP put to update. And while the data is easily accessible to anyone who can get access to the box, you should consider yourself "proper f'cked" if that happens anyway.

But you still have the issue of re-encrypting the databags should that need to happen. My best suggestion is to store the encrypted values in a single data bag and add a rake task that does the encryption/revocation for you. Then you minimize the impact of something that simply should not need to happen that often.

Another option is to still use Riak but store the credentials themselves (as opposed to a decryption key) and pull them in when the client runs. The concern I have there is how that affects idempotence and would it cause the recipe to be run every single time just because it can't checksum properly? You probably get around this with a file on the filesystem telling Chef to skip the update using "not_if".

Wrap Up

As you can see, there's no silver bullet here. Right now I have two needs, storing credentials for S3/EBS access and storing database passwords. That's it. We don't use passwords for user accounts at all. You can't even use password authentication with SSH on our servers. If I don't have your pubkey in the users data bag, you can't log in.

The AWS credentials are slowly becoming less of an issue. With the Identity Access beta product, I can create limited use keys that can only do certain things and grant them access to specific AWS products. I can make it a part of node creation to generate that access programatically. That means I still have the database credentials issue though. For that, I'm thinking that the startup script for an appserver, for instance, will just have to pull the credentials from Riak (or whatever central location you choose) and update a JNDI string. It spreads your configuration data out a bit but these things shouldn't need to change to often and with proper documented process you know exactly how to update it.

One thing that this whole thing causes is that it begins to break down the ability to FULLY automate everything. I don't like running the knife command to do things. I want to be able to programatically run the same thing that Knife does from my own scripts. I suppose I could simply popen and run the knife commands but shelling out always feels like an anti-pattern to me.

I'd love some feedback on how other people are addressing the same issues!

Automating EBS Snapshot validation with @fog - Part 2

2010-12-02T03:13:00.001-05:00

This is part 2 in a series of posts I'm doing - You can read part 1 here

Getting started

I'm not going to go into too much detail on how to get started with Fog. There's plenty of documentation on the github repo (protip: read the test cases) and Wesley a.k.a @geemus has done some awesome screencasts. I'm going to assume at this point that you've at least got Fog installed, have an AWS account set up and have Fog talking to it. The best way to verify is to create your .fog yaml file, start the fog command line tool and start looking at some of the collections available to you.

For the purpose of this series of posts, I've actually created a small script that you can use to spin up two ec2 instances (m1.small) running CentOS 5.5, create four (4) 5GB EBS volumes and attach them to the first instance. In addition to the fog gem, I also have awesome_print installed and use it in place of prettyprint. This is, of course, optional but you should be aware.

WARNING: The stuff I'm about to show you will cost you money. I tried to stick to minimal resource usage but please be aware you need to clean up after yourself. If, at any time, you feel like you can't follow along with the code or something isn't working - terminate your instances/volumes/resources using the control panel or command-line tools. PLEASE DO NOT JUST SIMPLY RUN THESE SCRIPTS WITHOUT UNDERSTANDING THEM.

The setup script

The full setup script is available as gist on github - https://gist.github.com/724912#file_fog_ebs_demo_setup.rb

Things to note:

Change the key_name to a valid key pair you have registered with EC2
There's a stopping point halfway down after the EBS volumes are created. You should actually stop there and read the comments.
You can run everything inside of an irb session if you like.

The first part of the setup script does some basic work for you - it reads in your fog configuration file (~/.fog) and creates an object you can work with (AWS). As I mentioned earlier, we're creating two servers - hdb and tdb. HDB is the master server - say your production MySQL database. TDB is the box which will be running as the validation of the snapshots.

In the Fog world, there are two big concepts - models and collections. Regardless of cloud provider, there are typically at least two models available - Compute and Storage. Collections are data objects under a given model. For instance in the AWS world, you might have under the Compute model - servers, volumes, snapshots or addresses. One thing that's nice about Fog is that, once you establish your connection to your given cloud, most of your interactions are the same across cloud providers. In the example above, I've created a connection with Amazon using my credentials and have used that Compute connection to create two new servers - hdb and tdb. Notice the options I pass in when I instantiate those servers.

image_id
key_name

If I wanted to make these boxes bigger, I might also pass in 'flavor_id'. If you're running the above code in an irb session, you might see something like the following when you instantiate those servers: Not all of the fields may be available depending on how long it takes Amazon to spin up the instance. The above shot is after the instance was up and running. For instance, when you first created 'tdb', you'll probably see "state" as pending for quite some time. Fog has a nice helper method for all models call 'wait_for'. In my case I could do:

tdb.wait_for { print "."; ready?}

And it would print dots across the screen until the instance is ready for me to log in. At the end, it will tell you the amount of time you spent waiting. Very handy. You have direct access to all of the attributes above via the instance 'tdb' or 'hdb'. You can use 'tdb.dns_name' to get the dns name for use in other parts of your script for example. In my case, after the server 'hdb' is up and running, I now want to create the four 5GB EBS volumes and attach them to the instance:

I've provided four device names (sdi through sdl) and I'm using the "volumes" collection to create them (AWS.volumes.new). As I mentioned earlier, all of the attributes for 'hdb' and 'tdb' are accessible by name. In this case, I have to create my volumes in the same availability zone as the hdb instance. Since I didn't specify where to create it when I started it, Amazon has graciously chosen 'us-east-1d' for me. As you can see, I can easily access that as 'hdb.availability_zone' and pass it to the volume creation section. I've also specified that the volume should be 5GB in size.

At the point where I've created the volume with '.new' it hasn't actually been created. I want to bind it to a server first so I simply set the volume.server attribute equal to my server object. Then I 'save' it. If I were to log into my running instance, I'd probably see something like this in the 'dmesg' output now:

sdj: unknown partition table

sdk: unknown partition table

sdl: unknown partition table

sdi: unknown partition table

As you can see from the comments in the full file, you should stop at this point and setup the volumes on your instance. In my case, I used mdadm and created a RAID0 array using those four volumes. I then formatted them, made a directory and mounted the md0 device to that directory. If you look, you should now have an additional 20GB of free space mounted on /data. Here I might make this the data directory for mysql (which is the case in our production environment). Let's just pretend you've done all that. I simulated it with a few text files and a quick 1GB dd. We'll consider that the point-in-time that we want to snapshot from. Since there's no actual constant data stream going to the volumes, I can assume for this exercise that we've just locked mysql, flushed everything and frozen the XFS filesystem. Let's make our snapshots. In this case I'm going to be using Fog to do the snapshots but in our real environment we're using the ec2-consistent-snapshot script from Aelastic. First let's take a look at the state of the hdb object:

Notice that the 'block_device_mapping' attribute now consist of an array of hashes. Each hash is a subset of the data about the volume attached to it. If you aren't seeing this, you might have to run 'hdb.reload' to refresh the state of the object. To create our snapshots, we're going to iterate over the block_device_mapping attribute and use the 'snapshots' collection to make those snapshots:

One thing you'll notice is that I'm being fairly explicity here. I could shorthand and chain many of these method calls but for clarity, I'm not.

And now we have 4 snapshots available to us. The process is fairly instant but sometimes it can lag. As always, you should check the status via the .state attribute of an object to verify that it's ready for the next step. Here's a shot of our snapshots right now:

That's the end of Part 2. In the next part, we'll have a full fledged script that does the work of making the snapshots usable on the 'tdb' instance.

Automating EBS Snapshot validation with @fog - Part 1

2010-12-02T01:35:00.001-05:00

Background

One thing that's very exciting about the new company is that I'm getting to use quite a bit of Ruby and also the fact that we're entirely hosted on Amazon Web Services. We currently leverage EBS, ELB, EC2 S3 and CloudFront for our environment. The last time I used AWS in a professional setting, they didn't even have Elastic IPs much less EBS with snapshots and all the nice stuff that makes it viable for a production environment. I did, however, manage to keep abreast of changes using my own personal AWS account.

Fog

Of course the combination of Ruby and AWS really means one thing - Fog. And lot's of it.

When EngineYard announced the sponsorship of the project, I dove headlong into the code base and spent what time I could trying to contribute code back. The half-assed GoGrid code in there right now? Sadly, some of it is mine. Time is hard to come by these days. Regardless, I'm no stranger to Fog and when I had to dive into the environment and start getting it documented and automated, Fog was the first tool I pulled out and when the challenge of verifying our EBS snapshots (of which we're currently at a little over 700), I had no choice but to automate it.

Environment

A little bit about the environment:

- A total of 9 EBS volumes are snapshotted each day
- 8 of the EBS volumes are actually raid0 mysql data stores across two DB servers (so 4 disks on one/4 disks on another)
- The remaining EBS volume is a single mysql data volume
- Filesystem is XFS and backups are done using the Aleastic ec2-consistent-snapshot script (which currently doesn't support tags)

The end result of this is to establish a rolling set of validated snapshots. 7 daily, 3 weekly, 2 monthly. Fun!

Mapping It Out

Here was the attack plan I came up with:

- Identify snapshots and groupings where appropriate (raid0, remember?)
- create volumes from snapshots
- create an m1.xlarge EC2 instance to test the snapshots
- attach volume groups to the test instance
- assemble the array on the test instance
- start MySQL using the snapshotted data directory
- run some validation queries using some timestamp columns in our schema
- stop MySQL, unmount volume, stop the array
- detach and destroy the volumes from the test instance
- tag the snapshots as "verified"
- roll off any old snapshots based on retention policy
- automate all of the above!

I've got lots of code samples and screenshots so I'm breaking this up into multiple posts. Hopefully part 2 will be up some time tomorrow

Fix it or Kick It and the ten minute maxim

2010-11-09T01:12:00.002-05:00

One of the things I brought up in my presentation to the Atlanta DevOps group was the concept of "Payment". One of the arguments that people like to trot out when you suggest an operational shift is that "We can't afford to change right now". My argument is that you CAN'T afford to change. It's going to cost you more in the long run. The problem is that in many situations, the cost is detached from the original event.

Take testing. Let's assume you don't make unit testing an enforced part of your development cycle. There are tons of reasons people do this but much of it revolves around time. We don't have time to write tests. We don't have time to wait for tests to run. We've heard them all. Sure you get lucky. Maybe things go out the door with no discernible bugs. But what happens 3 weeks down the road when the same bug that you solved 6 weeks ago crops up again? It's hard to measure the cost when it's so far removed from the origination.

Configuration management is the same way. I'm not going to lie. Configuration management is a pain in the ass especially if you didn't make it a core concept from inception. You have to think about your infrastructure a bit. You'll have to duplicate work initially (i.e. templating config files). It's not easy but it pays off in the long run. However as with so many things, the cost is detached from the original purchase.

Fix it?

Walk with me into my imagination. A scary place where a server has started to misbehave. What's your initial thought? What's the first thing you do? You've seen this movie and done this interview:

log on to the box
perform troubleshooting
think
perform troubleshooting
call vendor support (if it's an option)
update trouble ticket system
wait
troubleshoot
run vendor diag tools

What's the cost of all that work? What's the cost of that downtime? Let's be generous. Let's assume this is a physical server and you paid for 24x7x4 hardware support and a big old RHEL subscription. How much time would you spend on each task? What's the turn around time to getting that server back into production?

Let's say that the problem was resolved WITHOUT needing replacement hardware but came in at the four hour mark. That's three hours that the server was costing you money instead of making you money. Assuming a standard SA salary of $75k/year in Georgia, that works out to $150. That's just doing a base salary conversion not calculating all the other overhead associated with staffing an employee. What if that person consulted with someone else during that time, a coworker at the same rate, for two of those hours. $225. Not too bad, right? Still a tangible cost. Maybe one you're willing to eat.

But let's assume the end result was to wipe and reinstall. Let's say it takes another hour to get back to operational status. Woops. Forgot to make that tweek to Apache that we made a few weeks ago. Let's spend an hour troubleshooting that.

But we're just talking man power at this point. This doesn't even take into account end-user productivity, loss of customers from degraded performance or any host of issues. God forbid that someone misses something that causes problems to other parts of the environment (like not setting the clock and inserting invalid timestamps into the database or something. Forget that you shouldn't let your app server handle timestamps). Now there's cleanup. All told your people spent 5 hours to get this server back into production while you've been running in a degraded state. What does that mean when our LOB is financial services and we have an SLA and attached penalties? I'm going to go easy on you and let you off with 10k per hour of degraded performance.

Get ready to credit someone $50k or worse cut a physical check.

Kick it!

Now I'm sure everyone is thinking about things like having enough capacity to maintain your SLA even with the loss of one or two nodes but be honest. How many companies actually let you do that? Companies will cut corners. They roll the dice or worse have a misunderstanding of HA versus capacity planning.

What you should have done from the start was kick the box. By kicking the box, I mean performing the equivalent of a kickstart or jumpstart. You should, at ANY time, be able to reinstall a box with no user interaction (other than the action of kicking it) and return it to service in 10 minutes. I'll give you 15 minutes for good measure and bad cabling. My RHEL/CentOS kickstarts are done in 6 minutes on my home network and most of that time is the physical hardware power cycling. With virtualization you don't even have a discernible bootup time.

Unit testing for servers

I'll go even farther. You should be wiping at least one of your core components every two weeks. Yes. Wiping. It should be a part of your deploy process in fact. You should be absolutely sure that should you ever need to reinstall under duress that you can get that server back into service in an acceptable amount of time. Screw the yearly DR tests. I'm giving you a world where you can perform bi-monthly DR tests as a matter of standard operation. All it takes is a little bit of up front planning.

The 10 minute maxim

I have a general rule. Anything that has to be done in ten minutes can be afforded twenty minutes to think it through. Obviously, it's a general rule. The guy holding the gun might not give you twenty minutes. And twenty minutes isn't a hard number. The point is that nothing is generally so critical that it has to be SOLVED that instant. You can spend a little more time up front to do things right or you can spend a boatload of time on the backside trying to fix it.

Given the above scenario, you would think I'm being hypocritical or throwing out my own rule. I'm not. The above scenario should have never happened. This is a solved problem. You should have spent 20 minutes actually putting the config file you just changed into puppet instead of making undocumented ad-hoc changes. You should have spent an hour when bringing up the environment to stand up a CM tool instead of just installing the servers and doing everything manually. That's the 10 minute maxim. Take a little extra time now or take a lot of time later.

You decide how much you're willing to spend.

Transitions

2010-11-08T15:12:00.002-05:00

I haven't had a chance to mention this but those of you who I'm connected with on LinkedIn are aware that I'm starting with a new company on Wednesday. I'm taking a few days to get some house work done and then diving in. I don't like switching companies in general but I'm really excited about this opportunity. In addition to having almost a blank slate, I'm working with a much smaller team and a chance to contribute back to the community. It's also a chance for me to work in the Atlanta startup scene; something I've been hoping to do for a few years now.

So what about the previous company? Well they're looking to back fill my position. Please feel free to contact me if you're interested. I can put you in touch with the right people. Fair warning, it's a challenging place to work. They'll tell you the same thing. I've blogged about working at a "traditional" company before right here so you can go back and glean information from that.

Using Hudson and RVM for Ruby unit testing

2010-11-02T23:56:00.000-04:00

As with everything lately, something popped up on Twitter that prompted a blog post. In this case, @wakaleo was looking for any stories/examples for his Hudson book. I casually mentioned I could throw in some notes about how we use Hudson on the Padrino project.

Prerequisites

Here's what you'll need:

Hudson
RVM
Hudson Ruby Metrics plugin (if you want code coverage reports)

I'll leave you to get Hudson working. There are prebuilt packages for every distro under the sun. If you can't get past this step, you'll need to rethink a few things.

Setting up RVM

Once you have it installed, log in as your Hudson user and set up RVM.

RVM Protip - If there are any gems (like say Bundler) that you ALWAYS install, edit .rvm/gemsets/default.gems and .rvm/gemsets/global.gems and add them there. In my examples, I did not do that.

You'll want to go ahead and install all the VMs you plan on testing against. We use 1.8.7, 1.9.1, 1.9.2, JRuby, RBX and REE:

for i in 1.8.7 1.9.1 1.9.2 jruby ree rbx; do rvm install ${i}; done

This will take a while. When it's done, we can now dive into configuring our job in Hudson

What is the Matrix?

So you've got Hudson running and RVM all set up? Open the Hudson console and create a new job of type "Build multi-configuration project". From the job configuration screen, you'll want to set some basics - repository, scm polling and the like. The key to RVM comes under "Configuration Matrix"

The way any user-defined variables work in Hudson, whether a build parameter or matrix configuration, is that you provide a "key" and then a value for that key. The value for that key is accessible to your build steps as a sigil variable. So if your key is my_funky_keyname_here, you can reference $my_funky_keyname_here in your build steps to get that value. With a configuration matrix, each permutation of the matrix provides the value for that key in the given permutation. So if I have:

foo as one axis with 6 values (1, 2, 3 ,4 ,5 ,6) and bar with 3 values (1, 2, 3)

each combination of foo and bar will be available to my build steps as $foo and $bar. The first run will have $foo as 1 and $bar as 1. Second run will have $foo as 2 and $bar as 1. On an on until the combinations are exhausted.

This makes for some REALLY powerful testing matrices. In our case, however, we only need one axis - rubyvm

Hudson Protip - Don't get creative with your axis or parameter names. In our case, we'll be performing shell script steps. Don't call your axis "HOME" because that will just confuse things. Just don't do it.

So now we've added an axis called 'rubyvm' and provided it with values '1.8.7 1.9.1 1.9.2 jruby rbx ree'. As explained, this means that our build steps will iterate over each value of 'rubyvm' for us and repeat our build steps.

Configuring your job

Now that you've got your variables in place, you can write the steps for your job. This took me a little bit of time to work out the best flow. There were some things with how RVM operates with the shell that caught me off-guard initially (the rvm command being a function alias versus an executable). I've broken the test job into three steps:

Create my gemset, install bundler and run bundle install/bundle check
Run my unit tests
Destroy my gemset

In addition to taking advantage of the variable provided by the configuration matrix, we're also going to take advantage of some variables exposed by Hudson in a given job run - $BUILD_NUMBER. Using these two bits of information, we can build a gemset name for RVM that is unique to that run and that ruby vm.

Step 1:

#!/bin/bash -l

rvm use $rubyvm@padrino-$rubyvm-$BUILD_NUMBER --create

gem install bundler

bundle install

bundle check

This uses the --create option of RVM to create our gemset. If our build number is 99 and our ruby vm is ree, we're creating a gemset called padrino-ree-97 for ree. Pretty straightforward.

Next we install bundler and then run the basic bundler tasks. All operations are performed in the workspace for your hudson project. This is typically the root directory of your SCM repository. If the root of your repo doesn't contain your Gemfile and Rakefile, you'll probably want to make your first step a 'cd' to that directory.

The reason for using a full shebang line is to make sure that RVM instantiates properly.

Step 2:

#!/bin/bash -l

rvm use $rubyvm@padrino-$rubyvm-$BUILD_NUMBER

rake test

Each build step is a distinct shell session. For that reason we need to "use" the previously created gemset. Then we run our rake tasks.

Step 3:

#!/bin/bash -l

rvm use $rubyvm@global

rvm --force gemset delete padrino-$rubyvm-$BUILD_NUMBER

This is the "cleanup" step. This cleans up our temporary gemsets that we created for the test run. My understanding was the each step was "independent". Should the middle step fail, the final step would still be executed. This doesn't appear to be the case anymore. For this reason, you'll probably want to occasionally go in and clean up gemsets from failed builds. If your build passes, the gemset will clean itself up. There's probably justification for some sort of "cleanup" job here but I haven't gotten around to trying to pass variables as artifacts to other build steps.

Now you can run the job and watch as Hudson gleefully executes your test cases against each ruby vm. How many of those run concurrently is dependent on how many workers you have configured globally in Hudson.

Unit Testing Protip - One thing you'll find out early on is how concurrent your unit tests REALLY are. In the case of Padrino, ALL of our unit tests were using a hardcoded path (/tmp/sample_project) for testing. My first major step once I got added to the project was to refactor ALL of our tests to make that dynamic so that we could run more than one permutation at a time. You can see an example of how I did that here. Essentially I created an instance variable for our temp directory using UUID.new.generate. It was the quickest way to resolve the problem. If your tests aren't capable of running in parallel, that's one way to address it.

One thing to be aware of: if you have intensive unit tests and your hudson server isn't very powerful, you simply may not have the capacity to run multiple tests at the same time. I had to spin up some worker VMs on other machines around the house to serve as Hudson slave nodes. Our unit tests were actually taking LONGER when we tried to run them in parallel because of the strain of compiling native extension gems and actually running the tests.

Optional profit! step

Code coverage is important. However it makes NO sense to run code coverage tasks on EVERY VM permutation. You only need to run it once (unless you have some VM dependent code in your application). What I've done is take advantage of "Post build actions" to kick off a second job I've defined. This job does nothing but runs our code coverage rake tasks. Steps 1 and 3 are the same as above without the rubyvm variable. Step 2 is different:

#!/bin/bash -l

rvm use 1.8.7@padrino-rcov-$rubyvm-$BUILD_NUMBER

bundle exec rake hudson:coverage:clean

bundle exec rake hudson:coverage:unit

We've broken the coverage tests into a unique rake task so they don't impact normal testing. This creates a code coverage report that's visible in Hudson under that project's page. Currently we don't run the coverage report job unless the primary job finishes.

Wrap up

That's pretty much it in a nutshell. I'm looking to move Hudson to a more powerful VM here at the house as soon as the hardware comes in. I should be able to then run all the tests across all VMs at one time. Screenshots for each of the steps described in this post are available here

Designed for Developers - Why people keep asking you to use Github

2010-10-28T05:25:00.000-04:00

I'll be the first to admit that I'm a Github fanboy. The shocker is that my love of Github has nothing to do with the DVCS underneath. While Git plays a major part of what makes github so great, the bigger reason github is so successful is this:

Github is designed for developers

What do I mean by that? Let's compare a series of screenshots from various code hosting sites:

Code Hosting Solutions comparison

I want you to take a look at the screenshots very carefully especially the "project" pages. What's the one thing you notice about Github compared to the others (excluding BitBucket). What's the focus of the project?

It's all about the code

You'll see quite clearly that with all the sites except for BitBucket, the focus of the project is the code itself. Not only is the focus of the project the code but everything about the code is about the community. I can "watch" a developer or project. I can easily see from the first page how to download the codebase. However the biggest part of what makes Github a success is one button:

Fork

From the start of a project page, not only can I easily browse the code and am provided with the information I need to checkout the code but I'm invited with a single button to become a contributor to that project. Immediately, I'm a potential contributor to that project. If I change something and push the code back to my fork, I can push one button and send a message to the project maintainers asking them to merge the changes back in. As a project maintainer, I have an easy way to evaluate the impact of the change and communicate with the requester and other team members about said change. At the bottom of the pull request page, I'm provided the information on how to easily merge those changes into my main tree.

Designed for Developers

I've been on a bit of a tear lately about usability in developer-targeted products. The latest target of my ire has been Atlassian. Let me clarify that I think Atlassian makes some wonderful products. Confluence is one of the best wikis out there. JIRA is a great issue tracking system for Developers.

However, Atlassian has some "duds" in my opinion. The biggest thorn in my side these days is Bamboo. Bamboo is Atlassian's Continuous Integration server. Like most Atlassian products, its primary target is Java developers. Everything about Bamboo is designed around the Java development toolchain - Maven, Ant and the like. But I don't have a problem with that. What I have a problem with is the over-complication. I grabbed the latest beta of Bamboo at the recommendation of one of the Bamboo developers who heard my rant on Twitter one day. He asked for some feed back and I provided it in a very detailed email. I'm happy to say that the new interface for adding build plans in Bamboo is much simpler than previous versions. I can't do screenshots of our company Bamboo install but previous versions had a VERY complicated multitab build plan configuration.

One point I mentioned in my email is that Bamboo felt like it lacked a focus. Jira was very clearly about Issues. That was the "unit of work". Confluence was very clearly about being a wiki. That was its "unit of work". Bamboo didn't have a singular focus. It was a CI server but what was the unit of work? A build plan? Test results? Fisheye integration? It wasn't clear.

Compare that with Hudson which had a very clear focus. The strength in Hudson is that it performs tasks. Those tasks are typically centered around CI but they don't have to be. In Hudson I can define a job that does nothing more than list directories. I don't even need to back it with a VCS. Bamboo, sadly, in the beta version still hasn't gotten this part right. I can't define a build plan without having a repository somewhere. It still assumes that I want to define all my work inside of an ant script. Using the "shell" builder is still VERY limiting.

You can see some sample comparison shots between the two here. I'll try to actually setup a repo that Bamboo can use and do a deeper comparison later.

So what's the focus of Google Code, Launchpad...

Going back to code hosting and comparing Github to the others, I think it's clear that they lack a focus. They try to do too much. They "feel" like they were designed by project managers and targeted at them. Maybe it was a faulty assumption that to effectively manage a large project, you had have all of the extra stuff. I don't know. Launchpad and others DO some things better than Github. Issue tracking is one. Github issue tracking is a pretty weak area for them. However here's where Github understands its focus and strengths.

Where Github lacks, it makes up for in integration. Github doesn't TRY to be the project manager's tool. It doesn't try to be a good issue tracker. What it DOES do is say "I suck at this. My focus is on the code and making working with and contributing to the code dead simple. I'll add hooks for the other stuff"

And they do. Github has a boatload of service hooks for everything from issue tracking to project management to irc and IM. They even have a "generic" hook that will submit JSON to a url for you so you can write your own receiver.

About BitBucket, backend technology and focus

I haven't mentioned much about BitBucket. The main reason is that at this point, BitBucket is simple attempting to feature copy from Github except using Mercurial in the background. Sadly, this isn't enough I think. If my only reason for using BitBucket is the DVCS tool then I honestly might as well use Github. I'll get more engagement there. See this quote from Mark Philips from Basho about why the moved from BitBucket to Github:

Why? There are several reasons, the primary of which is that GitHub,
the application, lends itself to more collaboration when developing
open source software. Again, this was a decision made on the basis of
community development; technically-speaking we were satisfied with
what Bitbucket offered.

The issue wasn't the technology. Mercurial and Git are pretty much at feature parity (as is Bazaar). One thing mercurial doesn't do out of the box is cherry picking but it's supported with extra configuration. Mercurial has hg incoming which let's you see what people are working on. Git has staging. Mercurial has better Windows support than Git. It's really six in one, half dozen in the other.

However what BitBucket DOESN'T have is the community. You see, BitBucket was playing catchup to Github. Simply copying the social aspects of Github isn't enough. Github has too much momentum precisely because they had the focus right from the start - code is king.

As a developer, my key focus is my code. It's what says the most about me. As a developer who wants to attract other developers, the best way to do that is showing the code and making that contribution as easy as possible. Github gets that.

That's why people keep asking you to switch to Github.

Potato Candy - A family recipe

2010-10-22T22:03:00.000-04:00

With Halloween right around the corner and Thanksgiving beyond that, it's getting about the time of year when I get to make Potato Candy. Yes, candy made from potatoes.

I don't know the real story behind it. Ever since I was a little pile of baby fat, it's something the kids in my family have eaten. My uncle only made it for Thanksgiving and I think Christmas get-togethers. I've tried to find a bit of history about it over the years but nothing ever concrete. My uncle's family is Irish so that's as stereotypical of a reason as any. What I did seem to track down is that it's pretty unique to the Southeast. We do things weird here, ya'll.

Not long after I married my wife (a Michigan native), her aunt was putting together a family cookbook. Now that I was part of the family I got to contribute a few things. I had my mom and step-mom provide a few entries but I reserved one for myself - Potato Candy. Since the "secret" is out and because freaking @jtimberman got me thinking about candy, I figured I'd add it here for all my interweb friends.

Ingredients

1 Potato about the size of your fist. Seriously. Don't get it any bigger. If you've got big hands, find someone with normal sized hands and compare.
1 jar of peanut butter. Creamy not Crunchy. The last thing you want to deal with when making this stuff is nuts. Trust me.
2 bags of powdered sugar. Yes, you will probably use ALL of it.
Wax paper and plenty of counter top space

Peel and boil the potato as you would to make mashed potatoes. When it gets sufficiently soft, mash that bastard up. No lumps. Again, trust me. As smooth as you can get it.

Dump it into a large mixing bowl and reach for the strongest and sturdiest spoon/stirring instrument you can find. Start folding in the first bag of powdered sugar.

This is where it gets fun. As the powdered sugar gets mixed in, this thing is going to get thick and heavy very quickly. It's going to be VERY hard to mix. Did you trust me on the sturdy spoon part? You should have. Don't even think about putting this in an electric mixer. It will burn out the motor. I've literally broken 1/4 inch dowel wooden spoons in this stuff. Your arm is going to hurt. You're going to have to put your back into it.

When you physically can't mix it ANYMORE put it aside for a minute. Spread out a nice sized area on the counter with wax paper and cover it in powdered sugar. This crap is sticky and you're going to need to manipulate it. Once you've gotten the workspace ready, start spreading the "mash" on the wax paper. Usually about 1/4 to 1/2 inch thick is good. You'll probably screw it up the first time around. I did.

Open the jar of peanut butter and start spreading it on the mash. Peaks are okay but you really want to get a good layer on there.

Now, the hard part

Somehow you're going to need to roll from one end of this beast to the other. Like a jelly roll. It's really hard and don't feel too bad if it isn't pretty. The end result is still good. You'll probably want to cover your hands in powdered sugar.

Once you've got it rolled up, flatten it back out. Stick it in the fridge overnight. The next day, cut it into smallish 1.5inch slices and enjoy.

As I said earlier, I've tried to do some research each year. The best picture I can find outside of making some myself is this one.

You probably won't be able to eat more than one or two pieces. It's REALLY rich and really thick. If you give it to kids, do it early in the day so they have time to burn it off.

Enjoy ya'll!

PyCon DevOps piggy back

2010-10-21T23:17:00.004-04:00

So I had a random idea the other night and like any other random idea I immediately sent it to Twitter.

This of course brought feedback which is the whole point, right?

The idea was to have a Velocity style conference in the South East. We all know my love for Atlanta and my half-disdain/half-jealousy of the West coast. So I threw the idea out on twitter and immediately got my first reply from Joe Heck with a bit of reality thrown in:

@lusis nice idea. critical mass with either be easy or impossible to get. You might consider riffing on existing conferences ... PyCon2011

Awesome idea so I headed off to to read up on how PyCon does that kind of thing. I shot off an email to the pycon-organizers mailing list and got some really nice responses. I also got a private tweets from people on the list as well.

The end result is this. If I want to hitchhike on the back of PyCon for a devops-related conference, here are the requirements/suggestions:

Involve Python in some way
Will need to take advantage of the Open Spaces system

This essentially means unless I (or someone else) is giving a full blown talk on Python and DevOps, it will be an ad-hoc thing. We can't reserve the spaces until the day of the conference. I'm also not sure how big the spaces are. I think this is the same place LISA was held years ago so you might be able to snag a dividable room segment?

So what does everyone think? I'm considering giving a talk on the state of devops toolchains in Python (func, cobbler, fabric, kokki, overmind, whatever else) but I don't know that I'm ready for that yet after a single LUG presentation ;)

I know that Mitchell H. of Vagrant fame was considering heading into town for it. Vagrant isn't just for Rubyists ;)

I'm open to ideas. I'd love to just have the conference I sent the tweet about but when I really think about it, I don't think I can pull something like that off in this amount of time.

Many thanks to the pycon-organizers folks for the input - Doug Hellmann, Vern Ceder and Jesse Noller. Also to Dean Goodmanson for his feedback via Twitter.

Latest Vogeler update - MongoDB, protobufs, Riak and war!

2010-10-12T01:38:00.001-04:00

I wanted to take a minute to post an update about Vogeler to those who are following the project. Let's get the easy stuff out of the way - it's not abandoned. Far from it.

There have been several reasons why I haven't made any commits lately, the least of which is both kids have been sick recently and I haven't been able to get a good solid block of time to work on it.

Technical Hurdles

Another reason is that I almost went down a rabbit hole with regards to swappable persistence. In the process of refactoring the persistence backend, I realized it should be fairly easy, using the model I put into place, to go ahead and implement MongoDB and Riak support. I started with MongoDB when I hit a wall. MongoDB does not allow dots in key names. When I ran into that issue, I realized that I made some dangerous assumptions based on the fact that I started with Couchdbkit as the interface to CouchDB:

I was using an ORM when I should have used a lower level driver. You see couchdbkit does some nice stuff like translating native Python datatypes to the appropriate datatypes. If I define a row as having DictProperty(), couchdbkit converts that into the commensurate CouchDB JSON datatypes. If I use ListProperty(), the same thing. This is really evidenced in Futon and makes using Futon as your interface to Vogeler very appealing. However this is VERY couchdb specific.

The pymongo driver, however, didn't like my strategy of dumping execution results that way. You can see the "gist" of what I'm talking about here:

;

I brought the issue up on the MongoDB mailing list here. I opened an issue for myself to braindump my thoughts. One of my biggest goals (data transparency) was starting to fall apart for me. I decided to shelve MongoDB for a moment and look at Riak. I wanted to make sure that I at least thought about how a generic model would work across multiple document stores. That's when I ran into the biggest cockblock:

protobuf

I'm almost firmly convinced that protobuf is a piece of trash. Google has some smart people but protobuf is something that quite obviously came out of the mind of someone who was sent off to "solve the RPC problem". There are quite a few issues I have with protobuf:

Despite being a "universal" format, it works well in exactly TWO languages - C and Java. Everything else is an afterthought. Don't get me started on Python support. The one guy at Google who supports protobuf on Python can't make it work on anything but Python 2.5 because that's all Google uses. He's unwilling to cut a new PyPi package just to fix all the 2.5 assumptions because he doesn't want to bump the version number. You can't even install it on anything higher than 2.5 without hacking setup.py.
You have to precompile your protos before use. I understand what Google is trying to accomplish but seriously? So I have to build the protobuf compiler to compile protos to ship with my code. There's a reason why people like FFI folks.

There are alternatives like Apache Avro that have promise but they also have their own issues. However, Basho has committed to using protobuf which does make sense. Write your own serialization framework or use an existing one? Easy answer when Google wrote one for you.

So I started to noodle out what route I wanted to take when something else came out of left field.

Sgt first Class Lance Vogeler

I have a search setup in Tweetdeck on my Droid for Vogeler. It was nice to stay on top of people mentioning the project. The name for the project came out of me pretty much immersing myself in the latest S.M. Stirling Emberverse books. One of the characters was named Ingolf Vogeler. I really enjoyed the books and liked the name so I picked it. I'm also considering using Ritva for another project.

So one day my phone starts going nuts with Vogeler alerts. I was already getting the occasional history tweet about the real Ingolf Vogeler but it turns out a soldier from Georgia, of all places, was KIA in Afghanistan. He did 8 tours in Afghanistan and 4 tours in Iraq. Politics aside (I'm personally entirely against these campaigns), I didn't want to "pollute" the twitter stream. Regardless of what I think of the current military climate in my country, I have the utmost respect for most of the members in our armed forces.

However what struck me most is that SFC Vogeler left behind a wife. A wife carrying his unborn child. That pretty much did me in. As a father myself, I was pretty torn up thinking about this happening to my wife. Yes, it was a known risk but that doesn't make it any less sad. I decided, in addition to making a donation of my own to his family and holding off on Vogeler till it wasn't alerting on Twitter so much that I would think hard about what my software means.

Steve Jobs asked a guy who emailed him this question "What have you created lately?" Someone else recently said that entrepreneurs are busy creating the next social media app that means fuckall when they could be affecting change with the software they write. That got me thinking. Could I help affect this family somehow with my project that happened to share a name with them? The best I could come up with is this:

If you use Vogeler, are interested in it or just feel like a random act of kindness, please make a small donation to the Vogeler family. My wife and I agreed that should I ever make ANY money off of the project in any identifiable form that I would donate what I could to the family. Vogeler is just a small project. I have no grand aspirations of getting integrated into some mainstream project. I'm just trying to scratch an itch - a niche itch at that. One reason I'm so gung-ho about DevOps is that, as a family man, I don't WANT to be dealing with stupid shit taking time away from my family. I've done it and I'm done with it. If my phone goes off, it's not going to be from some stupid mistake that I made editing a config file or lack of metrics causing an "oh shit we're out of space" moment. I'm past that in my career and I'm past having to work places where that's the norm rather than the exception. My family is first and foremost and anything I can do to keep it that way, I'm going to do it.

So I'm taking the Vim approach to Vogeler. I'm not going to ask anyone to go against his conscience. If you feel like a small donation to this family implies consent to the stupidity of my government then I fully understand. But if you think that open source software and the broader open source community can make a difference in more than just writing software, throw a small donation their way.

I'm going to be starting back up on Vogeler now. I've decided that for now, I'm going to attempt to keep things as generic as possible but continue to code against CouchDB. I'll keep revisiting MongoDB and Riak support but the primary target is CouchDB. If any Basho folks are reading this, if you can remove the hard dep in setup.py on protobuf, that would be awesome. You can't even install from PyPi with it in there anyway. If the MongoDB folks take a gander at this, can you do something about the dots in key names? Thanks!

Why I hope BankSimple succeeds but fear they won't have a choice in the matter

2010-10-08T03:30:00.001-04:00

My current political affiliation is not the most popular with people right now. That's understandable. I tend to approach things from a rational and unemotional perspective. Call me callous. Call me a dipshit. I'm consistent if nothing.

But the issue that's been bugging me lately transcends party or political affiliation. It's one that touches me as a technologist, an IT worker and a citizen.

DevOps and the changing IT landscape

I've blogged/tweeted/talked/rambled enough about DevOps to make people sick. In some regards we risk crossing the line with our enthusiasm and killing interest in the topic. Ignore for a minute that much of what makes up "DevOps" is not new ground. That's not to say that the "movement" hasn't introduced some amazing things.

The point is that aspects of DevOps as a philosophy are changing how people approach IT holistically. Automation. Rapid iteration. Frequent changes. Breaking down organizational silos.

So how does this play into BankSimple?

It actually plays into it in two ways. The first is that, like DevOps, BankSimple isn't doing anything "new" per se. If you're talking "old" business, banking ranks right up there with prostitution and in shares more in common with it than most would want to admit. What's different is how BankSimple is doing it. The BankSimple guys want to make money, no doubt but they're approaching it from a different focus. It's been formed, not by the traditional cadre but by "techies". It's almost "Agile Banking".

You can guarantee that because of that perspective and the history of the cofounders, the business and by extension the IT aspect will be run differently.

If you can't beat 'em, buy 'em.

I'm a hands-off kind of guy when it comes to my government. Many people thought that, based on concepts like "hope" and "change", the government was going to turn around and focus on its customers - the citizens. It hasn't yet turned out that way.

Someone asked an interesting question at the last DevOps meetup in Atlanta.

"Do you think traditional companies feel threatened by startups who use more agile methods to beat them to market?"

The answer:

"Not really. If they get to be too big of a threat, just buy them out.".

That, of course, assumes they're willing to sell. Some people might have that luxury but when the VCs start to chime in that choice may be stripped from you. On the flipside, some might believe that by selling what they've worked so hard to build up will be destroyed yet they feel an obligation to the people who started with them to sell.

But let's assume that, in the case of BankSimple they have the goals of REALLY changing banking and finance (which I think they do). Assume they don't have to sell (which I don't know).

If you can't buy em, buy a lobbyist

I was talking to an individual a few weeks back about the idea that traditional companies would get eaten alive by leaner more agile startups. We were talking specifically about the financial sector but his comment applies. So I, with a head full of steam raving about agile operations and time to market, was hit head on with this reality in one comment:

"Yeah but do they have lobbyists?"

Every company I've worked at, save one, has had lobbyists on the books. At one company we actually had different colored checks that we printed for lobbyists vice expense checks. Some people think lobbyists just "lobby" for things like deregulation and "leave my business" alone but that's not the case.

Without going into too much detail, at one company we actually LIKED a certain amount of regulation. You see a certain amount of regulation is just enough to restrict entry into the market by new players. We liked that each state had its own set of regulations. That in one state we were classified under one set of laws while in another state we were classified differently. This ARTIFICIALLY raises the barrier to entry for competitors. It keeps them small. It relegates them to one of two positions:

Small enough for us not to care because they can't afford to expand into new markets
Big enough to justify buying them to eliminate the competition.

Sure, sometimes a company will succeed and grow big enough to be REAL competition but that's fine too. Helps prevent antitrust investigations. It's a win-win all around.

Not all lobbyists are bad. I would argue that the ACLU is a "good" lobbyist but they can also exist to help preserve an existing business either buy demanding hands-off or demanding more regulation under the guise of "consumer protection".

Sometimes the tree of innovation must be watered with the blood of failed business models

I sent this out as a tweet earlier tonight. With no disrespect meant to the original quote that inspired it, I firmly believe this to be true. Without even discussing politics or regulation or monopolies, the fact of the matter is that to progress as a society some things have to die. Horse and buggy. Media distribution. Publishing. All of these changes help move us to greater things. Many a business is built on convenience and inefficiencies in the supply chain. FedEx and UPS would probably have not been needed had the post office not sucked so bad. In an age when information is passed slowly, traditional news organizations made sense. BankSimple is part of that progression just as DevOps is part of that transition in IT.

But I hope that Alex and the gang aren't as naive as some people (myself included) have been about DevOps. Just as DevOps working its way into traditional organizations will have to deal with the Boogymen of HIPPA, SOX and PCI DSS, so will BankSimple have to deal with an establishment that, if threatened, sadly has the power to essentially make them "illegal".

We need them to succeed so we can move on to better and brighter things.

.plan

2010-10-06T01:14:00.000-04:00

TODO

Work through Vogeler issue queue (http://github.com/lusis/vogeler/issues)
@padrinorb test suite enhancement - functional/integration (http://github.com/padrino/padrino-framework)
Integrate libvirt into @fog (http://github.com/geemus/fog)
Finish integrating Riak support into @padrinorb (http://github.com/seancribbs/ripple)
Create a distro that uses LFS for the explicit purpose of hosting dynamic language applications (http://lusislog.blogspot.com/2010/09/distributions-and-dynamic-languages.html)
Do all of the above while maintaining my marriage and raising two kids ;)

Distributions and Dynamic Languages - A Manifesto

2010-09-29T06:03:00.003-04:00

Background

There's been a lot of talk recently on Twitter and in various posts across the Intertubes about how the various distributions handle dynamic languages and the package system those languages use. This has been a sore spot for me for a LONG time. Recently I had a chance to "stub out" my feelings in a comment on HN. I've been meaning to write this post for a few weeks but just haven't had the time. I'm making the time now.

Distro vendors find themselves in an interesting spot. In general, the difference between Linux distributions has boiled down to a few categories:

Support
Management tools
Package format
Default desktop

For the home/desktop user, the last two (and more importantly the last one) are the biggest deciding factors. For the "enterprise" user, the first is typically key. But not all enterprises are enterprises. Would anyone argue that Facebook or Google or Twitter are not enterprise users? Of course not. However those companies don't tend to need the same level of support and have the same hang-ups as Coca-Cola or Home Depot. The latter two companies are the traditional enterprise that does things like troubleshoot servers when they fail. The former are the forward thinking companies that say "Fuck it. Pull the server and put another one in. We don't have time for this 'bench' shit."

In the same vein, the first group of companies are the kind that use Linux as a platform where the second group uses RedHat or Suse as an OS to host JBoss or Oracle or DB2. Those vendors say "We run on distros X and Y and we only support those". You don't have a choice in the second group. The first group may have standardized on a distro but the distro itself is also irrelevant. Those companies use Chef and Puppet and similar tools to totally abstract that out. The distro becomes a commodity. They just want Linux.

This is the new type of company and this is the type of company that distro vendors have to worry about.

So having said that, how does those tie into the dynamic language debacle of late? Increasingly, applications in the PaaS/SaaS space are being written in dynamic languages. The product is just different than Oracle or DB2. So these companies need to consider which distro will make using those dynamic languages as easy as possible. Frankly, they've all pretty much fucked it up. The main reason? Traditional software products.

The biggest selling point of an enterprise distro was support. That, or the fact that you were required to run RedHat or Suse for your RAC cluster. One of the main reasons that enterprise distros were able to be supported platforms for Oracle or DB2 is that they "stabilized" things. In this case that meant long term support (LTS) models and a consistent base operating system. If you ported your product to run on RHEL4, you could guarantee that RedHat would never break compatibility for the life of that product support cycle (I think it's 7 years right now?). You could also be assured that version X of a package would be available for the platform should you need it.

The Problem

That worked fine for binary COTS products. Not so fine for the world of dynamic languages where new versions of a Gem or Python package come out daily. And ESPECIALLY not when the language package system allows for multiple versions of the same package to be installed alongside each other. But is this really a big deal? The distros can just upgrade python to 2.7 right? Nope and the reason why?

Management tools

I don't fault the distro vendors for using python (as an example) as the higher level management language for the OS. In fact, having now gotten into Python, I think it's a wonderful idea. It is, language wars aside, a very approachable and consistent language. It allows them to quickly iterate those tools and especially in the case of Python, the core language changes very little. It's mature.

So now distro vendors have gone and written core parts of the operating system to use Python. Combine that with the package manager restrictions and LTS and you have a system where, if you upgrade Python, you've broken the system beyond repair. This is why RHEL5 is still on Python 2.4.

This is the where we find ourselves today. Distro vendors have to continually package all the python modules they want to supply in native package format to the version of the runtime they use. Eventually the module/gem maintainer is going to stop supporting that module on such old runtimes. Now they essentially have to maintain backports for the life of the LTS terms. This is madness. Why would you put yourself in this situation? I didn't know this but FreeBSD evidently solved this problem a while ago by moving all core scripts away from Perl.

The Manifesto

So here's my manifesto. My suggestion if you will as a long time Linux user, enterprise customer and dynamic language programmer.

Stop it. Get out of the game now. As much as you would like to think your customers care about LTS for Perl/Python/Ruby, they don't. Your LTS is irrelevant six months after you cut a new release of a distro. RHEL6 is shipping with Ruby 1.8.6. Seriously? Not even 1.8.7? I understand they have a long development cycle for new distro versions which is why I'm saying get out. You can't keep up.

But what about our management tools?

I've solved that for you to. system-python, system-ruby, system-perl. Isolate them. Treat them as you would /opt/python or /opt/ruby. Make them untouchable. Minimize your reliance on any module/gem/library you don't directly maintain (i.e. a gtk python module). Understand that you will be wasting resource on backporting this module for 5 or 7 years. No more '/usr/bin/env python'. Shebang that bastard to something like '/usr/lib/system-python/bin/python'

So now that you've isolated that dependency, what about people who don't WANT to compile a new ruby or python vm? How do you provide value to them? The ActiveState model. /usr/lib/python27, /usr/lib/python31, /usr/lib/ruby187.

But wasn't the point of this whole discussion around DLR package management? We don't want to maintain a package per vm version of some library.

Then don't.

This is where the onus is on the language writers. Your package format needs to FULLY support installing from a locally hosted repo of some kind. You may not believe it but not every server has internet access. At our company, NONE of the servers can get to the Internet. The still serve content TO the internet but can't get out. Not by proxy. Not at all.

We're essentially forced to download python packages or jar files and copy them to a maven server or host them from apache to use them internally. Either that, or package them as RPMs. With the python packages, it's especially annoying because, while pip will happily pull from any apache-served directory of tarballs, we can't push from setup.py to it. We don't have ANY metadata associated with it at all.

So Ruby/Python/Perl guys, you need to either provide a PyPi/Gem server package that operates in the same way as your public repos do or make those tools operate EXACTLY the same with a local file path as they do with a URL. Look at createrepo for RPMs for an idea of how it can work if you need to. Additionally, tools like RVM and virtualenv really need to work with distro vendors. RVM does a stellar job at this point. Virtualenv has a way to go.

So now the distro vendors have things isolated. They ship said language repo server and by default point all the local language package tools to that repo path or server. Now if the user chooses to grab module X from PyPi to host locally, they've made that decision. It doesn't break they OS. You don't offer support for it unless you really want to and this whole fucking problem goes away.

EDIT:

I realize I'm not saying anything new here. I also realize that distro vendors realize that the distro itself is a commodity. RedHat figured that out a long time ago. Look at the JBoss purchase and everything since then. Additionally, virtualization removes any reason you might have for picking distro X over distro Y because of hardware support in the distro.

Hiring for #devops - a primer

2010-09-22T03:05:00.001-04:00

I've written about this previously as part of another post but I've had a few things on my mind recently about the topic and needed to do a brain dump.

As I mentioned in that previous post, I'm currently with a company where devops is part of the title of our team. I won't go into the how and why again for that use case. What I want to talk about is why organizations are using DevOps as title in both hiring and as an enumerated skillset.

We know that what makes up DevOps isn't anything new. I tend to agree with what John Willis wrote on the Opscode blog about CAMS as what it means to him. The problem is that even with such a clear cut definition, companies are still struggling with how to hire people who approach Operations with a DevOps "slant". Damon Edwards says "You wouldn't hire an Agile" but I don't think that's the case at all. While the title might not have Agile, it's definitely an enumerated skill set. A quick search on monster in a 10 mile radius from my house turned up 102 results with "Agile" in the description such as:

experienced Project Manager with heavy Agile Scrum experience
Agile development methodologies
Familiar with agile development techniques
Agile Scrum development team

Yes, it's something of a misuse of the word Agile in many situations but the fact of the matter is that when a company is looking for a specific type of person, they tend to list that as a skill or in the job description. Of course Agile development is something of a formal methodology whereas DevOps isn't really. I think that's why I like the term "Agile Operations" more in that regard. But in the end, you don't have your "Agile Development" team and so you really wouldn't have your "Agile Operations" team. You have development and you have operations.

So what's a company to do? They want someone who "does that devops thing". How do they find that person? Some places are listing "tools like puppet, chef and cfengine" as part of skill sets. That goes a long way to helping job seekers key off of the mindset of an organization but what about the organization? How do they determine if the person actually takes the message of DevOps to heart? I think CAMS provides that framework.

Culture and Sharing

What kind of culture are you trying to foster? Is it one where Operations and Development are silos or one where, as DevOps promotes, the destruction of artificial barriers between the groups? Ask questions of potential employees that attempt to draw that out of them. Relevance to each role is in parenthesis.

Should developers have access to production? Why or why not? (for Operations staff)
Should you have access to production? Why or why not? (for Development staff)
Describe a typical release workflow at a previous company. What were the gaps? Where did it fail? (Both)
Describe your optimal release workflow. (Both)
Have you even been to a SCRUM? (Operations)
Have you ever had operations staff in a SCRUM? (Development)
At what point should your team start being involved/stop being involved in a product lifecycle? (Both)
What are the boundaries between Development and Operations? (Both)
Do you have any examples of documentation you've written? (Both)
What constitutes a deployable product? (Both)
Describe your process for troubleshooting an outage? What's the most important aspect of an outage? (Both)

Automation and Metrics

This is somewhat equivalent to a series of technical questions. The key is to deduce the thought process a person uses to approach a problem. Some of these aren't devops specific but have ties to it. Obviously these might be tailored to the specific environment you

Describe your process for troubleshooting an outage? What's the most important aspect of an outage? (Both)
Do you code at all? What languages? Any examples? Github repo? (Operations)
Do you code outside of work at all? Any examples? Github repo? (Development)
Using psuedo-code, describe a server. An environment. A deployable. (Operations)
How might you "unit test" a server? (Operations)
Have you ever exposed application metrics to operations staff? How would you go about doing that? (Development)
What process would you use to recreate a server from bare metal to running in production? (Operations)
How would you automate a process that does X in your application? How do you expose that automation? (Development)
What does a Dashboard mean to you? (Both)
How would you go about automating production deploys? (Both)

A few of these questions straddle both aspects. Some questions are "trick questions". I'm going to assume that these questions are also tailored to the specifics of your environment. I'm also assuming that basic vetting has been done.

So what are some answers I like to hear vice don't ever want to hear? Anything that sounds like an attitude of "pass the buck" is a red-flag. I really like seeing an operations person who has some sort of code they've written. I also like the same from developers outside of work. I don't expect everyone to live, breathe and eat code but I've known too many people who ONLY code at work and have no interest in keeping abreast of new technologies. They might as well be driving a forklift as opposed to writing code.

I think companies will benefit more from a "technologist" than someone who is only willing to put in 9to5 and never step outside of a predefined box of responsibilities. I'm not suggesting that someone forsake family life for the job. What I'm saying is that there are people who will drag your organization down because they have no aspirations or motivations to make things better. I love it when someone comes in the door and says "Hey I saw this cool project online and it might be useful around here". I love it from both developers and operations folks.

Do with these what you will. I'd love to hear other examples that people might have.

Follow up to #vogeler post

2010-09-12T14:23:00.001-04:00

Patrick Debois was kind enough to comment on my previous post and asked some very good questions. I thought they would fit better in a new post instead of a comment box so here it is:

I read your post and I must say I'm puzzled on what you are actually achieving. Is this a CMDB in the traditional way? Or is it an autodiscover type of CMDB, that goes out to the different systems for information? In the project page you mention à la mcollective. Does this mean you are providing GUI for the collected information? Anyway, I'm sure you are working on something great. But for now, the end goal is not so clear to me. Enlighten me!

Good question ;) I think it sits in an odd space at the moment because it tries to be flexible and by design could do all of those things. Mentioning Mcollective may have clouded the issue but is was more of a nod to similar architectural decisions - using a queue server to execute commands on multiple nodes.

My original goal (outside of learning Python) was to address two key things. I mentioned these on the Github FAQ for Vogeler but it doesn't hurt to repost them here for this discussion:

What need is Vogeler trying to fill?

Well, I would consider it a “framework” for establishing a configuration management database. One problem that something like a CMDB can create is that, to meet every individual need, it tends to over complicate. One thing I really wanted to do is avoid forcing you into my model and trying to provide ways for you to customize the application.

I went the other way. Vogeler at the core, provides two things – a place to dump “information” about “things” and a method for getting that information in a scalable manner. By using a document database like CouchDB, you don’t have to worry about managing a schema. I don’t need to know what information is actually valuable to you. You know best what information you want to store. By using a message queue with some reasonable security precautions, you don’t have to deal with another listening daemon. You don’t have to worry about affecting the performance of your system because you’re opening 20 SSH connections to get information or running some statically linked off-the-shelf binary that leaks memory and eventually zombies (Why hello, SCOM agent!).

In the end, you define what information you need, how to get it and how to interpret it. I just provide the framework to enable that.

So to address the question:

If we're being semantic, yes it's probably more of a configuration database than a configuration MANAGEMENT database. Autodiscovery, though not in the traditional sense, is indeed a feature. Install the client, stand up the server side parts and issue a facter command via the runner. You instantly have all the information that facter understands about your systems in CouchDB viewable via Futon. I could probably easily write something that scanned the network and installed the client but I have a general aversion to anything that sweeps networks that way. More than likely, you would install Vogeler when you kicked a new server and managed the "plugins" via puppet.

I hope that makes sense. Vogeler is the framework that allows you to get whatever information about your systems you need, store it, keep that information up to date and interpret it however you want. That's one reason I'm not currently providing a web interface for reporting right now. I just simply don't know what information is valuable to you as an operations team. Tools like puppet, cfengine, chef and the like are great and I have no desire to replace them but you COULD use this to build that replacement. That's also why I use facter as an example plugin with the code. I don't want to rewrite facteri. It just provides a good starting tool for getting some base data from all your systems.

Let's try a use case:

I need to know which systems have X rpm package installed.

You could write an SSH script, hit each box and parse the results or you could have Vogeler tell you. Let's assume that the last run of "package inventory" was a week ago:

vogeler-runner -c rpms -n all

The architecture is already pretty clear. Runner pushes a message on the broadcast queue, all clients see it ('-n all' means all nodes online) and they in turn push the results into another queue. Server pops the messages and dumps them into the CouchDB document for each node. You could then load up Futon or a custom interface you wrote and load the CouchDB design doc that does the map reduce for that information. You have your answer.

Now let's try something of a more complicated example:

I need to know what JMX port all my JBoss instances are listening on in my network.

Well I don't provide a "plugin" for you to get that information, a key for you to store it under in CouchDB or a design doc to parse it by default. But I don't need to. We take the Nagios approach. You define what command returns that information. A shell script, a python script, a ruby script whatever works for you. All you need to tell me is what key you want to store it under and something about the basic structure of the data itself. Maybe your script provides emits JSON. Maybe it emits YAML. Maybe it's a single string. Maybe you run multiple JBoss instances per machine each listening on different JMX ports (as opposed to aliasing IPs and using the standard). I'll take that and create a new key with that data in the Couch document for that system. You can peruse it with a custom web interface or, again, just use Futon.

Does that help?

Notes on #vogeler and #devops

2010-09-12T03:51:00.003-04:00

UPDATE: There's some additional information about Vogeler in the followup post to this one:

Background

So I've been tweeting quite a bit about my current project Vogeler. Essentially it's a basic configuration management database built on RabbitMQ and CouchDB. I had to learn Python for work, we may or may not be using those two technologies so Vogeler was born.

There's quite a bit of information on Github about it but essentially the basic goals are these:

Provide a place to store configuration about systems
Provide a way to update that configuration easily and scalably
Provide a way for users to EASILY extend it with the information they need

I'm not doing a default web interface or much else right now. There's three basic components - a server process, a client process and a script runner. The first two don't act as traditional daemons but instead monitor a queue server for messages and act on that.

In the case of the client, it waits for a command alias and acts on that alias. The results are stuck on another queue for the server. The server sits and monitors that queue. When it sees a message, it takes it and inserts it in the database with some formatting based on the message type. That's it. The server doesn't initiate and connections directly to the clients and neither do the clients talk directly to the server. All messages that the clients see are initiated by the runner script only.

That's it in a nutshell.

0.7 release

I just released 0.7 of the library to PyPi (no small feat with a teething two year old and 5 month old) and with it, what I consider the core functionality it needs to be useful for people who really are interested in testing it. Almost everything is configurable now. Server, Client and Runner can specify where each component it needs lives on the network. CouchDB and RabbitMQ are running in different locations from the server process? No problem. Using authentication in CouchDB? You can configure that too. Want to use different RabbitMQ credentials? Got it covered.

Another big milestone was getting it working with Python 2.6. No distro out there that I know of is using 2.7 which is what I was using to develop Vogeler. The reason I chose 2.7 is that was the version we standardized on and since I was learning a new language and 2.7 was a bridge to 3, I chose that one. But when I went to started looking at trying the client on other machines at home, I realized I didn't want to compile and setup the whole virtualenv thing on each of them. So I got it working with 2.6 which is what Ubuntu is using. For CentOS and RedHat testing, I just used ActivePython 2.7 in /opt/.

Milestones

As I said 0.7 was a big milestone release for me because of the above things. Now I've got to do some of the stuff I would have done before if I hadn't been learning a new language:

Unit Tests - These are pretty big for me. Much of my work on Padrino has been as the Test nazi. Your test fails, I'm all up in your grill.
Refactor - Once the unit tests are done, I can safely being to refactor the codebase. I need to move everything out of a single .py with all the classes. This also paves the way for allowing swappable messaging and persistence layers. This is where unit tests shine, IMHO. Additionally, I'll finish up configuration file setup at this point.
Logging and Exception handling - I need to setup real loggers and stop using print messages. This is actually pretty easy. Exception handling may come as a result of the refactor but I consider it a distinct milestone.
Plugin stabilization - I'm still trying to figure out the best way to handle default plugins and what basic document layout I want.

Once those are done, I should be ready for a 1.0 release however before I cut that release, I have one last test.....

The EC2 blowout

This is the part I'm most excited about. When I feel like I'm ready to cut 1.0, I plan on spinning up a few hundred EC2 vogeler-client instances of various flavors (RHEL, CentOS, Debian, Ubuntu, Suse...you name it). I'll also stand up distinct RabbitMQ, CouchDB and vogeler-server instances.

Then I fire off the scripts. Multiple vogeler-runner invocations concurrently from different hosts and distros. I need to work out the final matrix but I'll probably use Hudson to build it.

While you might think that this is purely for load testing, it's not. Load testing is a part of it but another part is seeing how well Vogeler works as a configuration management database - the intended usage. What better way than to build out a large server farm and see where the real gaps are in the default setup? Additionally, this will allow me to really standardize on some things in the default based on the results.

At THAT point, I cut 1.0 and see what happens.

How you can help

What I really need help with now is feedback. I've seen about a 100 or so total downloads on PyPi across releases but no feedback on Github yet. That's probably mostly due to such minimal functionality before now and the initial hurdle. I've tried to keep the Github docs up to date. I think if I convert the github markdown to rst and load it on PyPi, that will help.

I also need advice from real Python developers. I know I'm doing some crazy stupid shit. It's all a part of learning. Know a way to optimize something I'm doing? Please tell me. Is something not working properly? Tell me. I've tried to test in multiple virtualenvs on multiple distros between 2.6 and 2.7 but I just don't know if I've truly isolated each manual test.

Check the wiki on github and try to install it yourself. Please!

I'm really excited about how things are coming along and about the project itself. If you have ANY feedback or comments, whatsoever, please pass it on even if it's negative. Feel free to tell me that it's pointless but at least tell me why you think so. While this started out as a way to learn Python, I really think it could be useful to some people and that's kept me going more than anything despite the limited time I've had to work on it (I can't work on it as part of my professional duties for many reasons). I've been trying to balance my duties as a father of two, husband, Padrino team member along with this and I think my commitment (4AM...seriously?) is showing.

Thanks!

Five open source Projects I wish I could fund

2010-07-21T22:31:00.002-04:00

I've always said to myself that if I ever become independently wealthy, I'm going to bankroll some things I've always wanted that the opensource community hasn't felt a need to provide. Mind you, I'm not independently wealthy so don't expect to see much from me.

Anyway, here's my current "wishlist":

OpenWire Ruby drivers for ActiveMQ.

For that matter, I'd love wire level drivers for a bunch of stuff. In the case of ActiveMQ, it's nice that it's all plaintext but it doesn't support some of the same semantics as the OpenWire drivers and quite honestly wasn't very reliable in the testing I did. Say it with me folks, stateless protocols are not the way to talk to queue servers and ESPECIALLY not over HTTP. REST semantics don't map properly to core message queue concepts.

Non-Win32/DLL Ruby drivers for MSMQ and other Microsoft products

This really bit me in the ass at the AJC. It would have made my life a whole lot easier if we had a method for talking to MSMQ from a non-Windows platform. Sure, Microsoft documents the protocols for the most part but unless I'm planning on learning C and implementing a native extension, I don't see me doing it.

An open source ETL/DW/BI suite built on NoSQL. Bonus points for supporting rolling warehouse loads.

It may sound silly but I always thought that of all the promise of NoSQL concepts, the fact that your warehouse is denormalized makes it a great fit. I also think Map/reduce is a much more logical construct for BI reporting. There are a few headaches though which is why, even as a self-contained suite, it will take effort to gain traction:

ETL vendors would need to support the NoSQL engine on the Load side
BI/Reporting tools would need to support the NoSQL engine
Report creators (many times, employees from each business unit stakeholder and non-technical) need to learn Map/Reduce concepts for scheduled reports
Map/reduce is a poor/impossible choice for Ad-hoc queries at least as far as the current crop of NoSQL engines is concerned.

Essentially, you would HAVE to create your own suite - soup to nuts - and provide a way to move people from thinking in SQL for report generation. Maybe a hybrid approach makes more sense. Assuming I were king for a day, the warehouse side would be a hierarchical design - all data is dumped denormalized into a NoSQL engine. Scheduled reporting is done via Map/reduce against that data. Additionally a second load phase either concurrent with or post NoSQL load (does that make it ETLL?) dumps a business rule defined amount of data in a traditional RDBMS store for Ad-hoc purposes. I dunno. I could be over-engineering it ;)

PostgreSQL and MySQL move to a pluggable replication architecture based on message queues.

I'm not sure if this is still the case but many years ago, DB2 was using MQ Series for geographical replication. Message queues are message agnostic and implement all the features required of replication - guaranteed delivery and ordered delivery for instance. Imagine how easy it would be to scale out MySQL read slaves if they weren't all hitting the master server? Message queues are perfect for this. I might implement it something like this with ActiveMQ:

Replication messages are pushed to a queue for known slaves. One queue per slave.
Said messages are duplicated into a Topic
New slaves subscribe to the Topic and come current
New slave is then converted to its own queue

Slaves never talk to the master server directly. You can spin up slaves at any time even without a backup. Just bring the slave up, point to the topic and get current on your own time. At some given point, you're converted to your own queue and unsub from the topic.

A DSL for implementing random binary protocols.

I thought this was what protobuf did but as I look at it more, I realize I might have been mistaken. Imagine if you could take the MSDN docs that describe the MSMQ protocols. Convert that information into said DSL and execute 'foo' against the DSL. Blammo, you have a driver for that protocol. Is that even possible?

Anyway, there goes my business ideas for the next century. I do hope someone runs off with them and does something fun. In seriousness, I can't be the only person who's ever thought of these things. Hell, look at the database replication one. I straight stole that from IBM.

Besides, there's probably patents on all of these ideas already =P

No operations team left behind - Where DevOps misses the mark

2010-07-13T21:53:00.002-04:00

I'm a big fan of the "DevOps" movement. I follow any and everyone on twitter who's involved. I've watched SlideShare presentation after presentation. I've pined for a chance to go to Velocity. I watched the keynote live. I've got "the book". These guys are my heroes, not because they did something new per se but because they put a name on it. Gave it a face from the formless mass. Brought it to the forefront.

Any operations guy worth his salt has been doing some part of what is constituting DevOps for a long time. We automated builds. If we had to do something more than once, we wrote a script to handle it. My favorite item from ThinkGeek was a sticker that said "Go away or I will replace you with a very small shell script". We pxe-booted machines from kickstart files. We were lazy and didn't want to have to deal with the same bullshit mistakes over and over. When I read the intro to the Web Operations book, I was shouting outloud because this was the FIRST book that accurately described what I've been doing for the past 15 years.

I tell you all that so you don't think I'm down on the "tribe" (as Tim O'Reilly called us). These are my people. We're on the same wavelength. I love you guys. Seriously. But just like any intervention, someone has to speak out. There's a "trend" that seems to be forming that's leaving some operations teams behind and those folks don't have a choice.

I mentioned in a previous post that I'm working for a new company. Because of legal restrictions and company security policy, among other things, I can't go into too many details. However, the same things I'm going to be talking about apply to more than just our company.

The company recently formed a dedicated group called "DevOps". The traditional SA/Operations team was reformed into a "DevOps Support" and a handful of other folks were formed into a "DevOps Architecture" team. Right now that second group consists of me and two of the senior staff who moved over from the original SA team. Now you might look at this and say "Yer doin' it wrong!" but there's some logic behind this thought process. Without breaking out a few people from the daily operational support issues, no headway could be really made on implementing anything. This isn't to imply anything about how the company operates or the quality of the product. It's simply a fact of trying to retrofit a new operational model on top of an already moving traditional business process. The same issues arose when teams started migrating from a waterfall to agile. Sure you could implement agile in the NEXT project but forget about upsetting the boat on the current product line. In addition to changing how developers operated, you had a whole host of other stakeholders who needed to be convinced.

I once had a manager who I really disliked but he had a saying - "It's like changing the tires on the race car while it's going around the track"

That's the position many traditional companies are in right now. Walking in the door and telling them they really should be doing X instead of Y is nice. Everyone with a brain knows it makes sense. It's obviously more efficient, reduces support issues, makes for a better work environment and cures cancer but it simply cannot be implemented by burning the boat. So, yes, some companies will have to form dedicated groups and work with stakeholders and go through the whole process that a DevOps mentality is trying to replace just to implement it.

But that's not the only roadblock.

Sarbanes-Oxley

Any publicly traded company regardless of industry has its hands tied by three letters - SOX. Excluding specific sector requirements - HIPPA for medical, PCI for financial, FCPA, GLBA or (insert acronym here), Sarbanes-Oxley puts vague and onerous demands on public companies. Hell, you don't even have to be publicly traded. You could be a vendor to a publicly traded company and subject to it by proxy. Sarbanes-Oxley is notoriously ambiguous about what you actually have to DO to pass an audit. Entire industries have sprung up around it from hardware and software to wetware.

What's most amazing about it is that, I personally think implementing a DevOps philosophy across the board would make compliance EASIER. All change control is AUTOMATICALLY documented. Traditional access rules aren't an issue because no human actually logs onto servers for instance.

However in the end you have to convince the auditor that what you are doing matches with the script they have. In every company I've been at we've had the same workflow. It's like all the auditors went to the same fly by night school based on some infomercial: "Make big money as a SOX auditor. Call now for your free information packet!"

Change is requested by person W
Change is approved by X stake holders
Change is approved by Y executive
Change is performed by person Z
The person who requested the change can't approve it.
The person approving the change can't perform the actual work.
So on and so forth.

Continuous deployment? Not gonna happen. It can't be done with that level of handcuffing.

Security Controls

Moving past the whole SOX issue, there are also security concerns that prevent automation. It's not uncommon for companies to have internal VPNs that have to be used to reach the production environment. That means the beautiful automated build system you have is good up until, say, QA. Preproduction and on requires manual access to even GET to the servers. This model is used in companies all over the world. Mandatory encryption requirements can further complicate things.

Corporate Hierarchy

I was recently asked what I found was the biggest roadblock to implementing the DevOps philosophy. In my standard roundabout way of thinking something through, I realized that the biggest roadblock is people. People with agendas. People who have control issues. People who are afraid of sharing knowledge for fear of losing some sort of role as "Keeper of the Knowledge". Those issues can extend all the way to the top of a company. There's also the fear of change. It's a valid concern and it's even MORE valid when a misstep can cost your company millions of dollars. You have no choice but to move slow and use what works because you know it works. It's not the most efficient but when it's bringing in the money, you can afford to throw bodies at the issue. You can afford to have 10 developers on staff focused on nothing but maintaining the current code base and another 10 working on new features.

The whole point of this long-winded post is to say "Don't write us off". We know. You're preaching to the choir. It takes baby steps and we have to pursue it in a way that works with the structure we have in place. It's great that you're a startup and don't have the legacy issues older companies have. We're all on the same team. Don't leave us behind.

Locked down!

2010-07-08T22:18:00.001-04:00

So I started at the new company today. I was supposed to start Tuesday but there was some delay in my on-boarding.

But that's neither here nor there.Here's the interesting thing. The company is a publicly traded financial services company. It's not enough to be publicly traded but to also be in financial services is like taking that giant cake of government scrutiny and slapping on another layer for fun.

Did I mention they're also international?

Anyway, I get my laptop and get logged in. This thing is locked down TIGHT. The kicker is that it's running Windows XP. Because of corporate policy, the only tools I'm allowed to install are cygwin, putty and winscp3. If I want to use an IDE, it's got to be eclipse. Nothing else is approved. Boot-level disk encryption. OS level disk encryption. Locked....down.

So I pretty much spent my entire afternoon trying to get cygwin running in something resembling usefulness. Mind you I haven't used Cygwin in AGES. I haven't used a windows machine for work in at least 6+ years. I've been fortunate enough to work for companies that allowed me to wipe the corporate install and run Linux as long as I didn't bother to ask for help with it. Where I ran into the next problem was with dealing with random cygwin issues.

So I hit google and start searching. Click the first result:

KA-BLOCK as Kevin Smith is fond of saying on twitter.

Blocked because it's a blog. Next result. Same thing. Finally I get a mailing list archive that isn't blocked and get most of the issues resolved. Meanwhile I've set off probably 20 alerts not because of any malicious activity but because I couldn't be sure if the search result would be a proxy violation or not. Hell, half the mirrors for cygwin I tried were blocked in the category of "Software Downloads". Really frustrating.

I finally get some semblance of a working system but I find myself wondering how I'm going to manage my standard workflow with this machine. It's going to be a challenge to say the least. In talking with my peers, it's pretty clear that they all have the same concerns and issues. Most of the time they work entirely in windowed screen sessions on one of the internal servers. This is fine by me but it's a big change in my workflow. I've been using the same keybinds for the past 6 years or so. I pretty much have to unlearn ALL of them because I can't use them on Windows. The upshot is that I got gnome-terminator installed via cygwin ports. The hardest part was the fact that the homepage for gnome-terminator was blocked, you got it, because it was a "blog".

The point of this post is not to disparage the company in any way, shape, form or fashion. It got me wondering though how in the world people accomplish anything in environments like this?

Forget the standard employee who uses email and the standard MS Office suite. What about developers who are developing code that runs on an entirely different OS. How many bugs and delays have companies had because the developer was unable to use an OS that mirrors that of the production environment. This particular company is a java shop. Java is a little more lax in this area but you still have oddities like "c:/path/to/file" that are entirely different on the server side. More so how many steps had to be injected in the workflow to get around that kind of issue.

While I really HATE working on OSX at least it's more posix compliant than windows. My biggest headaches are how services are managed differently and the fact that it's not quite unix-alike for my tastes. It's like the uncanny valley.

I guess I'm feeling some trepidation because in addition to having to learn a new workflow - and a slower one at that - I'm also going to be working in Python. I'm excited about the work I'll be doing (DevOps - see my previous post about DevOps as a title) and the impact it will have but I also feel like I'm doubly behind - new workflow and a new language. The only thing that could make me more nervous is if the entire backend were Solaris - my weakest unix ;)

Anyway, I'll be fine. One upshot is that I AM allowed (as far as I was told) to run VirtualBox in host-only mode. Using some guest/host shared folder magic, I should be able to minimize the impact of the slower workflow.

The opposite of DevOps

2010-06-22T23:06:00.002-04:00

I thought about whether or not to write this post but I think it's an interesting example of the kinds of problems that the DevOps methodology is trying to solve.

I turned in my two weeks notice with the paper on Friday. There were several reasons but none of them are a negative reflection on my employer. The role I was originally hired for was no longer valid and there wasn't a transition path because of platform changes in the back office. I could have stayed (and was asked to stay) but there wasn't anything long term where my skill set was useful. As with any key team member, meetings are called to discuss any outstanding issues, transition responsibilities and the like.

In preparation for the meeting today, I drew up a list my responsibilities at a macro level and then broke that down by task. As it's always been for me, that list spanned several "silos" in the traditional IT organization model. Here's a sample snippit:

Subversion	User Management, Repository Management
MySQL	User Management, Database Management, Performance Management
Linux	User Management, OS Configuration, Application Management, Code Management
Puppet	Configuration Management, Recipe Management
Ruby	VM Management, Gem Management

There were also entries for various internal applications that I've worked on (including some development) and supported from an operational perspective. Mind you, I was embedded as an operations guy with a specific development group in the organization. Sort of a DevOps-lite role.

What was really interesting about the meeting was how the lines were broken down. Literally lines were drawn as to where operations would stop supporting something and the group I was leaving would take over. Take the Linux example:

Application Management was intended to refer to software that was "standard" as a part of the distro. Things like Apache or MySQL. Code Management refers to our internally developed code (all Rails applications except for the sexy Sinatra webservice I wrote). In the end, the responsibilities were shifted divided like so:

- Operations Team supported up to the installation and configuration of Apache (including vhosts) with the exclusion of Passenger configuration.

- Passenger configuration and internal code would now be managed by the Development group. They would handle deployments themselves via Webistrano.

- MySQL? Passed on to those currently managing the MSSQL database servers.

That's the very definition of a silo'd IT infrastructure. Everything is thrown over the wall. At least the deploys remain with the development team. There is nothing "wrong" with this model of IT governance. It's not agile but many companies use it. Contrast that to the position I'll be starting on the 6th of July.

An explicit group is being formed inside the company called "DevOps". I know that at this point everyone is incredulous. I can hear it now; "You're doing it wrong!". Interestingly enough, I asked the same question during the interview process. We all know that DevOps is model and philosophy and not a title or department. You don't have two development teams - Agile and Waterfall. You have Developers and they use one methodology over the other. The same goes for Operations. The people I was interviewing with were cognizant of this fact as well. The reason the group is being called "DevOps" is strictly for organizational and political reasons. The goal of the team is to actually develop a set of operational and developmental processes,tools and guidelines that embody everything that DevOps represents.

This is being done inside of one division of the company for now with the Director of Development and Director of Operations essentially sharing a brain and heading things up. The work that this group does will establish and codify something that will be used throughout the rest of the company. We'll be doing development of tools, architecting systems and recommending/implementing solutions that will, among other things, define how the organization operates itself from an IT perspective. Additionally we'll be supporting this as any traditional operations team would but, for now, the group has a distinct title. My title is "Systems Architect" which nice and generic enough to apply to both traditional groups ;)

The only remotely distressing part of the whole thing is that I'll probably have to learn some Python. (tongue firmly in cheek). There's a point to be made that distribution vendors have settled on Python as the Lingua Franca of OS management - excluding things like Chef, Puppet and Nagios which have their own respective DSLs that abstract away much of the language they were written in. Yes, my precious Ruby will still be there when it comes to extending, say, Puppet. I'll probably still write quite a few service checks for Nagios in shell (should Nagios be the best fit).

But language wars aside, I think this gives a clear picture of exactly the types of problems the DevOps methodology is trying to solve. Agility and flexibility across all branches of IT produces a leaner, strong and faster organization that can spend less time "in the muck" (as John Willis likes to say) and start making the company more money.

Status of Riak support in Padrino

2010-06-21T09:30:00.002-04:00

Back story

(skip to the bottom if you want the nitty gritty status)

So I got this wild hair up my butt a week or so ago. I wanted to get into the Padrino internals and actually contribute to an open source project. I've been using open source software for years. I've made a good living from it. I owe a lot to it.

My problem has always been the fact that I'm not a programmer by education nor by trade. I've always been a systems/architecture guy. Of course, a systems guy is a programmer at some level thanks to shell scripting, administrative scripts and the like. The DevOps philosophy makes that even more tangible by treating your systems in the same way a developer would treat his code. The most I've ever done previously in terms of contribution has been a bug fix here and there or documentation. All very valuable but in some capacity, not as rewarding.

So I've started mucking about with Sinatra. I used it to build some web service gateways for our production environment. That led me to Padrino. I noticed that Padrino didn't have ORM support for a few of the other schemaless/NoSQL databases so I figured it was a good way to contribute. I picked Riak out of the blue because the first episode of the ChangeLog Show I listened to had the Riak guys on there. Little did I know ;)

Riak/Ripple Status

Once I got a handle on the Ruby driver that Sean Cribbs is writing for Riak, I dove right in and forked the Padrino code base. Github makes it EXCEEDINGLY easy to be a contributer and I really think they've ushered in a new wave of open source development.

The Riak ruby driver comes in two flavors - riak-client and ripple. Riak-client is a more "basic" wrapper around riak operations - CRUD, link-walking, map/reduce. Ripple is the "next-gen" (imho) driver that borrows design from ActiveRecord, MongoMapper and DataMapper.

After poking a bit with riak-client, I decided that the ripple driver was much more "in-line" with the other ORMs supported in Padrino. I started to go whole hog before I realized that ripple had some missing functionality I was expecting. This was not anyone's fault but my own. Essentially this prevented me from using the ripple ORM in Padrino's admin interface. Not a big deal but it would have been nice to have.

Sean Cribbs was VERY responsive over twitter and let me know that the features I was looking for would be added. He opened an issue and last night, support for update_attribute/update_attributes was added. I grabbed the latest build and went to town on my local padrino-framework fork.

This is where I ran into another issue. Essentially, the other NoSQL ORMs that padrino supports use some tricks for handling unique keys. I know this is trying to shoehorn RDBMS ideas on top of schema-less databases.

So how do the other ORMs handle it?

mongomapper/mongoid - supports 'validates_uniqueness_of' on model definitions
couchrest - supports unique records via a map/reduce job

Nothing similar exists in the ripple driver "yet". I say yet because I fired off an email to Sean and got a very well-thought out response.

As a side note, Basho, you hired a good man to bear the title of "Developer Advocate".

Essentially Sean brought up a good point. Since you don't have "transactions or global consistency", there's no good way to guarantee that a key is unique.

I've essentially got two options if I want to continue down the path of support Padrino's admin with Riak - do the map/reduce route or do a check myself before calling save. I'm still deciding the best route to take.

Alternatively, I could simply not worry about it and forgo admin support in the ORM. It's not a deal breaker and it's not a requirement per the Padrino folks. It just would have been nice.

So what's the status?

Right now, using ActiveModel/ActiveSupport 3.0.0.beta4, Ripple from ripple/master and my fork of padrino-framework, I can create models and things work as expected.

The biggest headache for me is having to edit the Gemfile and append the versions for ripple and activesupport. I'm also working on adding gem version support to require_dependencies.

So you wanna try it yourself? Go right ahead. You'll need to grab my fork of the framework (I keep it up to date with master) and seancribbs/ripple. There are rake tasks to build and install those locally. I would HIGHLY suggest you create an RVM gemset. Additionally you'll need to grab ActiveSupport 3.0.0.beta4.

After that:

padrino-gen project test -d ripple
cd test
edit Gemfile and append version 0.7.1 to the ripple line.
bundle install
padrino-gen model person name:string email:string phone:string

You should now have a working model using riak via ripple. Here's a Gist of me doing exactly that via padrino-console:

Anyway, I'm intent on getting ripple support back into upstream but I'm not going to make a pull request until ActiveSupport reaches 3.0. It feels half baked to have these extra steps to get it working and I'd really like to have admin support done before I do if possible.

Parsing Nagios Objects in Ruby Redux

2010-06-10T01:36:00.006-04:00

When I was with MediaOcean, one of the projects I was working on was a stack of ruby code for parsing/writing nagios configs. This was in parallel with setting up puppet. I actually got pretty far along but then DDS had a RIF and I was back on the market.

I had to put the code aside and just recently pulled it out again. Some/most of it is pretty ugly. I had quite a bit done though.

So here I am "refactoring" (and I use that term VERY VERY loosely) the code base. So I'm happily parsing away - iterating over the cfg_file/cfg_dir entries and build a nice big hash to hold all the object definitions. I dump it to a YAML file for validation when I notice this nice bit of junk in my timeperiod dump:

Oh yeah...that looks right =/

For those who don't know, nagios object definitions are pretty straightforward. Here's an example:

Easy enough to parse, right? Object type named (host). A clear beginning and end denoted by curly braces. Object attributes in a seemingly key/value type markup. Some people smash the opening brace up against the object type definition but that's easily caught.

Iterating over the definition in Ruby is pretty straightforward (assuming a perfect example):

What becomes a problem is timeperiod defintions. The syntax for those is somewhat complicated. Looking back at the YAML fragment above, you can see why. For timeperiods, while the rightmost value is the actual time frame, the left part (usually a day of the week) can actually be a specific day. So if I wanted to create an entry for Christmas in the U.S., I would define it like so:


december 25 00:00-00:00

Things can be even MORE complicated if I wanted to handle non-date holidays:

monday 1 september      00:00-00:00     ; Labor Day (first Monday in September)
thursday -1 november    00:00-00:00     ; Thanksgiving (last Thursday in November)

Ugly, no? I searched high and low for the ability to split starting from the right and didn't find anything native. I had to resort to implementing my own rsplit method for String:

It works but I also can't use it across the board. If I do, I break things that were working (alias, service_description) that can have a space in the value.

I'm not even going to get into trying to convert timeperiod definitions into Date objects yet. That gives me cold sweats.

Aggressive LVM

2010-05-26T16:14:00.004-04:00

So I ran into an interesting issue that I hadn't seen in quite some time. I've got a server running KVM. I use the Virtio driver and logical volumes to hold the OS and Data disks for each VM.

Every time I used the lvm tools on the hypervisor, I was seeing random vgs, pvs and lvs. It took me a second to noodle out the problem.

LVM was picking up the guest volume groups and activating them. For instance, many of my VMs have a data volume. Those were getting picked up and activated OUTSIDE of the vm because of the LVM scanning filter.

I know at one point, the default filter only scanned hd.* and sd.* as valid block devices but it looks like the default filter is now:

filter = [ "a/.*/" ]

which makes everything in /dev a valid target for scanning and activating.

Being that this particular box is 1U with a single hardware raid array, I knew that the only valid block devices would be /dev/sd.

I changed the filter line to this:

filter = [ "a|/dev/sd.*|", "r/.*/" ]

A quick vgscan -vvvv showed me that my guest volumes were no longer valid targets:

#filters/filter-regex.c:172         /dev/ajc-dd-dmzvirt1/isolv: Skipping (regex)
#label/label.c:160       /dev/sda5: lvm2 label detected
#filters/filter-regex.c:172         /dev/ajc-dd-dmzvirt1/vdisk-itgweb1: Skipping (regex)

Well I touched one

2010-04-07T13:05:00.002-04:00

Yep. I just got an opportunity to play around with an iPad for about 10 minutes here at the office. (thanks @jpg)

One thing that several people were saying around the office is:

I'm surprised by how fast it is

That confused me. I would expect any tech product I buy to be "fast". I expect my laptop not to be slow. I expect my phone to be reasonably fast. Why would I NOT expect an iPad to be fast? I don't intentionally buy slow tech. Anyway, as someone who has never used an iPhone it was interesting to say the least.

It's heavy

Some of that can be chalked up to the case that Jason had on it but honestly, I really could NOT see myself using this for long periods of time. When I sit down to read, I read. It's not uncommon that Michelle and I would read for hours on end in bed. With her being pregnant, her Kindle has been awesome. She loves it. You could read a Kindle for 5 hours straight and not get tired holding it, nor would your eyes get strained. I can't say that about an iPad (not that I've tried to read on it for 5 hours). Yes the "pages" of the book are beautiful but it will still be a strain on me.

Yes, it's fast.

I wasn't "surprised" how fast it was until I opened Google Maps. It was really snappy. Maybe it's doing some insane caching locally. I have no idea but it was really nice. Safari was really fast as well.

I don't really like the iPhone "desktop"

Coming from my Droid, I really dislike having all those icons on the screen. It's cluttered and hard to find applications.

The Kindle app is really nice

This would be a possible selling point for me. I'm going to do a little research about how it integrates with an existing Kindle.

Netflix is slick

Enough said. This would be great for taking all the kid's shows that Gus likes to watch on the road.

I was wrong about exporting (partially)

You CAN export directly from Pages (I didn't give Numbers a go). It can export as Pages, PDF or Word (no clue on version).

UI transitions are slick and confusing.

Even Jason, who is pretty much integrated in the Apple ecosystem got confused for a minute. When Pages is in landscape mode, there's no UI other than a blank page and the keyboard. When you go to portrait mode, it puts the menu bar up top. This is pretty confusing and honestly, annoying. I would never want to type on the smaller portrait keyboard. That means to do certain operations, I have to swap from landscape to portrait. Several applications behave differently between portrait and landscape. Some of them worked "intuitively" like Mail and some didn't like Pages.

So did I like it? Yes, for the most part. I really would have liked to give the Elements application a shot. THAT is something of a killer app for me. I'm big on educational software right now because of Gus (and as of tomorrow, Nat). In fact, I'm really considering hitting the Apple Store on the way home to buy one but I probably won't.

Why?

No sideloading of content.

This is a BIG problem for me. I hate iTunes with a passion. Ignore for a moment that we keep ALL of our media on a server at home, the only reason I have a Windows machine is for gaming. My main desktop is a Linux machine as is my netbook. My wife has her laptop but it's hers. I've taken great pains to keep that gaming machine as optimized as possible. iTunes is a pain in the ass.

Limited format support/Lock-in

Okay so I could load MP3s on it but I would just rather use my Droid. So that's not an issue. What is an issue is videos. All of our tv shows are converted from the MythTV backend to MKV format and archived. Obviously the iPad doesn't support MKV only MP4. There won't be any loss in quality but what there will be is the pain in the ass of re-encoding everything to MP4. To be fair, I would only export a few shows ahead of time so it could be handled then.

App Store/iTunes Store

This is kind of a deal breaker. Let's assume for a moment that I go whole hog and start buying the kid's shows via iTunes. For those of you without toddlers, let me clarify something - they are obsessive. To a scary fault. When Gus gets it in his head that he wants to watch Thomas or Dinosaur Train or Elmo, that is ALL he wants to watch. So now I've got all these shows I bought from iTunes that he likes. The ONLY place he can watch them is on the iPad. I can't export them to my server and play them on my TV unless I buy an Apple TV.

Let me describe our video usage pattern at home. We record PBS shows via MythTV. We stream Netflix with the XBOX 360. Streaming is nice but isn't an "anywhere/anytime" option. So assuming I've bought shows via iTunes, I can't watch them anywhere with my current setup except the iPad or the Windows machine where I've grudgingly installed iTunes. I can't hook the iPad up to the TV and I can't export it so that I can.

So that removes the value of the iPad as a video mechanism unless I want to buy multiple copies of various content just so I can use it elsewhere.

So that relegates it to an eBook reader. This has promise but as I've said earlier, it's way too heavy for long term usage. The interactive benefits don't outweigh that for me.

So what am I left with? It's not a very multiple user friendly device. It doesn't support multiple accounts so I can't have both my social networking profile and my wife's on it. It's mine or her's not a shared resource. It's too expensive as just an eBook reader.

It's not very portable. Yes, it's more portable than a laptop but it's also much more limited in use. Streaming Netflix is nice but I'm not always in range of an AP. It doesn't have a GPS so using it as a physical map replacement is a no go.

I could wait for the 3G version but 3G is on the way out and I ALREADY have a data plan with Verizon for my Droid. Why would I ALSO pay money to AT&T? Sure we pay that cost with our Kindle but it's part of the purchase price.

So where does it really fit in? Educational use? Absolutely. As I said, something like the Elements application would be awesome for the kids but honestly, I don't feel comfortable giving it to my toddler to play with like I would a $50 LeapFrog toy. If he were older, yes but he's coordination isn't there yet.

So will I buy one? Nope. Not right now. I want to see what comes out in the same space with Android running on it. I want to get a device that has a longer lifespan. I also want something more open. I want to get something that I can EASILY move content to and from without any external requirements. I can pair my Droid with any bluetooth device in my house and export the songs I bought from Amazon. I don't have to fire up some application like iTunes to do it. I don't want to get locked in to having to be wrapped from head to toe in Apple gear just to get the flexibility I desire.

iPad confusion

2010-04-01T14:27:00.003-04:00

EDIT: I want to clarify that my original information was wrong. Pages (from my first hand experience) allows you to export as PDF,DOC or Pages format as well as email).

Color me confused about the iPad. I'm already grumpy because today is a terrible news day online. Every other story is either an iPad story or is a stupid joke.

My biggest problem is that there's this "hole" that people keep overlooking when they try to define WHERE the iPad fits:

You still have to have a computer running iTunes to actually USE the thing.

Yes, I mean use it.

All system updates have to happen via iTunes. Until it has OTA updating, you still need to have a computer that can run the bloated mess that is iTunes. So yeah, give it to grandma since she doesn't have a computer. It works great until there's a mandatory system update that prevents her from accessing the app store. You want to get stuff OFF the system? Pages documents? Spreadsheets? Gotta have iTunes to do the offloading and conversion.

Look, the iPad looks great. The fact that sites are already moving to HTML5 and open standards just for it bodes well for every one out there but make no mistake, until you can use it in entirety without needing iTunes, it's NOT going to be the savior of the technophobe.

So who's it for?

- It's not for someone who doesn't want a computer. You still need one for iTunes.

- It's definitely not for someone who likes to tinker

- It's too heavy, as even the most ardent fans have said in reviews, for long-term reading or handheld viewing.

- It's too expensive as a companion device

I'll give Apple credit for inventing but that doesn't mean it's a sure thing. There are some LEGITIMATE crossed wires as to where this thing really fits.

DevOps - Developers to Operations

2010-03-29T13:56:00.003-04:00

I apologize in advance since this post is likely to be leaner than the previous one. My background is NOT in development and I've never officially held a development role. Having said that, I'm looking to the comments for people to provide something else I "missed". I need a guest blogger for this section ;)

Production

I understand that production is important. I understand that it sucks when bugs happen but would it kill you to give me a little more insight into HOW things operate so I can do my job better?

I mean seriously. I wouldn't have to have to bug you all the time if you would just give me the tools to get the information I need. I'm not interested in your job. I'm not going to break something. Give me a limited account on the servers so I can look at the logs real time. Better yet, I heard of this nifty thing called Splunk. You think we can get the free version up and running somewhere so I can see aggregated logging?

Oh...you have to jump through 20 hoops to make that happen? Never mind, I'll just keep pestering you to email me the logs until the bug is fixed.

Change Control

Why the hell can't we push my bug fix out tonight? It's a minor fix and leaving it out there just means more data cleanup for you until we do fix it. Yes, it's already been through testing. Yes, there's some data cleanup but I've already written the SQL to do it. Look, the DBA even signed off on it.

Seriously, I'm handing you everything you need to get it done. Oh, you don't have the resources to do it tonight? How about cross-training one of us to do it for edge cases like these. I know that new features have to wait for major release but I've got the business unit breathing down my neck on this one bug. I guess I'll keep telling them we have to wait for you guys.

Bugs

I'm sorry but bugs will happen. Yes we write unit tests but those always have to be refined as new bugs are found. I don't enjoy having to revisit old code and fix the same thing over and over. I'm not intentionally putting these bugs out there for my health! If you would give us a proper preproduction environment we might catch these sooner. Yes, I know that the production dataset is much larger than our QA set. We asked for a refresh from production but you still haven't done it. We've got to get this stuff released now. No, I'm not being hyperbolic.

Look at it from my perspective. I'm trying to get requirements from 5 different groups of people. I need to understand the intricacies of revolving credit and interest payments. I have to find the BEST way to present a complex set of information in most appealing manner because the only thing these people care about is how fast can I get it done and how pretty is it?

I'm sorry my solution isn't the most optimal but it's only a stopgap. We're trying to work out a better long-term solution.

Attitude

Look, we're good developers. We have a process that works. The business units work with us because we don't let them down. We're extremely agile. We give them what we want and we're ready to generate some income but we're ALWAYS stymied by all your stupid roadblocks. We all have the same goal so why not let us help out?

You always treat us like a bunch of idiots. We can't POSSIBLY understand the complex and rich interconnectedness of your universe despite the fact that we wrote the code that actually RUNS that universe. As I said, I have no interest in being a system administrator. I just want to get my bugs fixed. I want to get my code released because I have five other projects waiting on this one to finish. It really ISN'T as complicated as you want to make it out to be. In the end, you're just copying files somewhere. I know we have 10 servers serving the application and I took that into account. See? Totally stateless application.

Oh, you're impressed that I understand statelessness? Your condescension is not surprising.

Cooperation

I know you think we're stupid because we don't understand the interactions of your fiber channel card and the SAN. I know you think we write buggy software to keep ourselves in a job. I know you have certain regulations and rules that you have to work with to maintain the integrity of the production environment.

I know all of these things but would it KILL you to work with us on this stuff? What do you need from us? Did we not give you enough information? Let us know and we'll make sure we have that next time. Can you give us some sort of production access to help troubleshoot issues? I hate having to ask you for everything just as much as you hate being asked. There really are some things we can do on our own. Just give us the guidelines (we're good at working with those, you know) and we'll offload some of that stuff for you. Don't worry we're not going to change anything, we just need to look.

DevOps - Operations to Developers

2010-03-29T12:35:00.003-04:00

This is part 2 in a general set of discussions on DevOps. Part 1 is here

Production

I have a general rule I've lived by that has served me well and it's NSFW. I learned it in these exact words from a manager many years ago:

"Don't f*** with production"

Production is sacrosanct. You can dick around with anything else but if it's critical to business operations, don't mess with it without good reason and without an audit trail. There is nothing more frustrating than trying to diagnose an outage because someone did something they THOUGHT was irrelevant (like a DNS change - I speak from experience) and causing a two hour outage of a critical system. It's even more frustrating when there's no audit trail of WHAT was done so that it can be undone. Meanwhile, you've got 20 different concerned parties calling you every five minutes asking "are we there yet?". How much development work would get done if you operated under the same interrupt driven environment?

Change Control

Yes, it's a hassle and boring and not very rockstar but it's not only critical but sometimes it's the law.

Side note: I pretty much hate meetings in general but they do serve a purpose. My main frustration is that meetings take away time where work could actually be getting done. They always devolve into a glorified gossip session. What should have taken 15 minutes to discuss ends up taking an hour as conversations that started while waiting for that last person to show up carry over into meeting proper. Sadly the person who is late is usually Red Leader and we can't seem to stay on target. Everyone has something they would rather be doing and usually it's something that will actually accomplish something rather than the stupid meeting.

The exception for me, has always been change control meetings. I typically enjoy those because that's when things happen. We're finally going to release your cool new feature into production that you've spent a month developing and fine tuning. Of course, this is when we find out that you neglected to mention that you needed firewall rules to go along with it. This is when we find out exactly what that new table is going to be used for and that we MIGHT want to put it in its own bufferpool. All of the things you didn't think of?We bring them to the surface in these meetings because these are pain points we've seen in the past. We think of these things.

Auditing

As mentioned in production, typically we don't have the benefit of looking over changes in source control. We can't check a physical object into SVN. Sure, there are amazing products like Puppet and Cfengine that make managing server configurations easier. We have applications that can track changes. We have applications that map our switch ports but it's simply not that easy for us to track down what changed.

Your application is encapsulated in that way. You know what changed, who changed it and (with appropriate comments) WHY it was changed.

Meanwhile a DNS change may have happened, a VLAN change, a DAS change...you name it. Production isn't just your application. It's all the moving parts underneath that power it. That application that you developed that is tested on a single server doesn't always account for the database being on a different machine or the firewall rules associated with it.

Yes, we'd love to have a preproduction environment that mimics production but that's not always an option. We have to have an audit trail. Things have to be repeatable. So no, we can't just change a line in a jsp for you to fix a bug that didn't get caught in testing. It would take us longer to do that on 10 servers than if we just pushed a new build.

Outages

Outages are bad, mmkay? You probably won't lose your job over a bug but I've had to deal with someone being fired because he didn't follow the process and caused an outage. It sucks but we're the one who gets the phone call at 2AM when something is amiss.

And even AFTER the outage, we have to fill out Root Cause Analysis reports sometimes after being up for 24 hours straight fixing a serious issue. You can either write a unit test for a bit of code or you can keep fixing the same bug after every release. We'd prefer you write the unit test, personally.

I know all of these things make us look like a slow, unmoving beast. I know you hate sitting in meeting after meeting explaining that the bug will be fixed just as soon as ops pushes the code. I know that we get pissy and blame you for everything that goes wrong with an application. We're sorry. We're just running on 2 hours of sleep in three days getting the new hardware installed for your application that someone thinks has to go online yesterday. Meanwhile, we're dealing with a full disk on this server and a flaky network connection on another. Cut us some slack.

DevOps and NoSQL - bad naming leads to confusion

2010-03-29T11:38:00.005-04:00

I've recently started following a few new topics (where recently means over the past year). Both of them have the potential to be paradigm shifts and, unfortunately, both have somewhat vague names that evoke responses on both sides of the issue.

The one I'm going to focus on right now is DevOps. I intend on doing another post on NoSQL but that all depends on how much free time I can finagle between setting up the nest for baby number 2 and work projects.

Background

I should clarify my background because that plays a large part in how I perceive both of these issues. I'm a systems engineer. No, I don't have a degree in engineering but I wouldn't call the work I've done over the years any less than that. I've been the intermediary between DBAs and Developers. I've spent 20+ hours on my feet in a frigid datacenter racking servers. I've done high-level architecture of disparate system integration. I've done low-level implementation of disparate system integration. I've been up at 4AM to do deploys of new code during the 30 minute maintenance window. I've been the guy getting the pages and been the guy calling people who we're supposed to get the pages.

I've been in big shops and small shops. I've been responsible for systems that pass millions of dollars and systems that are critical to education.

I don't say all this to toot my own horn. It's just background that is relevant to the discussion.

DevOps

So what's this DevOps thing that people keep throwing around? Well there are tons of opinions and all of them are like certain sphincter muscles. Not one is entirely on the money but the background work has been done here:

http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/ (which, coincidentally prompted this post)

So what is it? I think at the core it's about closely integrating the "SysOp" silo with the "Developer" silo as a methodology. But why is this important?

SysOps have always been apart from the rest of the IT department in a sense. While many groups have frequent overlapping areas, the operations team has the final responsibility. As I like to put it, they're the folks getting the phone call. Unless the organization is small, most developers aren't even in the loop unless a bug report is filed after an outage. As it was put elsewhere, many times software is thrown "over the wall" to be deployed. But why is this? I think that's key to the whole issue.

Roles, Responsibilities and Titles

I'm not a stickler for titles. I've held many over the years for Administrator to Director. In one interesting case, I was given a title (and the subsequent responsibility) simply for the purpose of interacting with a client who had firm opinions about only interfacing with someone at the same level. This didn't take away any responsibilities; only added to them. Titles, roles and responsibilities are all different things.

In this way, the organizational title for "IT Operations" denotes a clear differentiator from "Developer". There are certain expectations from your operations team. Production stays stable, for instance. Many times the goals of the Operations team are in direct opposition to those of the Development team. Make no mistake, however. The developers are part of revenue generation while those of operations are not. Operations exists as fire fighters. If Operations is doing its job properly, they aren't actually doing much of their primary responsibility. They have quite a bit of downtime.

So why is there a need for a DevOps movement?

I think on one hand, there is an increasing frustration from the end-user (in this case development) in its interaction with operations. Development methodologies are changing rapidly. Some changes are for the better (less bugs, more testing) while others create friction with how a production environment operates (frequent releases). Another aspect is people transitioning from one role to the other. You have people moving into development from an operations background and vice versa. People change. They discover that they enjoy X more than Y. With each of these transitions, a mindset and attitude is brought along. An Ego.

The developer who moves into production operations laments the slow sluggish pace at which things move. The operations guy who moves into development loves the fast and fluid nature of Agile development. Both feel the need to reconcile the two worlds thinking they can impart some sort of wisdom from one side of which the other was not aware.

Additionally, in times where the leanest team that is first to market often wins many people are wearing multiple hats. See the rise of IaaS (Infrastructure as a Service), Amazon Web Services, NoSQL and other technologies where traditional roles are eliminated.

Both sides have a lot to learn from each other and both sides need to understand the constraints each team has. This is where I feel DevOps has the most to offer as an ideal. Integrating operations into development and letting development be a part of operations. The specifics are still up in the air but I think there are some key areas that each side needs to understand about the other. I'll follow those up in the next post to for logical grouping purposes.

As always, comments are welcome!

Code on Github

2010-03-23T16:50:00.003-04:00

I'm working on pushing all my code snippets I've developed over the years to Github. Right now I have the Ruby-EWS and Ruby-Downtime stuff uploaded. I'll get the rest over as I can sanitize it.

http://github.com/lusis

Ummm...thank you Google Voice?

2010-03-23T12:37:00.003-04:00

I just got this wonderful bit of translation from Google Voice on a voicemail:

I just came across your resume on Monster. Here, and had a position available on the line area thought it'd be a good fit for you sexually a infrastructure design related positions in the offer an area love the chance to discuss with you at your earliest convenience

Mind you, I love infrastructure design but I'd hate to see how I could fit it into my sex life. The job WAS with Cox....

Usability and Performance testing in the Analog world

2010-03-09T09:08:00.004-05:00

Right now, our offices are located downtown. The AJC has been in this building since 1972 (I think). We're soon to be moving outside the Perimeter to new offices. Trust me, this is an important fact.

I started as a contractor here back in November of last year. I had an option for parking but at the time I didn't know how long I would be here (contracting can be volatile) and didn't know if I would plan on switching to Marta. Combined with the fact that the company WAS planning on moving offices, I decided not to get a parking pass.

Instead, I opted to use the pay parking lots behind the office. These spots are $3 a day or a monthly option of $40. This was about the same as the AJC parking pass so it was something of a wash. I went permanent with the paper in the beginning of February. Our move is scheduled at the beginning of April and parking passes are no longer available so I *HAVE* to use the parking vendor out back.

Of course this is when the parking company, Central Parking Systems decided to swap out the working payment terminals with new ones. The old ones, while a bit worn around the collar actually worked well. People understood them and things moved along quickly. I'm trying to find a picture of the old ones but the new ones look like the picture above.

Since these have been put in place, the lines to get a parking pass have been super long. I thought this was a familiarity issue. The new terminals are smaller, the screen is less readable and from a usability perspective, it's a pain in the ass. People have to stoop down to get the money in the machine and GOD FORBID you pay with a credit card. It's one of the quick swipe methods and from the angle you can NEVER "quickly remove your card" without also pulling up as you pull it out. This obviously isn't quick enough so you have to start again.

Well this morning I found out there's ANOTHER problem. There were two parking attendants watching people use the machines. Finally one guy spoke up and said the following:

"Folks, this terminal is running really slow and overloading the computer system. If you use X terminal or Y terminal, they're running really fast"

Let me position those other two terminals for you in relation to this one. X terminal is closest but by the time I get to it, pay and get back to my car (which I parked near THIS terminal I would already be done paying if I stayed in line.

The OTHER terminal would require me to get in my car, drive to it and come back to park near where I actually work.

So how slow were these new machines running? They were running so slow that they wouldn't actually take money. When I got up there, I decided to ask the guy how I would pay for a monthly option here. This is where it gets good:

"You can't buy monthly passes here anymore. Just go online to blahblahblahblah.com and register as a vendor. You get much cheaper parking and other rewards."

Are you kidding me? I'm only going to be here for another month. I'm not a "vendor" and I don't want to go to a stupid website to pay for freaking parking when I could pay just fine up until recently.

Where do I start with the screwups?

If they had ANY business intelligence or metrics, they would have known that this particular kiosk is the busiest one in the deck. It's near the best parking and it's centrally located. The fact that this one terminal was causing these kinds of issues is unacceptable. Load testing isn't strictly digital. Look at any major downtown event. The same work goes into capacity planning there from parking to traffic that any major website would undergo preparing for the Christmas holiday. The kiosk was running so slow that it was actually refusing to accept dollar bills. I decided to pay with my credit card and my ticket had printed while the screen still said processing payment.

But even IF the system could keep up with the load generated, having someone actually go onsite and see HOW users were using the damn things would have shown them the ergonomic bottlenecks.

Something as simple as a parking lot was foiled by lack of performance and usability testing. Amazing.

EDIT: I found a picture of what the old terminal looked like. It's not "pretty" but it worked and was ergonomically superior.

Finally getting to dig into Puppet

2010-03-05T09:45:00.004-05:00

Well I'm finally getting to dig into Puppet at a professional level. I've been handed the keys to establishing the Puppet infrastructure. Mixed Linux/Solaris clients. Beyond the hassle of building up-to-date RPMs for CentOS/RHEL, everything else is going smoothly.

I've got all of my configuration in version control and handy script to create new modules from templates. I'm already pushing out OS specific configurations as well as OS version specific stuff. Augeas is AMAZING. I don't know how I never found/saw it before.

We're also pushing out Splunk (Cox is a huge Splunk customer) across the board. I'm still undecided about its value but I haven't really looked at it since it first came out. It just seemed, at the time, like a huge overhead for a problem that was already tackled (syslogging).

My next step is deciding how to integrate everything into my Kickstart scripts. There's a TON of options out there. I'm really trying to avoid setting up Cobbler but I might be at the end of what can be done with Kickstart already.

I'm still frustrated with managing current versions of Ruby and gems with RPM but gem2rpm has made some of that easier. I'm not responsible for the Solaris boxen so I have no idea how those guys plan on managing those bad boys.

Increasing frustration with managing Ruby on RHEL/CentOS

2010-02-24T12:00:00.003-05:00

I am getting INCREASINGLY frustrated having to manage installations of Ruby on RHEL and its derivatives. It's to the point where I'm ready to shoot myself.

A bit of background:

RHEL5 and its offspring (referred from here on out as RHEL5) ship with a really dated version of Ruby (1.8.5). Yes there are various patches are backported but in the end, it's old. NOTHING works with it. No modern Gems work with it (for all intents and purposes). God forbid you try to run a modern Rails-based application on it.

The MAIN reason is one that plagues not only Ruby but Perl, PHP and Python on the same platform:

language-specific packaging repositories

With Perl, you have CPAN. With PHP, you have PECL/PEAR. With Python you have Eggs. All of these are somewhat antithetical to the OS package management system. Taking it even further, it's the same problem with Debian offshoots as well.

This isn't a problem for most end-users. They have the luxury of installing stuff wily-nilly. If it breaks, you just reinstall. However for those of tasked with maintaining systems in production environments, this is simply unacceptable. There is a requirement to keep systems in a consistent state. Software must be tested and work its way through the system before it gets to production.

OS updates are the same way. You don't simply have a cronjob that does a 'apt-get upgrade' or 'yum -y update'. Any smart company manages those packages using an internal repository. Packages aren't graduated to the production repository until they've been tested and verified not to break the system. A system MUST be maintainable. At one company, we actually managed our OWN application code via RPM. We had 3 repositories (dev, qa, production) and each class of server pointed to those repositories. As we built new RPMs, they were copied into each repository as needed for testing.

Now many people are probably thinking how convoluted that sounds but personally, I have no desire to have the financial future of my company at the hands of one broken upstream package. How would you feel if your banking site went offline because of the same issue? One broken upgrade can put someone out of a job.

So what happens if you say "screw it" and just use CPAN or rubygems or PECL outside of the system package management? Now what happens when you upgrade from something like PHP 5.2 to PHP 5.3? Now you have to rebuild any native extensions you have installed. Oh but look, the ImageMagick native extension requires that's compatible with version X of your interpreter also requires a new version of ImageMagick. But that version happens to be a major release instead of a point release. Let's hope nothing critical to the system requires a specific version of ImageMagick.

The specifics of the previous are somwhat hypothetical but everyone has run into the situation.

It's a difficult beast to tame. The closest I've come to having this process be somewhat simply is CPAN2RPM. However nothing that clean exists for Ruby. The best you can hope for is something like RVM (which in and of itself is pretty brilliant).

I still don't know the best way to manage Ruby on CentOS right now. There are countless repositories out there that are trying to do the same thing.

One thing we did at RBX which I've been doing currently is the concept of a "current" symlink. It's a bit convoluted but it works out for now and is pretty much the way that RVM works. Having this wrapped in a wrapper RPM keeps it managed via native package management but it's far from ideal.

Time to retire the beast

2010-02-12T23:52:00.003-05:00

So my primary server at home for the LAN is an aging dual P3 with 1GB of memory (it won't even support any beyond that).

Currently it has the following disk layout:

/dev/mapper/DataVolGroup01-DataVol
184G 87G 88G 50% /data01
/dev/mapper/DataVolGroup00-DataVol00
880G 834G 1.5G 100% /data00

My mythtv backend (a somewhat freshly built X2 Dual Core Processor 4400+) has this disk layout:

/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth

I finally got the new drives I ordered from Amazon back in November - 3x1TB Hitachi Deskstars.

Now the mythbackend looks like this:
/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth
/dev/sdc1 932G 4.2M 932G 1% /myth2

The 880GB on lansrv is actually an external SATA array connected via Multilane to the server (4x500GB in a RAID10). I've been wanting to retire that box for some time as it draws WAY too much power for the service it provides. The mythbackend is becoming the hub for storing our archived records, pictures and music on the network.

However I didn't really have an easy place to dump that data on lansrv temporarily. Now I do. Here's the tentative plan:

1- Migrate all the lansrv data over to new drive on mythbackend.
2- Pull two of the 500GB drives out of the external array and put in the new 1TB drives.
3 - Mirror the new 1TB drives
4 - Rebuild the remaining 500GB drives as a mirrored pair.
5 - Stripe the two mirrors together
6 - Figure out what to do with the spare 1TB and 2 500GB drives.

Most likely the 1TB drive remaining will end up as a hot-spare for the 1TB mirror. Hitachi's aren't called Deathstars for nothing. The other two 500GB drives will probably go somewhere internal to the mythbackend box and become a mirrored pair as well and added to the new LV.

If I can manage to fit an eSATA card with 2 ports in the machine, I might consider moving all 4 500GB drives into a new array and adding it to the volume group.

Who knows? I'll figure something out but for now I had to have somewhere to dump the data temporarily while I shifted hardware and data around.

As an interesting side note, the 4 500GB drives have been running in the multilane enclosure since I worked at CLA. That's at least 5 years. I don't even remember what KIND of drive they were but I'm damn sure going to make note of it when I pull them.

Apple predictions

2010-01-27T10:15:00.004-05:00

I wanted to get some quick predictions out for the Apple event today in the event that the device is a tablet:

- WiFi
- Bluetooth
- WhisperNet (like Kindle) via Verizon or possibly Clearwire/Sprint (and thus be carrier independent when LTE is rolled out across the board.)
- Wireless HDMI (aka WiDi)
- Wireless Charging

I don't think the device will have ANY external ports whatsoever. Depending on the range of WiDi, you can sit on your couch and broadcast a iTunes show to the TV while doing other stuff on the tablet itself. If you want, you leave the charging pad by the television or somewhere else.

Imagine you're riding on the train watching the latest episode of whatever. You get home and the device picks up where it left off broadcasting to your television.

I think social gaming a la Farmville will be there as well. Garage band on this thing would actually be pretty cool.

Anyway, some of these ideas are my own and others are things I've picked up from various podcasts but if I were building the next gen "device", those are the features I would pick.

Ugly queries

2010-01-10T23:16:00.004-05:00

I've never been a fan of ORMs in general. I love the concept. I think it's awesome but since the early days of hibernate, I've always had problems with automatically generated SQL statements. It's not as bad as it used to be but sometimes you get "gems" in the wild.

Today's gem comes courtesy of the Geokit plugin for Rails. Our newest application at AJC uses Geokit for finding properties within a 10-mile radius of the entered address. We've got a table with about 500k rows of geocoded metro Atlanta property data in MySQL.

Enter the query:


SELECT
*,
(ACOS(least(1,COS(0.593807786352424)
*COS(-1.47255067854929)
*COS(RADIANS(parcels.latitude))
*COS(RADIANS(parcels.longitude))+
COS(0.593807786352424)
*SIN(-1.47255067854929)
*COS(RADIANS(parcels.latitude))
*SIN(RADIANS(parcels.longitude))+
SIN(0.593807786352424)*SIN(RADIANS(parcels.latitude))))*3963.19)
AS distance FROM `parcels`   
WHERE (((parcels.latitude>32.5769814714641
AND parcels.latitude<35.4683785285359>-86.1150637195914
AND parcels.longitude<-82.6268142804086))
AND ((ACOS(least(1,COS(0.593807786352424)
*COS(-1.47255067854929)*COS(RADIANS(parcels.latitude))
*COS(RADIANS(parcels.longitude))+ COS(0.593807786352424)
*SIN(-1.47255067854929)
*COS(RADIANS(parcels.latitude))
*SIN(RADIANS(parcels.longitude))+
SIN(0.593807786352424)
*SIN(RADIANS(parcels.latitude))))*3963.19)<= 100))  
ORDER BY  distance asc 
LIMIT 21;

This bad boy is a DOOZY. It's got all the makings of exploiting everything in MySQL that can't be optimized - function results for comparison, ordering by function results and even a LIMIT thrown in for good measure.

And guess what? There's currently not much that can be done to optimize this query. Indexes won't help. The query cache is useless because A) the queries are unique to each source address and B) it'll just get pushed out anyway.

I'm looking around for some tips on this and the most likely solution is using spatial indexes in MySQL but that's a whole other issue.

Saving the newspaper industry

2009-11-24T23:17:00.004-05:00

I debated whether or not to post this but I figured what the hell. If somehow it gets read by my employer and causes issues, I'll cope.

I've recently become employed as a contractor with a large newspaper in the Atlanta area. I was and am really excited by the opportunity. The technologies are cool and it's a market I never thought I'd be in.

This particular paper has been hit pretty hard by the declining revenue in the industry because they are so large. They're currently in the process of moving out of the cornerstone building they've been in downtown since (I think) 1972. It's a gorgeous building and I would have love to been there during the hey day. These days the building is like 30% occupied or something absurd. One of the paths I take to get lunch takes me through the back past all the old printing and sorting equipment. It's very ghostly and I plan on getting some pictures before we move.

Anyway, I was talking to a coworker about a project they're working on. He needed some feedback about solutions from a system engineering side. During the conversation, we got off on a tangent about the company and recent history. It was legitimate discussion because the information helped me understand the purpose of the application and what it needed to do, how it was being accessed and such.

One thing he mentioned to me was that at the last publisher's meeting, they pointed out that digital was only 10% of the revenue and that they would be focusing more on how to increase print revenue.

I was flabbergasted. I mentioned this to my wife to gauge her thoughts and she came to the same conclusion I did.

Why would you scale back from the growing market to try and preserve the dying one?

Here's the deal. I'm pretty typical (I think) of the modern reader (and I LOVE to read).

Sidebar: I remember when this paper was actually TWO papers - one the conservative publication and the other the liberal one. Up until about 9 years ago, I had a subscription to the paper including Sunday. I got great joy out of sitting down and reading the newspaper. I still do.

Anyway, as I said I'd like to think I'm pretty representative of the transition in society. I'm 35 years old. I'm old enough to remember pre-internet and young enough to be hooked into the new stuff. Some of that might have something to do with IT but that's becoming less and less of a gap. Technology is now so pervasive and accessible that what used to be the domain of the geek is now the domain of the mom. Everyone, for the most part, has access to the Internet from any number of devices not the least of which is the cell phone.

So when I hear a statement like I heard above I have to wonder how anyone could make such a statement about trying to "life-support" the print side of things. In addition to the AMAZING post mortem from John Temple, former editor/president/publisher of the Rocky Mountain News. The market for that is slowly dying - literally. I'm not trying to be grim or hyperbolic. According to this, the largest percentage of newspaper subscribers is in the 65+ range. My age range only accounts for 17%.

What I'm trying to say is that once that subscription base starts passing on, things don't look to good. By the time I'm at that range, there simply won't be any subscription base left. From my perspective (and I'm going to be somewhat crass here), why would I buy a newspaper subscription when I can read the same news from 10 different sources on my cell phone in the morning while I'm taking a dump? I wouldn't. These days I find myself not even bothering to pick up a copy of the paper at the doctor's office or mechanic. I'm either tweeting, facebooking, posting on forums or reading RSS feeds of the aforementioned 10 different sources. I'm pretty damn efficient at it too. The generation below me is even BETTER at it. They'll be doing things when I'm 64 (cue music) I won't understand and I'll be bitching about the good old days of the tubes and the propensity of children to walk across my grass.

Newspapers cannot compete on news. At least as far as my "outside" eyes can see. News is a commodity in a sense. Word travels fast these days and it's only going to get faster. From the perspective of a print paper, the news is already old as soon as it happens in this modern age. 500 people have already Twittered, iReported, blogged , Facebooked, Youtubed and done interviews with newspapers overseas before it's been sent to the presses. Using any random bit of aggregation services and software all of that information has been compiled and presented to me in a way that I wish to consume it*. If it hasn't been, I can easily do so with a single Google search when I want to know more. I can easily participate with a hashtag, url shortening service and a cell phone if I want. I become a part of it. I'm a member of the community around that bit of information.

So what can be done? Where should newspapers go? I thought about this a lot on the way home from work tonight. I had a plan all devised in my head. It was all so clear. Too clear in fact. Obviously someone else has to have seen this as clear as I did. Maybe I got some of the ideas from John Temple's speech. I wasn't sure.

When I finally got downstairs in front of my computer, I did a quick search (on Google of course). I pulled up the copy of the Temple speech. I also searched for one phrase - "How to save the newspapers". The fourth result down** was this article on truliablog.com. There were all of my grand ideas laid bare and done so almost a year ago. This almost entirely sums up what I had in mind and plan on repeating here in condensed form.

So what was my idea that obviously isn't just my idea?

Community. The Eastern Standard Tribe kind.

Back "in the day", some of us geeks were identified by the web communities of which we were a part. We're you a Slashdotter or did you read Kuro5hin? Maybe you spent your time digging and burying. If tech wasn't your thing, were you a Deviant? These days it's much of the same but the communities are even more diverse. People like, I think, having a sense of belonging and it's much more if not entirely acceptable to belong to a community where you've probably never met half the people. You're known by an avatar and a nickname. I have people I consider friends who call me by my character name in World of Warcraft and yet we all know each other's real names and personal information just as much as some of my friends here in town. I went to parties years ago with people I only knew from IRC and was called by my nickname the entire night. I addressed others the same way.

Newspapers can establish that kind of community in the modern age. When there were two separate newspapers in town, you might have an idea about a person based on which one they read. These days, I find myself building relationships and communities with people around the world based on the most niche of ideas - a mailing list for an open source project, a coding language or a forum dedicated to an author I might like. If the author is reasonably current, I might even be able to follow them on Twitter or become a fan on Facebook. It's no less of a community than the one I actually live in which has an equally vivid online presence.

So my plan was for the newspapers to be breeding grounds for those communities. Create the tools. Forums. Aggregation. Whatever is appropriate for making it easy for people to build those communities. They will still be providing the same service they were before which, at the core, is delivery of information. Ebay is nice but Craiglist is better when I want to go local. Building local communities online can even help spur the local economy which can always use help regardless of where "local" is.

Take advantage of the personal capital you have built up in your existing personalities. If you have a reporter that people trust, that person should be right in the thick of the community. Clark Howard is someone who has an amazing community of like-minded people from everywhere that he's broadcast but I would wager he's loved nowhere else in the world more than he is right here in Atlanta where we've been listening to him locally for years. Modern news is interactive. It's no longer a matter of writing an editorial and walking away from it. You'll have to be accountable for what you say because if you aren't, you'll lose trust from the community and, as in any relationship, you may never be able to earn that trust back.

Be open. This somewhat ties to the previous statement but I'm thinking more along the lines of "open to ideas". I'm amazed by how much CNN has integrated iReport into the main "product". After the President's speech to Congress months back, CNN had two people on who represented each side of the issue. The amazing part is that those two people were from the community CNN had provided and weren't talking heads. They were real people that best represented the viewpoints of their community at CNN. Mind you I don't think the person representing one of the sides was particularly articulate but he wasn't a pundit or a news guy so I can't be too critical. Make the community part of the product. When someone feels a sense of ownership, it becomes more personal.

Don't be afraid of Google. Don't buy into the paywall tripe being spread by some business men. Don't cut off your nose to spite your face and all that. While your focus may be the traditional concept of local, there are expats all over the world who would belong in that "local" community. Those people might even be your greatest source of revenue. I can see me living in Michigan years down the road and wishing I could get an Frosted Orange. You get the idea. I still need to know about the community and if Google doesn't know about it, I won't know about it.

Commit to the community. Don't be halfway about it. If you're going to do it, do it. Nothing is more frustrating that getting involved with something (an online game or whatever) only to see it die on the vine because of lack of support from the people who put it out there in the first place. While people will gladly contribute more than you can handle, they'll need some nudging to maintain focus. If you have a sub-community built around Foodies in Atlanta, maybe you should have some prominent Foodies actually be involved in some officially recognized capacity. Moderators are a valuable thing. A forum is only as good as its signal to noise ratio. If appointing community managers is out of the question, at least provide the community with the means to self-police. Don't discount Whuffie in self-moderating environments.

Accept organic. While the communities will need some nudging to help get started, realize that they WILL take on a life of their own. They may grow beyond what you might originally envision. They'll merge. They'll split. The Atlanta Baklava community might realize that while anyone who DOESN'T like baklava is obviously deficient in some capacity, maybe they're better suited in merging with the Atlanta Pastry community as a whole.

So where does the news come out of all this? Where's the news in this modern newspaper? The communities are built FROM the news. You'll still have your traditional reporting "model" but the communities will spring up out of that news. I started following a few new people on Twitter after the #atlflood event and every single picture I saw of the flood came from either Flickr, Twitpic or *ding-ding* ajc.com. There were people watching the #atlflood hashtag who had the same experience I did. I would hope that the newpaper was ready to capitalize on that.

I don't claim to have all the answers. I've only been on the job for two weeks now. The thing is that I REALLY like the job. I enjoy the people I'm working with and I want the company to succeed. Maybe that's what's needed. Less old-school baggage and a fresh perspective from the outside. It's also more selfish than that. I would love to be at the ground level of helping to engineer something as amazing as bringing the newspapers into the modern age.

* Think Manfred Macx in Accelerando
** The first result was from Time magazine. The second was from another newspaper. The third was an article on how to save newspaper clippings.

Exchange Web Services via Ruby

2009-11-20T22:36:00.003-05:00

I realized as I went to post an update on this that I don't think I ever posted an actual blog entry about it.

While I was on contract at HSWI, the email was hosted on Exchange. The development team was all on Mac workstations but the front-side was all on Windows. Since I was running Linux full-time, Exchange access wasn't a big deal. I ran Mutt and used LDAP/IMAPS/SSMTP. Digging around the Mail.app on the OS X side however, I realized that the client side was using Exchange Web Services for much of the functionality.

Wanting to do some poking around with Ruby and SOAP, I figured it would be a fun exercise to talk to the Exchange server with Ruby. I also had an itch to scratch thinking I could use it as an address book source for Mutt. I got it working in a day or so after dealing with some broken functionality in either the WSDL from the Exchange side or how SOAP4R tried to translate it.

You can find the code here.

Anyway, I'm no longer with HSWI but over at the AJC, we're also using Exchange in a much greater capacity. Anyway, so I whipped out the "old" code and was depressed to find out it didn't work.

Every attempt gave me a 401 error. It made no sense since I could access OWA, ActiveSync with my DROID (awww yeah) and even access the EWS wsdls on the server.

As I started to poke around online, I started to realize that in some of the more complex configurations, the Exchange server is hidden behind an ISA server or something. I don't do the Microsoft world much anymore and I have no real access to the environment.

What I noticed though, was that the internal IP is different than the external one. That gave me an idea to make sure and test the code externally from home while NOT being on the VPN.

It worked!

As I poked around with THAT information, I remembered an interesting thing that happened the one time I logged. on to a Windows machine at the office. If I fired up I.E. and went to the OWA url, I was logged in directly.

What I'm guessing is that the Exchange server has a different set of criteria for authenticating internally than externally. I'm not up to date on current Exchange and AD implementations so I have no idea what the configuration is that's preventing me from using EWS but still allowing me to use OWA from Firefox under Linux and OS X.

If any MS guys out there have any idea what's causing this, I'd be interested to know so I can either work around it or document it appropriately.

I have a *FEELING* that it's somehow related to NTLM but I don't know how to force my EWS call to bypass it just yet.

I am seriously having too much fun with conky

2009-09-15T23:44:00.005-04:00

Colors and conditionals, oh my!
Notice how the Network and Music sections change. The possibilities are endless. I was having fun with scrolling marquees earlier.

Secure IMAP conky script

2009-09-15T16:13:00.004-04:00

So I've been using Conky again. I decided that I wanted to add a mailcheck to the display.

Of course conky doesn't support imaps (yet) and I didn't want to dick around with stunnel so I hacked up this quick ruby script to do it for me. I spent most of the time trying to figure out why it wouldn't display properly in Conky only to FINALLY stumble onto

text_buffer_size 2048

Here's the link:
http://dev.lusis.org/conky-imaps/

New desktop layout

2009-09-12T23:20:00.004-04:00

So I was messing with Crunchbang Linux on my Acer Aspire One. I really liked the darker colors and the Conky config. I was also enjoying how simply and quick everything was. What really got me was that I found quite a few new apps that I didn't know existed (for instance Gwibber).

I decided to try and do something similar with a little bit more flare on the desktop side. These screenshots are the result. The first is with terminator up. The sceond is just the desktop.

The party that cried wolf

2009-09-10T10:39:00.000-04:00

Disclaimer: I'm not a Republican. I'm not a Democrat. I'm registered as a Libertarian but the following could easily be applied to any party in power including my own.

There's been quite a bit of fallout with regards to the hoohah over the President's speech to school children this week. Much of it will be overshadowed by the President's speech to Congress last night but I think the school address issue is symptomatic of a bigger issue - the Republican party has become the boy who cried wolf.

From cries of socialism to "death panels", the Republican's have given voice to the conspiracy theorists of the uber-right. The people who see threat at every turn. The kind of people that end up on the "news of the weird" segments. Some of this characterization is unfair. I'm just as distrustful of the government as the next person. I think it's our job as citizens to question the motives and actions of our government. I think it's our right to dissent but these types of statements do no favor to opposition to the current environment.

Socialism
This is the one that bugs me the most. Ever since Barack Obama became president, there have been cries of the United States becoming a socialist government. First off, there is simply no way short of a rewrite of the Constitution that we could ever become a truly Socialist form of government. Can Congress pass laws that are socialist in nature? Sure but we also have a separation of powers. Any one of those laws could and can be challenged in the court system to be overturned. "Socialism" has become a boogyman word. Just like the boy who cried wolf, eventually the word looses its power and no one listens. I firmly believe that there are members of our Congress and even our own President would love to move control of much of our economy into the hands of the government but that doesn't mean that we're becoming a socialist country or that the Constitution is immediately invalid. What SHOULD be focused on is the overreaching interpretation of the Commerce Clause that Congress likes to abuse and, as stated during the Sotomayor hearings, depends on to accomplish its goals - both sides.

Healthcare debate
Up until the school speech fiasco, this was the biggest "cry wolf" to date. It started before there were any bills even up for vote but most of the kerflufle was related to HR3200. I'm not going to recount everything on FactCheck. The work has already been done. The point to make is that instead of focusing on what the bill actually says, the wolf-criers are extrapolating specific provisions with conditional language into, in some cases, wild flights of fancy and worst case scenarios. See the following examples:

Government deciding treatments
The actual language of the bill simply sets up a private-public advisory committe that makes recommendations on minimum benefits. Somehow this got extrapolated into the comittee actively deciding what treatment each person would get. Nothing in the bill actually gives this panel that power. All they would do would make recommendations as what minimum coverage should be allowed. First off, this is no different than current insurance companies. Secondly, while one could argue that any recommendation that this panel makes would most likely end up as the baseline coverage, that's still not the same thing as the government deciding your treatments in any given situation.

Health care rationing
Nothing in the bill actually says anything about rationing. Again, the wolf-criers are taking things to extremes. They're mixing up what is happening in other countries with language in the bill stating out of pocket expenses will be capped for individuals and families. While it's fair to consider how healthcare is performing in other countries and what changes have been made in those plans, they're not indicitive of anything specific in HR3200 or the concept of government-run healthcare in general.

Those are just a few examples. The current cry-wolf rhetoric on healthcare is a mixture of misinterpretations, worst-case scenarios and comparisons to other countries. By giving voice to these types of statements, the actual issue is being buried under easily debunked conspiracy theories. Instead, the discussion should be on the cost of implementing the bill (both short and long term) and if it's even a valid role of the federal government.

School speech
So we come to the school speech. Anyone with half a brain knew that this would end poorly. Neal Boortz summed it up pretty well but I wanted to mention it. It became the penultimate example of crying wolf. From the minute it was announced, conspiracy theorists were on top of it. There were actually TWO parts to the speech but they ended up getting lumped together into one big "OMFG". The first part was a set of recommended lesson plans sent out by the Education Department to schools in preparation. Most of it was innocuous in nature. There were a few parts that caught my attention but the biggest one was the line (paraphrasing) "Discuss with students why it's important that we listen to elected leaders like the President, congress, mayors.....". Depending on the interpretation of "important that we listen", one could assume that the lesson plans are stating that we should just do what they say. It's the difference between me telling my son "you need to listen to me right now (i.e. do what I say)" vs. "listen closely to what the President is saying (i.e. pay attention because it's important to understand what politicians are scheming). I can agree with the second one but not the first.

Then there's the speech itself. Did anyone HONESTLY think that he would use the speech as an "indoctrination" speech? Seriously? Our President is arrogant but not stupid. Even IF he had originally planned to do that, instead of giving him enough rope to hang himself the wolf-criers gave him plenty of time to rewrite it and make them look stupid. Which he did.

And while we're on the subject of "indoctrination". If you send your children to a public (I know Boortz likes "government") school, you are conceeding that they will be taught by the government. How hard is that to understand? If you don't like it, homeschool or send them to a private school. Yes, it's totally unfair that you still have to pay taxes to support those schools when you don't have children there but that's a whole other issue.

Use of the word radical
While I'm on the issue of words, I'm also pretty fed up with constant usage of the word "radical" to describe the policies of the current administration. It's bass-ackwards.

rad*i*cal:
2. Departing markedly from the usual or customary; extreme: radical opinions on education.
3. Favoring or effecting fundamental or revolutionary changes in current practices, conditions, or institutions: radical political views.

If you look at the policies of this administration in light of the rest of the world, technically our EXISTING system is radical. The administration's policies are more the norm for the rest of the world.

That's really all I have to say on the issue. True political discourse has been given over to the extremes on both sides. It's sad for those of us who want to address the real issues.

check_rdp request

2009-09-02T14:00:00.002-04:00

So I got a reply to my tweet about a free nagios plugin. I was excited at first until I started delving into the whole thing.

The request came from @cixelsyd:

@lusis random #nagios plugin suggestion: check_rdp verifies port, handshake, auth

Okay, I thought. Sounds interesting. Let's give it a shot.

So I did the first thing which was to see if I could find an existing module for any of the scripting languages I know. It was a long shot and it came up empty.

Not a biggie. Let's see what we can find out about the RDP protocol. Maybe I can knock something together....

About 2 hours later I was done reading various Technet entries. RDP is pretty convoluted and only gotten more so as Microsoft iterates over the various versions.

So I decide to find some code I could attempt to read through. Of course rdesktop was my destination.

After spending the last hour or so navigating the rdesktop source, I'm not quite sure if it's possible to even do a headless RDP client. Mind you my C is very limited.

My first attempt was to simply shortcircuit the rdesktop client after handling the authentication. Each attempt kept leading me to various X-related code. That's a whole other beast that I'm just not remotely competent enough to learn. I compiled it with debugging and used a Vista machine at home as the test server. Each and everytime, it wanted to do some sort of screen rendering.

I'm going to spend some more time on it tonight including reading the source for a few other client implementations. Unfortunately, rdesktop is the only one I know of that supports RDP5. At least one that I have access to the source code for.

My thinking is that, if I can't totally remove the need for client rendering capabilities, that I can somehow fool rdesktop into using a null framebuffer of some kind or faking the capabilities of the client side display. All I need to do is authenticate but I can't really tell if the client window has to be available even if all the credentials are present in the PDU since there's a Basic Settings Exchange before the Security state is even reached.

Another option that I can't test at work but can at home, is to see how the rdp2vnc code works. It might be possible to bring that session up only to tear it down.

The really annoying part is that this would still not be a very efficient plugin unless I proceeded to somehow implement JUST the process up to the Security Exchange.

Links:
http://msdn.microsoft.com/en-us/library/dd644611%28PROT.10%29.aspx
http://msdn.microsoft.com/en-us/library/cc240445%28PROT.10%29.aspx
http://rdesktop.org/

I has a sad.

2009-08-28T21:00:00.003-04:00

Yes, the title is stupid lolcat speak but I think it sums up how I felt this afternoon on the way home.

For those who don't live in Atlanta there's a local radio station that for the past 9 years has been donating airtime and personalities (as well as large amounts of cash from the radio personalities themselves) to a local non-profit. That non-profit is the Aflac Cancer and Blood Disorders Service at Children's Healthcare of Atlanta. CHOA is a worthy non-profit in its own right but the Aflac Cancer Center is a special kind of place. The goal is to get to a 100% cure rate. They're at 80% or so right now.

So every year for two days, 750AM WSB Radio in Atlanta has this Care-a-thon to help raise money. I think last year they raised over 1 million dollars and I haven't heard the final numbers for this year.

Michelle and I never gave to the Care-a-thon until we got pregnant with Gus. At that point, we figured it was "good karma" and considered it jokingly like an MSA should, god forbid, we ever need it. Luckily we haven't but we have taken Gus to the emergency room at one of the CHOA facilities before. These people love children. They know kids. They know how to take care of kids. They are simply wonderful.

Anyway, during this Care-a-thon they often interview the various staff members at the Center. A doctor here. A nurse there. The guy who runs the place. This year they interviewed a woman who's title I can't remember but her job was essentially to partner with the children as they went through chemo and other procedures at the hospital. She helped them understand what each procedure was going to do, what their cancer was...pretty much everything during the time the child was at the Center.

Today, however she read a letter she got from a parent of a child who sadly passed away in July. That's when I realized that not all of her work was the good. I hadn't really thought about the fact that there's this 10% that don't survive. She talked about how she had to tell this girl that the chemo wasn't working and that she was going to die.

I sort of froze for a moment. I couldn't stay that way for too long because I was sitting in traffic on the way home. Then my mind started to wander. This has happened entirely too much for both Michelle and I since Gus was born. I understand it's fairly normal for new parents. Most of these wandering are absurd situations that are 1 in 1 million of things to happen to you kind of things.

I'm going to tell you how my mind wandered because in the sadness of my mind's eye, I came to realize exactly how much I loved my son.

In my mind, I am with my son as he is now - 13 months old. We've gotten new that there's nothing else that can be done for him and he's going to die.

That got me choked up. I mean literally I choked and coughed back the tears.

I have no idea what a child of my son's age would have to go through for cancer treatment but I imagined him as he is now. Playful. Happy. I imagined those last minutes I would have with him.

Would he be playing and get tired and simply lay down? Would he be in pain? What could they do for him?

For me, those last minutes went as I remember the best times with him right now. Sleeping on my shoulder in the glider in his room. I love the those times. I'd like to think he likes them too. It's our thing. Our bonding.

In my mind Gus was sleeping with his head on my shoulder as he does every night, I can hear him with his little snore. I'm rocking in the glider and he's sleeping. Then the snoring just stops. No more breathing and it's over. I keep rocking him and I'm sad but at least we had the time we did together.

Now mind you at this point I'm trying desperately to NOT run off the road from the tears in my eyes. I'm about to vomit from trying to hold this all in. I'm cursing the traffic. All I want to do is get home to my wife and son and hold them both.

When I finally get home, I step around the corner to the kitchen and Gus sees me. He has a huge grin and comes running to me as only a 13 month old who's been walking solidly for all of a month and half can. It's more of a waddle.

He throws his arms up in the air begging me to pick him up. I do.

He lays his head on my shoulder and grips my neck in the way that he does when he wants to love on me.

I burst into tears. Michelle bursts into tears. Gus senses the mood and continues to hug me. He doesn't cry. He occasionally pulls back to look at me and make sure I'm okay. We stay like this - all three of us - for 5 minutes. I tell my story to Michelle a bit later and we have another good cry.

We talked about it tonight after Gus went down. It was cathartic. It felt good to get out all of these unspoken fears about our son. About his future. About what we would do in the gravest of situations. I think it broke through a mental wall that we both had with these "terrors" about something happening to him.

We both love our son so much. In the moment that I was hugging him so tightly, I couldn't feel anymore love than I did for him and my wife right then.

That's love, folks.

Free nagios plugins! Act now!

2009-08-27T10:30:00.002-04:00

Okay that's a silly title but that's what I'm doing.

I need to keep my skills up. I'm taking "commissions" for any Nagios plugins people might want written. I'll try and keep the dependencies to a minimum. Language will be anything from Perl to Ruby or possible a Bash script.

What I need from you:
- What service you want monitored.
- What information from the service you want monitored.
- If it's a commercial application, I'll need a trial version that works on a platform I have access to.

So if you want a plugin for monitoring MyCommericalERP package that only runs on AIX, I'm probably not going to be able to do it. I have an old RS6000 here and that's about it. I would gladly let you give me a Power system though if you wanted ;)

However if you use MyOpensourceERP that runs on Linux then there's a good chance I can come up with something.

Nagios Downtime Scheduler in Ruby

2009-08-27T10:16:00.004-04:00

It's been a while since I worked with anything in Ruby. I was getting into the groove when the whole DDS/MO layoffs happened. Since I got right back on my feet, I really didn't have time to dedicate any time to it.

So now 6+ months later, I had an opportunity to get back in the groove. It didn't go as smooth as I would have liked. I had to look up stuff that I knew by heart previously. That's just like any skill though. It atrophies with disuse.

In this case, none of the recuring downtime schedulers for Nagios were really cutting it. One of them had you add the downtime directly to the script. Others didn't work with Nagios 3. After a night of false alarms that escalated to my boss' boss, I decided to fix the damn problem.

So here's the first iteration of my ruby downtime scheduling script for Nagios 3. It's very rough. There's fuckall for input validation, for instance. It doesn't support anything more than daily recurrance. I'm going to be cleaning it up but I'm pretty happy with how it's shaping up. It has a setup mode and you can interactively add downtimes (as long as you don't fuck up the input - heh)

http://dev.lusis.org/nagios/downtime/

It's not yet approved on exchange.nagios.org. I'm looking for any feedback or feature requests. I want to keep it at its core a downtime scheduler and nothing more but I'd like to revisit my ruby Nagios config parser for the validation.

Like I said, it's been a while since I've done anything in Ruby so I know there are some major 'wtf' moments in there. Be gentle.

Nagios and Websphere

2009-06-25T09:19:00.002-04:00

I saw something interesting pop up on twitter:

http://code.google.com/p/nagios-was/

that made me think of my websphere monitoring work:

http://bit.ly/I6YUw

Admittedly, mine isn't actually a Nagios plugin but I could pretty easily convert it to one. The logic is all there. Maybe I'll download a trial version of WAS6 (mine was written again V5) and see if I can make it an actual plugin.

More Adobe AIR stuff

2009-06-25T09:03:00.003-04:00

So I recently wiped my Dell XPS M1710 laptop. I had purchased another 2GB of memory (bringing it to 4GB) and a 500GB 2.5" sata drive to replace the shitty 100GB that came with it.

I decided to go 64-bit Jaunty to really take full advantage of the memory. I haven't run a 64-bit desktop Linux since I ran Gentoo. I remember a whole host of problems then with chroot and such but figured things were better now.

Indeed they are except for a few closed source programs. One of those happens to be Adobe AIR.

Adobe provides some nice tips on their website for 64-bit linux installs but most of those were deprecated for Jaunty. TweetDeck would fire up but not let me click anything similar to the problem I had before with gnome-keyring vs. kwallet.

A little research later lead me to this post:

http://joefleming.net/2009/06/14/ubuntu-64-bit-and-adobe-air/

Essentially, getlibs makes all the problems I had before a non-starter. I was able to grab the 32-bit version of gnome-keyring and have it be a part of the package system (so no stray files) and start TweetDeck.

I'm still getting some cocked-up error about libcanberra but that appears to be a known issue:

https://bugs.launchpad.net/ubuntu/+source/ia32-libs/+bug/369498

Updates to mhs

2009-06-12T16:52:00.002-04:00

So I did some updates to MHS (the mysql health daemon I wrote for work). I was implementing our scripting framework from bash in perl to provide the same wrapper functionality we have in our shell scripts.

It's not as transparent but it logs script stop and start. This one is pretty useless outside of our shop because it's now implementing our own perl modules:

HSWI::Base
HSWI::Log
HSWI::DB

I also stopped using Proc::Daemon because of some logging issues I was seeing. No matter which order I did Proc::Daemon::Init(), logging would stop. I even changed Log::Log4perl to use init_and_watch but nothing worked. I ended up having to not only change to App::Daemon just to get the tty detach (we start mhs from an init script) but also stop using File::Dispatch::LogRotate because it doesn't honor setting ownership of log files.

As soon as I have time to sanitize it, I'll post a new copy on dev.lusis.org.

Tinis for Tatas

2009-06-08T11:30:00.002-04:00

I wanted to remind everyone who might possibly read this and lives in Atlanta that Crepe Revolution is hosting Tinis for Tatas tomorrow night:

http://creperevolution.wordpress.com/2009/05/03/tinis-for-tatas/

This is a pretty big deal as one of our dinner club friends is a survivor as well as one of my guild mates in World of Warcraft. If you're in the Atlanta area, please try and come.

MyISAM vs. InnoDB via Twitter

2009-06-04T08:35:00.003-04:00

So I saw the following tweet come across TweetDeck (using the search functionality as a custom feed no less):

ok #mysql / #database geeks on twitter (twittabase tweeks?) .. which is better: myisam or innodb, and where and why.

from @narendranag

140 characters is nowhere near enough space to answer that question. I'm going to put my thoughts here from the perspective of someone who's had to deal with large databases (500+ GB) in both MySQL and PostgreSQL doing both OLTP and OLAP work. Here's probably the best summation:

MyISAM
- Fast

InnoDB
- Reliable

That's not to say that MyISAM can't be reliable or that InnoDB can't be fast but that's the best way to look at it. But you can't balance which table type you choose based on those two criteria. There are reasons you might not need full ACID compliance that would make you still want to use InnoDB over MyISAM. Hell, MEMORY is faster than MyISAM but you don't want to use it for all your tables. Often times, the right answer is somewhere in the middle.

I tend to default to using InnoDB across the board. It's always defined as my default table type in my.cnf. I do this because often times the developers are coding around the assumption that the database supports things like foreign keys, transactions and the like. Admittedly, this is often hidden behind the ORM they use such as Hibernate but it's still important.

But what about speed? Why is InnoDB "slower" than MyISAM. It's basically because it's doing "more" than MyISAM. It's managing key constraints, logging transactions and all the other things that make it ACID compliant. Much of that "slowness", however, can be mitigated by getting away from the default InnoDB configurations and profiling the system over time to size bufferpools and the like:

Don't use a single tablespace file for InnoDB (innodb_file_per_table). The global tablespace will still need to be used but it's much slower than if you were to use a tablespace per table. This also gets you the benefit of being able to recover disk space after dropping an InnoDB table. If InnoDB is using the global tablespace file, the ONLY way to recover that space from dropping an InnoDB table or whole schema of InnoDB tables is to fully backup ALL schemas, remove the ibdata/log files, restart the database and then reload from backup. Not pretty.
Use the smallest possible primary key where you can. At one company, we were using GUIDs for the primary key. InnoDB prefaces every index with a subset of the first bytes of the Primary Key for that record. I can't remember the exact number off-hand but it was only a subset. Considering the first X bytes it was using could potentially be the same across multiple records, it took more work than if we had used ints. Additionally, not only were our primary key indexes unneccesarilly large, every subsequent index was as well. This wasn't as much a big deal on columns with smaller datatypes but indexes on columns of datatypes like varchar and lob were pretty ugly
Consider using a larger log file size. This has a trade-off in recovery time though. Your call there.
Get fancy with disks. If you have multiple volumes of different raid types, you can not only tell InnoDB to put the global tablespace and log files on a different path but you can also "move" the database to a different volume as well. This involves creating the database, shutting mysql down, moving the database directory from the mysql data directory to a new location and then symlinking it. Until MySQL or InnoDB gets me the ability to define where I want a given tablespace, this is the next best thing.

One area where InnoDB is faster than MyISAM natively is in concurrent CRUD operations. That's because InnoDB uses row-level locking. I'm not as clear on the specifics of the locking as I am with say DB2 (DB2 will actually lock the row before and after the one you're modifying) but it's better on multiple concurrent operations than table level locking.

So when would you want to use MyISAM then? One area we really found that using MyISAM made sense was on non-relational tables within a schema that normally had InnoDB tables. In one case, we had a table that had two columns - an id and a blob. That table was MyISAM. Conceivably anywhere you are denormalizing the data quite a bit, it can make sense to use a MyISAM table especially if it's a frequently rebuilt table (like a summary table updated from a cron job). Of course we've also used MEMORY tables for that same purpose. Just be careful how you intermix these in the code (the aforementioned Java transactions for instance).

So here's my recommendation:
OLTP tables - InnoDB with a few caveats
OLAP tables - MyISAM with a few caveats