On Noah - Part 3

This is the third part in a series on Noah. Part 1 and Part 2 are available as well

In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. This post is discussing the technology stack under Noah and the reasoning for it.

A little back story

I've told a few people this but my original intention was to use Noah as a way to learn Erlang. However this did not work out. I needed to get a proof of concept out much quicker than the ramp up time it would take to learn me some Erlang. I had this grandiose idea to slap mnesia, riak_core and webmachine into a tasty ball of Zookeeper clonage.

I am not a developer by trade. I don't have any formal education in computer science (or anything for that matter). The reason I mention this is to say that programming is hard work for me. This has two side effects:

• It takes me considerably longer than a working developer to code what's in my head
• I can only really learn a new language when I have an itch to scratch. A real world problem to model.

So in the interest of time, I fell back to a language I'm most comfortable with right now, Ruby.

Sinatra and Ruby

Noah isn't so much a web application as it is this 'api thing'. There's no proper front end and honestly, you guys don't want to see what my design deficient mind would create. I like to joke that in the world of MVC, I stick to the M and C. Sure, APIs have views but not in the "click the pretty button sense".

I had been doing quite a bit of glue code at the office using Sinatra (and EventMachine) so I went with that. Sinatra is, if you use sheer number of clones in other languages as an example, a success for writing API-only applications. I also figured that if I wanted to slap something proper on the front, I could easily integrate it with Padrino.

But now I had to address the data storage issue.

Redis

Previously, as a way to learn Python at another company, I wrote an application called Vogeler. That application had a lot of moving parts - CouchDB for storage and RabbitMQ for messaging.

I knew from dealing with CouchDB on CentOS5 that I wasn't going to use THAT again. Much of it would have been overkill for Noah anyway. I realized I really needed nothing more than a key/value store. That really left me with either Riak or Redis. I love Riak but it wasn't the right fit in this case. I needed something with a smaller dependency footprint. Mind you Riak is VERY easy to install but managing Erlang applications is still a bit edgy for some folks. I needed something simpler.

I also realized early on that I needed some sort of basic queuing functionality. That really sealed Redis for me. Not only did it have zero external dependencies, but it also met the needs for queuing. I could use lists as dedicated direct queues and I could use the built-in pubsub as a broadcast mechanism. Redis also has a fast atomic counter that could be used to approximate the ZK sequence primitive should I want to do that.

Additionally, Redis has master/slave (not my first choice) support for limited scaling as well as redundancy. One of my original design goals was that Noah behave like a traditional web application. This is a model ops folks understand very well at this point.

EventMachine

When you think asynchronous in the Ruby world, there's really only one tool that comes to mind, EventMachine. Noah is designed for asynchronous networks and is itself asynchronous in its design. The callback agent itself uses EventMachine to process watches. As I said previously, this is simply using an EM friendly Redis driver that can do PSUBSCRIBE (using em-hiredis) and send watch messages (using em-http-request since we only support HTTP by default).

Ohm

Finally I slapped Ohm on top as the abstraction layer for Redis access. Ohm, if you haven't used it, is simply one of if not the best Ruby library for working with Redis. It's easily extensible, very transparent and frankly, it just gets the hell out of your way. A good example of this is converting some result to a hash. By default, Ohm only returns the id of the record. Nothing more. It also makes it VERY easy to drop past the abstraction and operate on Redis directly. It even provides helpers to get the keys it uses to query Redis. A good example of this is in the Linking and Tagging code. The following is a method in the Tag model:

    def members=(member)
      self.key[:members].sadd(member.key)
      member.tag! self.name unless member.tags.member?(self)
    end

Because Links and Tags are a one-to-many across multiple models, I drop down to Redis and use sadd to add the object to a Redis set of objects sharing the same tag.

It also has a very handy feature which is how the core of Watches are done. You can define hooks at any phase of Redis interaction - before and after saves, creates, updates and deletes. the entire Watch system is nothing more than calling these post hooks to format the state of the object as JSON, add metadata and send the message using PUBLISH messages to Redis with the Noah namespace as the channel.

Distribution vectors

I've used this phrase with a few people. Essentially, I want as many people as possible to be able to use the Noah server component. I've kept the Ruby dependencies to a minimum and I've made sure that every single one works on MRI 1.8.7 up to 1.9.2 as well as JRuby. I already distribute the most current release as a war that can be deployed to a container or run standalone. I want the lowest barrier to entry to get the broadest install base possible. When a new PaaS offering comes out, I pester the hell out of anyone I can find associated with it so I can get deploy instructions written for Noah. So far you can run it on Heroku (using the various hosted Redis providers), CloudFoundry and dotcloud.

I'm a bit more lax on the callback daemon. Because it can be written in any language that can talk to the Redis pubsub system and because it has "stricter" performance needs, I'm willing to make the requirements for the "official" daemon more stringent. It currently ONLY works on MRI (mainly due to the em-hiredis requirement).

Doing things differently

Some people have asked me why I didn't use technology A or technology B. I think I addressed that mostly above but I'll tackle a couple of key ones.

ZeroMQ

The main reason for not using 0mq was that I wasn't really aware of it. Were I to start over and still be using Ruby, I'd probably give it a good strong look. The would still be the question of the storage component though. There's still a possible place for it that I'll address in part four.

NATS

This was something I simply had no idea about until I started poking around the CloudFoundry code base. I can almost guarantee that NATS will be a part of Noah in the future. Expect much more information about that in part four.

MongoDB

You have got to be kidding me, right? I don't trust my data (or anyone else's for that matter) to a product that doesn't understand what durability means when we're talking about databases.

Insert favorite data store here

As I said, Redis was the best way to get multiple required functionality into a single product. Why does a data storage engine have a pubsub messaging subsystem built in? I don't know off the top of my head but I'll take it.

Wrap up - Part 3

So again, because I evidently like recaps, here's the take away:

• The key components in Noah are Redis and Sinatra
• Noah is written in Ruby because of time constraints in learning a new language
• Noah strives for the server component to have the broadest set of distribution vectors as possible
• Ruby dependencies are kept to a minimum to ensure the previous point
• The lightest possible abstractions (Ohm) are used.
• Stricter requirements exist for non-server components because of flexibility in alternates
• I really should learn me some erlang
• I'm not a fan of MongoDB

If you haven't guessed, I'm doing one part a night in this series. Tomorrow is part four which will cover the future plans for Noah. I'm also planning on a bonus part five to cover things that didn't really fit into the first four.

Chef and Encrypted Data Bags - Revisted

In my previous post here I described the logic behind wanting to store data in an encrypted form in our Chef data bags. I also described some general encryption techniques and gotchas for making that happen.

I've since done quite a bit of work in that regard and implemented this at our company. I wanted to go over a bit of detail about how to use my solution. Fair warning, this is a long post. Lot's of scrolling.

A little recap

As I mentioned in my previous post, the only reliable way to do the encryption of data bag items in an automated fashion is to handle key management yourself outside of Chef. I mentioned two techniques:

• storing the decryption key on the server in a flat file
• calling a remote resource to grab the key

Essentially the biggest problem of this issue is key management and, in an optimal world, how to automate it reliably. For this demonstration, I've gone with storing a flat text file on the server. As I also said in my previous post, this assumes you tightly control access to that server. We're going with the original assumption that if a malicious person gets on your box, you're screwed no matter what.

Creating the key file

I used the knife command to handle my key creation for now:

knife ssh '*:*' interactive
echo "somedecryptionstringblahblahblah" > /tmp/.chef_decrypt.key
chmod 0640 /tmp/.chef_decrypt.key

Setting up the databags and the rake tasks

One of the previous things I mentioned is knowing when and what to encrypt. Be sensible and keep it simple. We don't want to throw out the baby with the bath water. The Chef platform has lots of neat search capabilities that we'd like to keep. In this vein, I've created a fairly opinionated method for storing the encrypted data bag items.

We're going to want to create a new databag called "passwords". The format of the data bag is VERY simple:

We have an "id" that we want to use and the plaintext value that we want to encrypt.

Rake tasks

In my local chef-repo, I've created a 'tasks' folder. In that folder, I've added the following file:

As you can see, this requires a rubygem called encrypted_strings. I've done a cursory glance over the code and I can't see anything immediately unsafe about it. It only provides an abstraction to the native OpenSSL support in Ruby with an additional String helper. However I'm not a cryptographer by any stretch so you should do your own due diligence.

At the end of your existing Rakefile, add the following:

load File.join(TOPDIR, 'tasks','encrypt_databag_item.rake')

If you now run rake -T you should see the new task listed:

rake encrypt_databag[databag_item]  # Encrypt a databag item in the passwords databag

If you didn't already create a sample data bag and item, do so now:

mkdir data_bags/passwords/
echo '{"id":"supersecretpassword","data":"mysupersecretpassword"}' > data_bags/passwords/supersecretpassword.json

Now we run the rake task:

rake encrypt_databag[supersecretpassword]

Found item: supersecretpassword. Encrypting
Encrypted data is <some ugly string>
Uploading to Chef server
INFO: Updated data_bag_item[supersecretpassword_crypted.json]

You can test that the data was uploaded successfully:

knife data bag show passwords supersecretpassword

{
"data": "<some really ugly string>",
"id": "supersecretpassword"
}

Additionally, you should have in your 'data_bags/passwords' directory a new file called 'supersecretpassword_crypted.json'. The reason for keeping both files around is for key management. Should you need to change your passphrase/key, you'll need the original file around to reencrypt with the new key. You can decided to remove the unencrypted file if you want as long as you have a way of recreating it.

Using the encrypted data

So now that we have a data bag item uploaded that we need to use, how do we get it on the client?
That will require two cookbooks:

• databag_decrypt
• A cookbook which needs the decrypted data. example

The general idea is that, in any cookbook you need decrypted data, you essentially do three things:

• include the decryption recipe

include_recipe "databag_decrypt::default"

• assign the crypted data to a value via databag search
  

  password = search(:passwords, "id:supersecretpassword").first

• assign the decrypted data to a value for use in the rest of the recipe
  

  decrypted_password = item_decrypt(password[:data])

From there, it's no different that any other recipe. Here's an example of how I use it to securely store Amazon S3 credentials as databag items:

include_recipe "databag_decrypt::default"
s3_access_key = item_decrypt(search(:passwords, "id:s3_access_key").first[:data])
s3_secret_key = item_decrypt(search(:passwords, "id:s3_secret_key").first[:data])
s3_file erlang_tar_gz do
  bucket "our-packages"
  object_name erlang_file_name
  aws_access_key_id s3_access_key
  aws_secret_access_key s3_secret_key
  checksum erl_checksum
end

Changing the key

Should you need to change the key, you'll need to jump through a few hoops:

• Update the passphrase on each client. Ease depends on your method of key distribution
• Update the passphrase in the rake task
• Reencypt all your data bag items.

The last one can be a pain in the ass. Since Chef currently doesn't support multiple items in a data bag json file, I created a small helper script in my chef-repo called 'split-em.rb'.
I store all of my data bag items in large json files and use split-em.rb to break them into individual files. Those file I upload with knife:

bin/split-em.rb -f data_bags/passwords/passwords.json -d passwords -o

Parsing data for svnpass into file data_bags/passwords/svnpass.json
Parsing data for s3_access_key into file data_bags/passwords/s3_access_key.json
Parsing data for s3_secret_key into file data_bags/passwords/s3_secret_key.json
#Run the following command to load the split bags into the passwords in chef
for i in svnpass s3_access_key s3_secret_key; do knife data bag from file passwords $i.json; done

You could then run that through the rake task to reupload the encrypted data:

for i in svnpass s3_access_key s3_secret_key; do rake encrypt_databag[$i]; done

Limitations/Gotchas/Additional Tips

Take note of the following, please.

Key management

The current method of key management is somewhat cumbersome. Ideally, the passphrase should be moved outside of the rake task. Additionally, the rekey process should be made a distinct rake task. I imagine a workflow similar to this:

• rake accepts a path to the encryption key
• additional rake task to change the encryption key in the form of oldpassfile/newpassfile.
• Existing data is decrypted using oldpassfile, reencrypted using new passfile and sent back to the chef server.

Optimally, the rake task would understand the same attributes that the decryption cookbook does so it can handle key managment on the client for you. I'd also like to make the cipher selection configurable as well an integrate it into the above steps.

Duplicate work

Seth Falcon at Opscode is already in the process of adding official support for encrypted data bags to Chef. His method involves converting the entire databag sans "id" to YAML and encrypting it. I wholeheartedly support that effort but that would obviously require a universal upgrade to Chef as well. The purpose of my cookbook and tasks is to work with the existing version.

AWS IAM

If you're an Amazon EC2 user, you should start using IAM NOW. Stop putting your master credentials in to recipes and limit your risk. I've created a 'chef' user who I give limited access to certain AWS operations. You can see the policy file here. It gives the chef user read-only access to 'my_bucket' and 'my_other_bucket'.
If you wanted to get REALLY sneaky, you could use fake two-factor authentication to store your key in S3:

• Encrypt data bag items with "crediential B" password except for one item "s3_credentials"
• s3_credentials (crendential A) is encrypted with a passphrase and managed similar to this article
• Use transient credentials to access S3 and grab a passphrase file (credential B)
• Decrypt data with secondary credentials

You would have to heavily modify the cookbook to do this. I think the current implementation is fine.

File-based passphrases

I'm not a big fan of the file-based passphrase method. While we agreed that you should consider yourself screwed if someone gets on the box, that still leaves poorly coded applications running as an attack vector. Imagine you have an application that must run as root. Now it can read the passphrase. Should that application become remotely exploitable, the passphrase file is vulnerable. I'm leaning to the method of a private server that allows RESTful access to grab the key. I've already added support in the cookbook for a passphrase type of 'url'.

Wrapup

I think that covers anything. I'd love some feedback on what people think. We've already implemented this in a limited scope for using IAM credentials in our cookbooks. I can easily revoke those should they get compromised without having to generate all new master keys.

Automating EBS Snapshot validation with @fog - Part 2

This is part 2 in a series of posts I'm doing - You can read part 1 here

Getting started

I'm not going to go into too much detail on how to get started with Fog. There's plenty of documentation on the github repo (protip: read the test cases) and Wesley a.k.a @geemus has done some awesome screencasts. I'm going to assume at this point that you've at least got Fog installed, have an AWS account set up and have Fog talking to it. The best way to verify is to create your .fog yaml file, start the fog command line tool and start looking at some of the collections available to you.

For the purpose of this series of posts, I've actually created a small script that you can use to spin up two ec2 instances (m1.small) running CentOS 5.5, create four (4) 5GB EBS volumes and attach them to the first instance. In addition to the fog gem, I also have awesome_print installed and use it in place of prettyprint. This is, of course, optional but you should be aware.

WARNING: The stuff I'm about to show you will cost you money. I tried to stick to minimal resource usage but please be aware you need to clean up after yourself. If, at any time, you feel like you can't follow along with the code or something isn't working - terminate your instances/volumes/resources using the control panel or command-line tools. PLEASE DO NOT JUST SIMPLY RUN THESE SCRIPTS WITHOUT UNDERSTANDING THEM.

The setup script

The full setup script is available as gist on github - https://gist.github.com/724912#file_fog_ebs_demo_setup.rb

Things to note:

• Change the key_name to a valid key pair you have registered with EC2
• There's a stopping point halfway down after the EBS volumes are created. You should actually stop there and read the comments.
• You can run everything inside of an irb session if you like.

The first part of the setup script does some basic work for you - it reads in your fog configuration file (~/.fog) and creates an object you can work with (AWS). As I mentioned earlier, we're creating two servers - hdb and tdb. HDB is the master server - say your production MySQL database. TDB is the box which will be running as the validation of the snapshots.

In the Fog world, there are two big concepts - models and collections. Regardless of cloud provider, there are typically at least two models available - Compute and Storage. Collections are data objects under a given model. For instance in the AWS world, you might have under the Compute model - servers, volumes, snapshots or addresses. One thing that's nice about Fog is that, once you establish your connection to your given cloud, most of your interactions are the same across cloud providers. In the example above, I've created a connection with Amazon using my credentials and have used that Compute connection to create two new servers - hdb and tdb. Notice the options I pass in when I instantiate those servers.

• image_id
• key_name

If I wanted to make these boxes bigger, I might also pass in 'flavor_id'. If you're running the above code in an irb session, you might see something like the following when you instantiate those servers: Not all of the fields may be available depending on how long it takes Amazon to spin up the instance. The above shot is after the instance was up and running. For instance, when you first created 'tdb', you'll probably see "state" as pending for quite some time. Fog has a nice helper method for all models call 'wait_for'. In my case I could do:

tdb.wait_for { print "."; ready?}

And it would print dots across the screen until the instance is ready for me to log in. At the end, it will tell you the amount of time you spent waiting. Very handy. You have direct access to all of the attributes above via the instance 'tdb' or 'hdb'. You can use 'tdb.dns_name' to get the dns name for use in other parts of your script for example. In my case, after the server 'hdb' is up and running, I now want to create the four 5GB EBS volumes and attach them to the instance:

I've provided four device names (sdi through sdl) and I'm using the "volumes" collection to create them (AWS.volumes.new). As I mentioned earlier, all of the attributes for 'hdb' and 'tdb' are accessible by name. In this case, I have to create my volumes in the same availability zone as the hdb instance. Since I didn't specify where to create it when I started it, Amazon has graciously chosen 'us-east-1d' for me. As you can see, I can easily access that as 'hdb.availability_zone' and pass it to the volume creation section. I've also specified that the volume should be 5GB in size.

At the point where I've created the volume with '.new' it hasn't actually been created. I want to bind it to a server first so I simply set the volume.server attribute equal to my server object. Then I 'save' it. If I were to log into my running instance, I'd probably see something like this in the 'dmesg' output now:

sdj: unknown partition table

sdk: unknown partition table

sdl: unknown partition table

sdi: unknown partition table

As you can see from the comments in the full file, you should stop at this point and setup the volumes on your instance. In my case, I used mdadm and created a RAID0 array using those four volumes. I then formatted them, made a directory and mounted the md0 device to that directory. If you look, you should now have an additional 20GB of free space mounted on /data. Here I might make this the data directory for mysql (which is the case in our production environment). Let's just pretend you've done all that. I simulated it with a few text files and a quick 1GB dd. We'll consider that the point-in-time that we want to snapshot from. Since there's no actual constant data stream going to the volumes, I can assume for this exercise that we've just locked mysql, flushed everything and frozen the XFS filesystem. Let's make our snapshots. In this case I'm going to be using Fog to do the snapshots but in our real environment we're using the ec2-consistent-snapshot script from Aelastic. First let's take a look at the state of the hdb object:

Notice that the 'block_device_mapping' attribute now consist of an array of hashes. Each hash is a subset of the data about the volume attached to it. If you aren't seeing this, you might have to run 'hdb.reload' to refresh the state of the object. To create our snapshots, we're going to iterate over the block_device_mapping attribute and use the 'snapshots' collection to make those snapshots:

One thing you'll notice is that I'm being fairly explicity here. I could shorthand and chain many of these method calls but for clarity, I'm not.

And now we have 4 snapshots available to us. The process is fairly instant but sometimes it can lag. As always, you should check the status via the .state attribute of an object to verify that it's ready for the next step. Here's a shot of our snapshots right now:

That's the end of Part 2. In the next part, we'll have a full fledged script that does the work of making the snapshots usable on the 'tdb' instance.

Automating EBS Snapshot validation with @fog - Part 1

Background

One thing that's very exciting about the new company is that I'm getting to use quite a bit of Ruby and also the fact that we're entirely hosted on Amazon Web Services. We currently leverage EBS, ELB, EC2 S3 and CloudFront for our environment. The last time I used AWS in a professional setting, they didn't even have Elastic IPs much less EBS with snapshots and all the nice stuff that makes it viable for a production environment. I did, however, manage to keep abreast of changes using my own personal AWS account.

Fog

Of course the combination of Ruby and AWS really means one thing - Fog. And lot's of it.

When EngineYard announced the sponsorship of the project, I dove headlong into the code base and spent what time I could trying to contribute code back. The half-assed GoGrid code in there right now? Sadly, some of it is mine. Time is hard to come by these days. Regardless, I'm no stranger to Fog and when I had to dive into the environment and start getting it documented and automated, Fog was the first tool I pulled out and when the challenge of verifying our EBS snapshots (of which we're currently at a little over 700), I had no choice but to automate it.

Environment

A little bit about the environment:

• - A total of 9 EBS volumes are snapshotted each day
• - 8 of the EBS volumes are actually raid0 mysql data stores across two DB servers (so 4 disks on one/4 disks on another)
• - The remaining EBS volume is a single mysql data volume
• - Filesystem is XFS and backups are done using the Aleastic ec2-consistent-snapshot script (which currently doesn't support tags)

The end result of this is to establish a rolling set of validated snapshots. 7 daily, 3 weekly, 2 monthly. Fun!

Mapping It Out

Here was the attack plan I came up with:

• - Identify snapshots and groupings where appropriate (raid0, remember?)
• - create volumes from snapshots
• - create an m1.xlarge EC2 instance to test the snapshots
• - attach volume groups to the test instance
• - assemble the array on the test instance
• - start MySQL using the snapshotted data directory
• - run some validation queries using some timestamp columns in our schema
• - stop MySQL, unmount volume, stop the array
• - detach and destroy the volumes from the test instance
• - tag the snapshots as "verified"
• - roll off any old snapshots based on retention policy
• - automate all of the above!

I've got lots of code samples and screenshots so I'm breaking this up into multiple posts. Hopefully part 2 will be up some time tomorrow

PyCon DevOps piggy back

So I had a random idea the other night and like any other random idea I immediately sent it to Twitter.

This of course brought feedback which is the whole point, right?

The idea was to have a Velocity style conference in the South East. We all know my love for Atlanta and my half-disdain/half-jealousy of the West coast. So I threw the idea out on twitter and immediately got my first reply from Joe Heck with a bit of reality thrown in:

@lusis nice idea. critical mass with either be easy or impossible to get. You might consider riffing on existing conferences ... PyCon2011

Awesome idea so I headed off to to read up on how PyCon does that kind of thing. I shot off an email to the pycon-organizers mailing list and got some really nice responses. I also got a private tweets from people on the list as well.

The end result is this. If I want to hitchhike on the back of PyCon for a devops-related conference, here are the requirements/suggestions:

• Involve Python in some way
• Will need to take advantage of the Open Spaces system

This essentially means unless I (or someone else) is giving a full blown talk on Python and DevOps, it will be an ad-hoc thing. We can't reserve the spaces until the day of the conference. I'm also not sure how big the spaces are. I think this is the same place LISA was held years ago so you might be able to snag a dividable room segment?

So what does everyone think? I'm considering giving a talk on the state of devops toolchains in Python (func, cobbler, fabric, kokki, overmind, whatever else) but I don't know that I'm ready for that yet after a single LUG presentation ;)

I know that Mitchell H. of Vagrant fame was considering heading into town for it. Vagrant isn't just for Rubyists ;)

I'm open to ideas. I'd love to just have the conference I sent the tweet about but when I really think about it, I don't think I can pull something like that off in this amount of time.

Many thanks to the pycon-organizers folks for the input - Doug Hellmann, Vern Ceder and Jesse Noller. Also to Dean Goodmanson for his feedback via Twitter.

.plan

TODO

• Work through Vogeler issue queue (http://github.com/lusis/vogeler/issues)
• @padrinorb test suite enhancement - functional/integration (http://github.com/padrino/padrino-framework)
• Integrate libvirt into @fog (http://github.com/geemus/fog)
• Finish integrating Riak support into @padrinorb (http://github.com/seancribbs/ripple)
• Create a distro that uses LFS for the explicit purpose of hosting dynamic language applications (http://lusislog.blogspot.com/2010/09/distributions-and-dynamic-languages.html)
• Do all of the above while maintaining my marriage and raising two kids ;)

 

 

 

 

Five open source Projects I wish I could fund

I've always said to myself that if I ever become independently wealthy, I'm going to bankroll some things I've always wanted that the opensource community hasn't felt a need to provide. Mind you, I'm not independently wealthy so don't expect to see much from me.

Anyway, here's my current "wishlist":

OpenWire Ruby drivers for ActiveMQ.

For that matter, I'd love wire level drivers for a bunch of stuff. In the case of ActiveMQ, it's nice that it's all plaintext but it doesn't support some of the same semantics as the OpenWire drivers and quite honestly wasn't very reliable in the testing I did. Say it with me folks, stateless protocols are not the way to talk to queue servers and ESPECIALLY not over HTTP. REST semantics don't map properly to core message queue concepts.

Non-Win32/DLL Ruby drivers for MSMQ and other Microsoft products

This really bit me in the ass at the AJC. It would have made my life a whole lot easier if we had a method for talking to MSMQ from a non-Windows platform. Sure, Microsoft documents the protocols for the most part but unless I'm planning on learning C and implementing a native extension, I don't see me doing it.

An open source ETL/DW/BI suite built on NoSQL. Bonus points for supporting rolling warehouse loads.

It may sound silly but I always thought that of all the promise of NoSQL concepts, the fact that your warehouse is denormalized makes it a great fit. I also think Map/reduce is a much more logical construct for BI reporting. There are a few headaches though which is why, even as a self-contained suite, it will take effort to gain traction:

• ETL vendors would need to support the NoSQL engine on the Load side
• BI/Reporting tools would need to support the NoSQL engine
• Report creators (many times, employees from each business unit stakeholder and non-technical) need to learn Map/Reduce concepts for scheduled reports
• Map/reduce is a poor/impossible choice for Ad-hoc queries at least as far as the current crop of NoSQL engines is concerned.

Essentially, you would HAVE to create your own suite - soup to nuts - and provide a way to move people from thinking in SQL for report generation. Maybe a hybrid approach makes more sense. Assuming I were king for a day, the warehouse side would be a hierarchical design - all data is dumped denormalized into a NoSQL engine. Scheduled reporting is done via Map/reduce against that data. Additionally a second load phase either concurrent with or post NoSQL load (does that make it ETLL?) dumps a business rule defined amount of data in a traditional RDBMS store for Ad-hoc purposes. I dunno. I could be over-engineering it ;)

PostgreSQL and MySQL move to a pluggable replication architecture based on message queues.

I'm not sure if this is still the case but many years ago, DB2 was using MQ Series for geographical replication. Message queues are message agnostic and implement all the features required of replication - guaranteed delivery and ordered delivery for instance. Imagine how easy it would be to scale out MySQL read slaves if they weren't all hitting the master server? Message queues are perfect for this. I might implement it something like this with ActiveMQ:

• Replication messages are pushed to a queue for known slaves. One queue per slave.
• Said messages are duplicated into a Topic
• New slaves subscribe to the Topic and come current
• New slave is then converted to its own queue

Slaves never talk to the master server directly. You can spin up slaves at any time even without a backup. Just bring the slave up, point to the topic and get current on your own time. At some given point, you're converted to your own queue and unsub from the topic.

A DSL for implementing random binary protocols.

I thought this was what protobuf did but as I look at it more, I realize I might have been mistaken. Imagine if you could take the MSDN docs that describe the MSMQ protocols. Convert that information into said DSL and execute 'foo' against the DSL. Blammo, you have a driver for that protocol. Is that even possible?

Anyway, there goes my business ideas for the next century. I do hope someone runs off with them and does something fun. In seriousness, I can't be the only person who's ever thought of these things. Hell, look at the database replication one. I straight stole that from IBM.

Besides, there's probably patents on all of these ideas already =P

The opposite of DevOps

I thought about whether or not to write this post but I think it's an interesting example of the kinds of problems that the DevOps methodology is trying to solve.

I turned in my two weeks notice with the paper on Friday. There were several reasons but none of them are a negative reflection on my employer. The role I was originally hired for was no longer valid and there wasn't a transition path because of platform changes in the back office. I could have stayed (and was asked to stay) but there wasn't anything long term where my skill set was useful. As with any key team member, meetings are called to discuss any outstanding issues, transition responsibilities and the like.

In preparation for the meeting today, I drew up a list my responsibilities at a macro level and then broke that down by task. As it's always been for me, that list spanned several "silos" in the traditional IT organization model. Here's a sample snippit:

Subversion	User Management, Repository Management
MySQL	User Management, Database Management, Performance Management
Linux	User Management, OS Configuration, Application Management, Code Management
Puppet	Configuration Management, Recipe Management
Ruby	VM Management, Gem Management

There were also entries for various internal applications that I've worked on (including some development) and supported from an operational perspective. Mind you, I was embedded as an operations guy with a specific development group in the organization. Sort of a DevOps-lite role.

What was really interesting about the meeting was how the lines were broken down. Literally lines were drawn as to where operations would stop supporting something and the group I was leaving would take over. Take the Linux example:

Application Management was intended to refer to software that was "standard" as a part of the distro. Things like Apache or MySQL. Code Management refers to our internally developed code (all Rails applications except for the sexy Sinatra webservice I wrote). In the end, the responsibilities were shifted divided like so:

- Operations Team supported up to the installation and configuration of Apache (including vhosts) with the exclusion of Passenger configuration.

- Passenger configuration and internal code would now be managed by the Development group. They would handle deployments themselves via Webistrano.

- MySQL? Passed on to those currently managing the MSSQL database servers.

That's the very definition of a silo'd IT infrastructure. Everything is thrown over the wall. At least the deploys remain with the development team. There is nothing "wrong" with this model of IT governance. It's not agile but many companies use it. Contrast that to the position I'll be starting on the 6th of July.

An explicit group is being formed inside the company called "DevOps". I know that at this point everyone is incredulous. I can hear it now; "You're doing it wrong!". Interestingly enough, I asked the same question during the interview process. We all know that DevOps is model and philosophy and not a title or department. You don't have two development teams - Agile and Waterfall. You have Developers and they use one methodology over the other. The same goes for Operations. The people I was interviewing with were cognizant of this fact as well. The reason the group is being called "DevOps" is strictly for organizational and political reasons. The goal of the team is to actually develop a set of operational and developmental processes,tools and guidelines that embody everything that DevOps represents.

This is being done inside of one division of the company for now with the Director of Development and Director of Operations essentially sharing a brain and heading things up. The work that this group does will establish and codify something that will be used throughout the rest of the company. We'll be doing development of tools, architecting systems and recommending/implementing solutions that will, among other things, define how the organization operates itself from an IT perspective. Additionally we'll be supporting this as any traditional operations team would but, for now, the group has a distinct title. My title is "Systems Architect" which nice and generic enough to apply to both traditional groups ;)

The only remotely distressing part of the whole thing is that I'll probably have to learn some Python. (tongue firmly in cheek). There's a point to be made that distribution vendors have settled on Python as the Lingua Franca of OS management - excluding things like Chef, Puppet and Nagios which have their own respective DSLs that abstract away much of the language they were written in. Yes, my precious Ruby will still be there when it comes to extending, say, Puppet. I'll probably still write quite a few service checks for Nagios in shell (should Nagios be the best fit).

But language wars aside, I think this gives a clear picture of exactly the types of problems the DevOps methodology is trying to solve. Agility and flexibility across all branches of IT produces a leaner, strong and faster organization that can spend less time "in the muck" (as John Willis likes to say) and start making the company more money.

Parsing Nagios Objects in Ruby Redux

When I was with MediaOcean, one of the projects I was working on was a stack of ruby code for parsing/writing nagios configs. This was in parallel with setting up puppet. I actually got pretty far along but then DDS had a RIF and I was back on the market.

I had to put the code aside and just recently pulled it out again. Some/most of it is pretty ugly. I had quite a bit done though.

So here I am "refactoring" (and I use that term VERY VERY loosely) the code base. So I'm happily parsing away - iterating over the cfg_file/cfg_dir entries and build a nice big hash to hold all the object definitions. I dump it to a YAML file for validation when I notice this nice bit of junk in my timeperiod dump:

Oh yeah...that looks right =/

For those who don't know, nagios object definitions are pretty straightforward. Here's an example:

Easy enough to parse, right? Object type named (host). A clear beginning and end denoted by curly braces. Object attributes in a seemingly key/value type markup. Some people smash the opening brace up against the object type definition but that's easily caught.

Iterating over the definition in Ruby is pretty straightforward (assuming a perfect example):

What becomes a problem is timeperiod defintions. The syntax for those is somewhat complicated. Looking back at the YAML fragment above, you can see why. For timeperiods, while the rightmost value is the actual time frame, the left part (usually a day of the week) can actually be a specific day. So if I wanted to create an entry for Christmas in the U.S., I would define it like so:

december 25 00:00-00:00


Things can be even MORE complicated if I wanted to handle non-date holidays:

monday 1 september 00:00-00:00 ; Labor Day (first Monday in September)
thursday -1 november 00:00-00:00 ; Thanksgiving (last Thursday in November)


Ugly, no? I searched high and low for the ability to split starting from the right and didn't find anything native. I had to resort to implementing my own rsplit method for String:

It works but I also can't use it across the board. If I do, I break things that were working (alias, service_description) that can have a space in the value.

I'm not even going to get into trying to convert timeperiod definitions into Date objects yet. That gives me cold sweats.

Code on Github

I'm working on pushing all my code snippets I've developed over the years to Github. Right now I have the Ruby-EWS and Ruby-Downtime stuff uploaded. I'll get the rest over as I can sanitize it.

http://github.com/lusis

Finally getting to dig into Puppet

Well I'm finally getting to dig into Puppet at a professional level. I've been handed the keys to establishing the Puppet infrastructure. Mixed Linux/Solaris clients. Beyond the hassle of building up-to-date RPMs for CentOS/RHEL, everything else is going smoothly.

I've got all of my configuration in version control and handy script to create new modules from templates. I'm already pushing out OS specific configurations as well as OS version specific stuff. Augeas is AMAZING. I don't know how I never found/saw it before.

We're also pushing out Splunk (Cox is a huge Splunk customer) across the board. I'm still undecided about its value but I haven't really looked at it since it first came out. It just seemed, at the time, like a huge overhead for a problem that was already tackled (syslogging).

My next step is deciding how to integrate everything into my Kickstart scripts. There's a TON of options out there. I'm really trying to avoid setting up Cobbler but I might be at the end of what can be done with Kickstart already.

I'm still frustrated with managing current versions of Ruby and gems with RPM but gem2rpm has made some of that easier. I'm not responsible for the Solaris boxen so I have no idea how those guys plan on managing those bad boys.

Exchange Web Services via Ruby

I realized as I went to post an update on this that I don't think I ever posted an actual blog entry about it.

While I was on contract at HSWI, the email was hosted on Exchange. The development team was all on Mac workstations but the front-side was all on Windows. Since I was running Linux full-time, Exchange access wasn't a big deal. I ran Mutt and used LDAP/IMAPS/SSMTP. Digging around the Mail.app on the OS X side however, I realized that the client side was using Exchange Web Services for much of the functionality.

Wanting to do some poking around with Ruby and SOAP, I figured it would be a fun exercise to talk to the Exchange server with Ruby. I also had an itch to scratch thinking I could use it as an address book source for Mutt. I got it working in a day or so after dealing with some broken functionality in either the WSDL from the Exchange side or how SOAP4R tried to translate it.

You can find the code here.

Anyway, I'm no longer with HSWI but over at the AJC, we're also using Exchange in a much greater capacity. Anyway, so I whipped out the "old" code and was depressed to find out it didn't work.

Every attempt gave me a 401 error. It made no sense since I could access OWA, ActiveSync with my DROID (awww yeah) and even access the EWS wsdls on the server.

As I started to poke around online, I started to realize that in some of the more complex configurations, the Exchange server is hidden behind an ISA server or something. I don't do the Microsoft world much anymore and I have no real access to the environment.

What I noticed though, was that the internal IP is different than the external one. That gave me an idea to make sure and test the code externally from home while NOT being on the VPN.

It worked!

As I poked around with THAT information, I remembered an interesting thing that happened the one time I logged. on to a Windows machine at the office. If I fired up I.E. and went to the OWA url, I was logged in directly.

What I'm guessing is that the Exchange server has a different set of criteria for authenticating internally than externally. I'm not up to date on current Exchange and AD implementations so I have no idea what the configuration is that's preventing me from using EWS but still allowing me to use OWA from Firefox under Linux and OS X.

If any MS guys out there have any idea what's causing this, I'd be interested to know so I can either work around it or document it appropriately.

I have a *FEELING* that it's somehow related to NTLM but I don't know how to force my EWS call to bypass it just yet.

Sold on unit testing

I'll tell anyone who asks that I'm NOT a programmer. I've always understood the value in unit tests but considered them somewhat limited in scope. I mean it's pretty much impossible to write a unit test for every possible scenario and (IMHO) impossible to write a unit test that simulates the flow of a user from application login to data entry to logout using the entire infrastructure (i.e. client browser -> through load balancer -> through app server -> database -> back).

Because of those reasons, while I understood the need for unit tests for "stupid stuff", the fact that you had to add unit tests for each bug that popped up over the life of a product felt "odd". Of course this wouldn't stop me from bitching at a developer who's lack of a unit test caused me to restore a filesystem because it would ascend to the parent directory if the file or directory it was trying to delete did not exist! (True story).

So here I am writing my ruby nagios library and realizing that I'm duplicating A LOT of code writing quick little scripts to test my output. I decide to write unit tests for each of my classes. I found an interesting little article on about.com talking about learning ruby via unit tests. It was an interesting concept and one that made sense to me. So I start with my contact class.

Basically all of the classes that exist for a specific nagios object (contact,service,host) have their own class. One method they have is called "hashify". It's different for each object type but essentially creates a nested hash like so:

{
"nagiosadmin"=>
{"service_notification_period"=>"24x7",
"host_notification_options"=>"d,u,r",
"service_notifications_enabled"=>nil,
"host_notification_enabled"=>nil,
"pager"=>nil,
"service_notification_commands"=>"notify-service-by-email",
"host_notification_period"=>"24x7",
"alias"=>"Nagios Admin",
"host_notification_commands"=>"notify-host-by-email",
"service_notification_options"=>"w,u,c,r",
"email"=>"root@localhost"},
"johnv"=>
{"service_notification_period"=>"24x7",
"host_notification_options"=>"d,u,r",
"service_notifications_enabled"=>nil,
"host_notification_enabled"=>nil,
"pager"=>nil,
"service_notification_commands"=>"notify-service-by-email",
"host_notification_period"=>"24x7",
"alias"=>"John E. Vincent",
"host_notification_commands"=>"notify-host-by-email",
"service_notification_options"=>"w,u,c,r",
"email"=>"root@localhost"}
}

So I start writing unit tests. I test that hashify returns a Hash. I test that I have a hash called 'johnv'. Then I start testing for each of the attributes for johnv. This is where it all falls apart. Attribute email is always returning nil even though I can see right in the debugger that it's set to 'root@localhost'.

After about 30 minutes of dicking around I finally realize that, being a Ruby beginner who has used some of the higher-level Ruby stuff for quickies without actually getting into the meat of the language (Rails, Ruport), I was totally confused on the usage of ":" in hashes much less anywhere else in Ruby. Two minutes after that, I stopped throwing symbols all over the place in my classes when building a hash ;)

My unit test passed! I continue to write the remaining tests and run them. I get a failure. I look in detail and realize that in my original Contact class, I had a typo in the hashify method that defined a key as 'host_notifications_period' instead of 'host_notification_period'!

So yeah, I'm sold on unit tests and I'm not writing another lick of code until I finish writing them for the classes I have now.

Nagios and Ruby

So as I mentioned in my post here, I've been working on a sort of Nagios toolbox in Ruby. I never really found anything at large about it and honestly, most people are doing stuff in shell scripts or with Perl.

I don't honestly blame them but I wanted a project to force me to learn Ruby better and I had a specific need. I could have this written already in Perl or even straight Bash but, again, I wanted to learn Ruby better.

As I said in my other post, here are the design goals starting out:

• Import existing configs (objects only. Not concerned about the operational parameters)
• Parse existing objects (apart from reading the configurations)
• Create objects (hosts,services,contacts, *groups, etc...)
• Write new configs
• Enforce/Support Templating

I'm not writing a new front end. I'm not writing a global configuration suite for Nagios. My goal is object management. Many of the defaults in nagios.cfg work OOB. However, any time I spend in #nagios, the MAJORITY of the questions are related to actually defining what to monitor and how to monitor it. The example included are pretty solid but they're VERY verbose to the point of being confusing. On the flip side, someone like me who can write a .cfg in his sleep, is still using vi to manage entries.

At my new company, they are BIG on automation. I mean EVERYTHING is automated from an SA perspective. The only exception comes with NEW environments. This is still a mishmash of manual processes and checklists with a slathering of the above automation. One of my first things I was tasked with when I walked through the door was to modernize the existing Nagios configuration. This project is part of that. I need to integrate anything I do seamlessly into the existing automation for it to be accepted as process.

So consider this a "declaration of intent to proceed" or somesuch politispeek. Feel free to comment on my lack of Ruby skill and where you think I can change things.

Here's the code I have so far:

http://dev.lusis.org/nagios/ruby/