Five open source Projects I wish I could fund

I've always said to myself that if I ever become independently wealthy, I'm going to bankroll some things I've always wanted that the opensource community hasn't felt a need to provide. Mind you, I'm not independently wealthy so don't expect to see much from me.

Anyway, here's my current "wishlist":

OpenWire Ruby drivers for ActiveMQ.

For that matter, I'd love wire level drivers for a bunch of stuff. In the case of ActiveMQ, it's nice that it's all plaintext but it doesn't support some of the same semantics as the OpenWire drivers and quite honestly wasn't very reliable in the testing I did. Say it with me folks, stateless protocols are not the way to talk to queue servers and ESPECIALLY not over HTTP. REST semantics don't map properly to core message queue concepts.

Non-Win32/DLL Ruby drivers for MSMQ and other Microsoft products

This really bit me in the ass at the AJC. It would have made my life a whole lot easier if we had a method for talking to MSMQ from a non-Windows platform. Sure, Microsoft documents the protocols for the most part but unless I'm planning on learning C and implementing a native extension, I don't see me doing it.

An open source ETL/DW/BI suite built on NoSQL. Bonus points for supporting rolling warehouse loads.

It may sound silly but I always thought that of all the promise of NoSQL concepts, the fact that your warehouse is denormalized makes it a great fit. I also think Map/reduce is a much more logical construct for BI reporting. There are a few headaches though which is why, even as a self-contained suite, it will take effort to gain traction:

• ETL vendors would need to support the NoSQL engine on the Load side
• BI/Reporting tools would need to support the NoSQL engine
• Report creators (many times, employees from each business unit stakeholder and non-technical) need to learn Map/Reduce concepts for scheduled reports
• Map/reduce is a poor/impossible choice for Ad-hoc queries at least as far as the current crop of NoSQL engines is concerned.

Essentially, you would HAVE to create your own suite - soup to nuts - and provide a way to move people from thinking in SQL for report generation. Maybe a hybrid approach makes more sense. Assuming I were king for a day, the warehouse side would be a hierarchical design - all data is dumped denormalized into a NoSQL engine. Scheduled reporting is done via Map/reduce against that data. Additionally a second load phase either concurrent with or post NoSQL load (does that make it ETLL?) dumps a business rule defined amount of data in a traditional RDBMS store for Ad-hoc purposes. I dunno. I could be over-engineering it ;)

PostgreSQL and MySQL move to a pluggable replication architecture based on message queues.

I'm not sure if this is still the case but many years ago, DB2 was using MQ Series for geographical replication. Message queues are message agnostic and implement all the features required of replication - guaranteed delivery and ordered delivery for instance. Imagine how easy it would be to scale out MySQL read slaves if they weren't all hitting the master server? Message queues are perfect for this. I might implement it something like this with ActiveMQ:

• Replication messages are pushed to a queue for known slaves. One queue per slave.
• Said messages are duplicated into a Topic
• New slaves subscribe to the Topic and come current
• New slave is then converted to its own queue

Slaves never talk to the master server directly. You can spin up slaves at any time even without a backup. Just bring the slave up, point to the topic and get current on your own time. At some given point, you're converted to your own queue and unsub from the topic.

A DSL for implementing random binary protocols.

I thought this was what protobuf did but as I look at it more, I realize I might have been mistaken. Imagine if you could take the MSDN docs that describe the MSMQ protocols. Convert that information into said DSL and execute 'foo' against the DSL. Blammo, you have a driver for that protocol. Is that even possible?

Anyway, there goes my business ideas for the next century. I do hope someone runs off with them and does something fun. In seriousness, I can't be the only person who's ever thought of these things. Hell, look at the database replication one. I straight stole that from IBM.

Besides, there's probably patents on all of these ideas already =P

No operations team left behind - Where DevOps misses the mark

I'm a big fan of the "DevOps" movement. I follow any and everyone on twitter who's involved. I've watched SlideShare presentation after presentation. I've pined for a chance to go to Velocity. I watched the keynote live. I've got "the book". These guys are my heroes, not because they did something new per se but because they put a name on it. Gave it a face from the formless mass. Brought it to the forefront.

Any operations guy worth his salt has been doing some part of what is constituting DevOps for a long time. We automated builds. If we had to do something more than once, we wrote a script to handle it. My favorite item from ThinkGeek was a sticker that said "Go away or I will replace you with a very small shell script". We pxe-booted machines from kickstart files. We were lazy and didn't want to have to deal with the same bullshit mistakes over and over. When I read the intro to the Web Operations book, I was shouting outloud because this was the FIRST book that accurately described what I've been doing for the past 15 years.

I tell you all that so you don't think I'm down on the "tribe" (as Tim O'Reilly called us). These are my people. We're on the same wavelength. I love you guys. Seriously. But just like any intervention, someone has to speak out. There's a "trend" that seems to be forming that's leaving some operations teams behind and those folks don't have a choice.

I mentioned in a previous post that I'm working for a new company. Because of legal restrictions and company security policy, among other things, I can't go into too many details. However, the same things I'm going to be talking about apply to more than just our company.

The company recently formed a dedicated group called "DevOps". The traditional SA/Operations team was reformed into a "DevOps Support" and a handful of other folks were formed into a "DevOps Architecture" team. Right now that second group consists of me and two of the senior staff who moved over from the original SA team. Now you might look at this and say "Yer doin' it wrong!" but there's some logic behind this thought process. Without breaking out a few people from the daily operational support issues, no headway could be really made on implementing anything. This isn't to imply anything about how the company operates or the quality of the product. It's simply a fact of trying to retrofit a new operational model on top of an already moving traditional business process. The same issues arose when teams started migrating from a waterfall to agile. Sure you could implement agile in the NEXT project but forget about upsetting the boat on the current product line. In addition to changing how developers operated, you had a whole host of other stakeholders who needed to be convinced.

I once had a manager who I really disliked but he had a saying - "It's like changing the tires on the race car while it's going around the track"

That's the position many traditional companies are in right now. Walking in the door and telling them they really should be doing X instead of Y is nice. Everyone with a brain knows it makes sense. It's obviously more efficient, reduces support issues, makes for a better work environment and cures cancer but it simply cannot be implemented by burning the boat. So, yes, some companies will have to form dedicated groups and work with stakeholders and go through the whole process that a DevOps mentality is trying to replace just to implement it.

But that's not the only roadblock.

Sarbanes-Oxley

Any publicly traded company regardless of industry has its hands tied by three letters - SOX. Excluding specific sector requirements - HIPPA for medical, PCI for financial, FCPA, GLBA or (insert acronym here), Sarbanes-Oxley puts vague and onerous demands on public companies. Hell, you don't even have to be publicly traded. You could be a vendor to a publicly traded company and subject to it by proxy. Sarbanes-Oxley is notoriously ambiguous about what you actually have to DO to pass an audit. Entire industries have sprung up around it from hardware and software to wetware.

What's most amazing about it is that, I personally think implementing a DevOps philosophy across the board would make compliance EASIER. All change control is AUTOMATICALLY documented. Traditional access rules aren't an issue because no human actually logs onto servers for instance.

However in the end you have to convince the auditor that what you are doing matches with the script they have. In every company I've been at we've had the same workflow. It's like all the auditors went to the same fly by night school based on some infomercial: "Make big money as a SOX auditor. Call now for your free information packet!"

• Change is requested by person W
• Change is approved by X stake holders
• Change is approved by Y executive
• Change is performed by person Z
• The person who requested the change can't approve it.
• The person approving the change can't perform the actual work.
• So on and so forth.

Continuous deployment? Not gonna happen. It can't be done with that level of handcuffing.

Security Controls

Moving past the whole SOX issue, there are also security concerns that prevent automation. It's not uncommon for companies to have internal VPNs that have to be used to reach the production environment. That means the beautiful automated build system you have is good up until, say, QA. Preproduction and on requires manual access to even GET to the servers. This model is used in companies all over the world. Mandatory encryption requirements can further complicate things.

Corporate Hierarchy

I was recently asked what I found was the biggest roadblock to implementing the DevOps philosophy. In my standard roundabout way of thinking something through, I realized that the biggest roadblock is people. People with agendas. People who have control issues. People who are afraid of sharing knowledge for fear of losing some sort of role as "Keeper of the Knowledge". Those issues can extend all the way to the top of a company. There's also the fear of change. It's a valid concern and it's even MORE valid when a misstep can cost your company millions of dollars. You have no choice but to move slow and use what works because you know it works. It's not the most efficient but when it's bringing in the money, you can afford to throw bodies at the issue. You can afford to have 10 developers on staff focused on nothing but maintaining the current code base and another 10 working on new features.

The whole point of this long-winded post is to say "Don't write us off". We know. You're preaching to the choir. It takes baby steps and we have to pursue it in a way that works with the structure we have in place. It's great that you're a startup and don't have the legacy issues older companies have. We're all on the same team. Don't leave us behind.

Locked down!

So I started at the new company today. I was supposed to start Tuesday but there was some delay in my on-boarding. 

But that's neither here nor there.Here's the interesting thing. The company is a publicly traded financial services company. It's not enough to be publicly traded but to also be in financial services is like taking that giant cake of government scrutiny and slapping on another layer for fun.

Did I mention they're also international?

Anyway, I get my laptop and get logged in. This thing is locked down TIGHT. The kicker is that it's running Windows XP. Because of corporate policy, the only tools I'm allowed to install are cygwin, putty and winscp3. If I want to use an IDE, it's got to be eclipse. Nothing else is approved. Boot-level disk encryption. OS level disk encryption. Locked....down.

So I pretty much spent my entire afternoon trying to get cygwin running in something resembling usefulness. Mind you I haven't used Cygwin in AGES. I haven't used a windows machine for work in at least 6+ years. I've been fortunate enough to work for companies that allowed me to wipe the corporate install and run Linux as long as I didn't bother to ask for help with it. Where I ran into the next problem was with dealing with random cygwin issues.

So I hit google and start searching. Click the first result:

KA-BLOCK as Kevin Smith is fond of saying on twitter.

Blocked because it's a blog. Next result. Same thing. Finally I get a mailing list archive that isn't blocked and get most of the issues resolved. Meanwhile I've set off probably 20 alerts not because of any malicious activity but because I couldn't be sure if the search result would be a proxy violation or not. Hell, half the mirrors for cygwin I tried were blocked in the category of "Software Downloads". Really frustrating.

I finally get some semblance of a working system but I find myself wondering how I'm going to manage my standard workflow with this machine. It's going to be a challenge to say the least. In talking with my peers, it's pretty clear that they all have the same concerns and issues. Most of the time they work entirely in windowed screen sessions on one of the internal servers. This is fine by me but it's a big change in my workflow. I've been using the same keybinds for the past 6 years or so. I pretty much have to unlearn ALL of them because I can't use them on Windows.  The upshot is that I got gnome-terminator installed via cygwin ports. The hardest part was the fact that the homepage for gnome-terminator was blocked, you got it, because it was a "blog".

The point of this post is not to disparage the company in any way, shape, form or fashion. It got me wondering though how in the world people accomplish anything in environments like this? 

Forget the standard employee who uses email and the standard MS Office suite. What about developers who are developing code that runs on an entirely different OS. How many bugs and delays have companies had because the developer was unable to use an OS that mirrors that of the production environment. This particular company is a java shop. Java is a little more lax in this area but you still have oddities like "c:/path/to/file" that are entirely different on the server side. More so how many steps had to be injected in the workflow to get around that kind of issue. 

While I really HATE working on OSX at least it's more posix compliant than windows. My biggest headaches are how services are managed differently and the fact that it's not quite unix-alike for my tastes. It's like the uncanny valley.

I guess I'm feeling some trepidation because in addition to having to learn a new workflow - and a slower one at that - I'm also going to be working in Python. I'm excited about the work I'll be doing (DevOps - see my previous post about DevOps as a title) and the impact it will have but I also feel like I'm doubly behind - new workflow and a new language. The only thing that could make me more nervous is if the entire backend were Solaris - my weakest unix ;)

Anyway, I'll be fine. One upshot is that I AM allowed (as far as I was told) to run VirtualBox in host-only mode. Using some guest/host shared folder magic, I should be able to minimize the impact of the slower workflow.

The opposite of DevOps

I thought about whether or not to write this post but I think it's an interesting example of the kinds of problems that the DevOps methodology is trying to solve.

I turned in my two weeks notice with the paper on Friday. There were several reasons but none of them are a negative reflection on my employer. The role I was originally hired for was no longer valid and there wasn't a transition path because of platform changes in the back office. I could have stayed (and was asked to stay) but there wasn't anything long term where my skill set was useful. As with any key team member, meetings are called to discuss any outstanding issues, transition responsibilities and the like.

In preparation for the meeting today, I drew up a list my responsibilities at a macro level and then broke that down by task. As it's always been for me, that list spanned several "silos" in the traditional IT organization model. Here's a sample snippit:

Subversion	User Management, Repository Management
MySQL	User Management, Database Management, Performance Management
Linux	User Management, OS Configuration, Application Management, Code Management
Puppet	Configuration Management, Recipe Management
Ruby	VM Management, Gem Management

There were also entries for various internal applications that I've worked on (including some development) and supported from an operational perspective. Mind you, I was embedded as an operations guy with a specific development group in the organization. Sort of a DevOps-lite role.

What was really interesting about the meeting was how the lines were broken down. Literally lines were drawn as to where operations would stop supporting something and the group I was leaving would take over. Take the Linux example:

Application Management was intended to refer to software that was "standard" as a part of the distro. Things like Apache or MySQL. Code Management refers to our internally developed code (all Rails applications except for the sexy Sinatra webservice I wrote). In the end, the responsibilities were shifted divided like so:

- Operations Team supported up to the installation and configuration of Apache (including vhosts) with the exclusion of Passenger configuration.

- Passenger configuration and internal code would now be managed by the Development group. They would handle deployments themselves via Webistrano.

- MySQL? Passed on to those currently managing the MSSQL database servers.

That's the very definition of a silo'd IT infrastructure. Everything is thrown over the wall. At least the deploys remain with the development team. There is nothing "wrong" with this model of IT governance. It's not agile but many companies use it. Contrast that to the position I'll be starting on the 6th of July.

An explicit group is being formed inside the company called "DevOps". I know that at this point everyone is incredulous. I can hear it now; "You're doing it wrong!". Interestingly enough, I asked the same question during the interview process. We all know that DevOps is model and philosophy and not a title or department. You don't have two development teams - Agile and Waterfall. You have Developers and they use one methodology over the other. The same goes for Operations. The people I was interviewing with were cognizant of this fact as well. The reason the group is being called "DevOps" is strictly for organizational and political reasons. The goal of the team is to actually develop a set of operational and developmental processes,tools and guidelines that embody everything that DevOps represents.

This is being done inside of one division of the company for now with the Director of Development and Director of Operations essentially sharing a brain and heading things up. The work that this group does will establish and codify something that will be used throughout the rest of the company. We'll be doing development of tools, architecting systems and recommending/implementing solutions that will, among other things, define how the organization operates itself from an IT perspective. Additionally we'll be supporting this as any traditional operations team would but, for now, the group has a distinct title. My title is "Systems Architect" which nice and generic enough to apply to both traditional groups ;)

The only remotely distressing part of the whole thing is that I'll probably have to learn some Python. (tongue firmly in cheek). There's a point to be made that distribution vendors have settled on Python as the Lingua Franca of OS management - excluding things like Chef, Puppet and Nagios which have their own respective DSLs that abstract away much of the language they were written in. Yes, my precious Ruby will still be there when it comes to extending, say, Puppet. I'll probably still write quite a few service checks for Nagios in shell (should Nagios be the best fit).

But language wars aside, I think this gives a clear picture of exactly the types of problems the DevOps methodology is trying to solve. Agility and flexibility across all branches of IT produces a leaner, strong and faster organization that can spend less time "in the muck" (as John Willis likes to say) and start making the company more money.

Status of Riak support in Padrino

Back story

(skip to the bottom if you want the nitty gritty status)

So I got this wild hair up my butt a week or so ago. I wanted to get into the Padrino internals and actually contribute to an open source project. I've been using open source software for years. I've made a good living from it. I owe a lot to it.

My problem has always been the fact that I'm not a programmer by education nor by trade. I've always been a systems/architecture guy. Of course, a systems guy is a programmer at some level thanks to shell scripting, administrative scripts and the like. The DevOps philosophy makes that even more tangible by treating your systems in the same way a developer would treat his code. The most I've ever done previously in terms of contribution has been a bug fix here and there or documentation. All very valuable but in some capacity, not as rewarding.

So I've started mucking about with Sinatra. I used it to build some web service gateways for our production environment. That led me to Padrino. I noticed that Padrino didn't have ORM support for a few of the other schemaless/NoSQL databases so I figured it was a good way to contribute. I picked Riak out of the blue because the first episode of the ChangeLog Show I listened to had the Riak guys on there. Little did I know ;)

Riak/Ripple Status

Once I got a handle on the Ruby driver that Sean Cribbs is writing for Riak, I dove right in and forked the Padrino code base. Github makes it EXCEEDINGLY easy to be a contributer and I really think they've ushered in a new wave of open source development.

The Riak ruby driver comes in two flavors - riak-client and ripple. Riak-client is a more "basic" wrapper around riak operations - CRUD, link-walking, map/reduce. Ripple is the "next-gen" (imho) driver that borrows design from ActiveRecord, MongoMapper and DataMapper.

After poking a bit with riak-client, I decided that the ripple driver was much more "in-line" with the other ORMs supported in Padrino. I started to go whole hog before I realized that ripple had some missing functionality I was expecting. This was not anyone's fault but my own. Essentially this prevented me from using the ripple ORM in Padrino's admin interface. Not a big deal but it would have been nice to have.

Sean Cribbs was VERY responsive over twitter and let me know that the features I was looking for would be added. He opened an issue and last night, support for update_attribute/update_attributes was added. I grabbed the latest build and went to town on my local padrino-framework fork.

This is where I ran into another issue. Essentially, the other NoSQL ORMs that padrino supports use some tricks for handling unique keys. I know this is trying to shoehorn RDBMS ideas on top of schema-less databases.

So how do the other ORMs handle it?

• mongomapper/mongoid - supports 'validates_uniqueness_of' on model definitions
• couchrest - supports unique records via a map/reduce job

Nothing similar exists in the ripple driver "yet". I say yet because I fired off an email to Sean and got a very well-thought out response.

As a side note, Basho, you hired a good man to bear the title of "Developer Advocate".

Essentially Sean brought up a good point. Since you don't have "transactions or global consistency", there's no good way to guarantee that a key is unique.

I've essentially got two options if I want to continue down the path of support Padrino's admin with Riak - do the map/reduce route or do a check myself before calling save. I'm still deciding the best route to take.

Alternatively, I could simply not worry about it and forgo admin support in the ORM. It's not a deal breaker and it's not a requirement per the Padrino folks. It just would have been nice.

So what's the status?

Right now, using ActiveModel/ActiveSupport 3.0.0.beta4, Ripple from ripple/master and my fork of padrino-framework, I can create models and things work as expected.

The biggest headache for me is having to edit the Gemfile and append the versions for ripple and activesupport. I'm also working on adding gem version support to require_dependencies.

So you wanna try it yourself? Go right ahead. You'll need to grab my fork of the framework (I keep it up to date with master) and seancribbs/ripple. There are rake tasks to build and install those locally. I would HIGHLY suggest you create an RVM gemset. Additionally you'll need to grab ActiveSupport 3.0.0.beta4.

After that:

• padrino-gen project test -d ripple
• cd test
• edit Gemfile and append version 0.7.1 to the ripple line.
• bundle install
• padrino-gen model person name:string email:string phone:string

You should now have a working model using riak via ripple. Here's a Gist of me doing exactly that via padrino-console:

Anyway, I'm intent on getting ripple support back into upstream but I'm not going to make a pull request until ActiveSupport reaches 3.0. It feels half baked to have these extra steps to get it working and I'd really like to have admin support done before I do if possible.

Parsing Nagios Objects in Ruby Redux

When I was with MediaOcean, one of the projects I was working on was a stack of ruby code for parsing/writing nagios configs. This was in parallel with setting up puppet. I actually got pretty far along but then DDS had a RIF and I was back on the market.

I had to put the code aside and just recently pulled it out again. Some/most of it is pretty ugly. I had quite a bit done though.

So here I am "refactoring" (and I use that term VERY VERY loosely) the code base. So I'm happily parsing away - iterating over the cfg_file/cfg_dir entries and build a nice big hash to hold all the object definitions. I dump it to a YAML file for validation when I notice this nice bit of junk in my timeperiod dump:

Oh yeah...that looks right =/

For those who don't know, nagios object definitions are pretty straightforward. Here's an example:

Easy enough to parse, right? Object type named (host). A clear beginning and end denoted by curly braces. Object attributes in a seemingly key/value type markup. Some people smash the opening brace up against the object type definition but that's easily caught.

Iterating over the definition in Ruby is pretty straightforward (assuming a perfect example):

What becomes a problem is timeperiod defintions. The syntax for those is somewhat complicated. Looking back at the YAML fragment above, you can see why. For timeperiods, while the rightmost value is the actual time frame, the left part (usually a day of the week) can actually be a specific day. So if I wanted to create an entry for Christmas in the U.S., I would define it like so:

december 25 00:00-00:00


Things can be even MORE complicated if I wanted to handle non-date holidays:

monday 1 september 00:00-00:00 ; Labor Day (first Monday in September)
thursday -1 november 00:00-00:00 ; Thanksgiving (last Thursday in November)


Ugly, no? I searched high and low for the ability to split starting from the right and didn't find anything native. I had to resort to implementing my own rsplit method for String:

It works but I also can't use it across the board. If I do, I break things that were working (alias, service_description) that can have a space in the value.

I'm not even going to get into trying to convert timeperiod definitions into Date objects yet. That gives me cold sweats.

Aggressive LVM

So I ran into an interesting issue that I hadn't seen in quite some time. I've got a server running KVM. I use the Virtio driver and logical volumes to hold the OS and Data disks for each VM.

Every time I used the lvm tools on the hypervisor, I was seeing random vgs, pvs and lvs. It took me a second to noodle out the problem.

LVM was picking up the guest volume groups and activating them. For instance, many of my VMs have a data volume. Those were getting picked up and activated OUTSIDE of the vm because of the LVM scanning filter.

I know at one point, the default filter only scanned hd.* and sd.* as valid block devices but it looks like the default filter is now:

filter = [ "a/.*/" ]

which makes everything in /dev a valid target for scanning and activating.

Being that this particular box is 1U with a single hardware raid array, I knew that the only valid block devices would be /dev/sd.

I changed the filter line to this:

filter = [ "a|/dev/sd.*|", "r/.*/" ]

A quick vgscan -vvvv showed me that my guest volumes were no longer valid targets:

#filters/filter-regex.c:172 /dev/ajc-dd-dmzvirt1/isolv: Skipping (regex)

#label/label.c:160 /dev/sda5: lvm2 label detected

#filters/filter-regex.c:172 /dev/ajc-dd-dmzvirt1/vdisk-itgweb1: Skipping (regex)