The opposite of DevOps

I thought about whether or not to write this post but I think it's an interesting example of the kinds of problems that the DevOps methodology is trying to solve.

I turned in my two weeks notice with the paper on Friday. There were several reasons but none of them are a negative reflection on my employer. The role I was originally hired for was no longer valid and there wasn't a transition path because of platform changes in the back office. I could have stayed (and was asked to stay) but there wasn't anything long term where my skill set was useful. As with any key team member, meetings are called to discuss any outstanding issues, transition responsibilities and the like.

In preparation for the meeting today, I drew up a list my responsibilities at a macro level and then broke that down by task. As it's always been for me, that list spanned several "silos" in the traditional IT organization model. Here's a sample snippit:

Subversion	User Management, Repository Management
MySQL	User Management, Database Management, Performance Management
Linux	User Management, OS Configuration, Application Management, Code Management
Puppet	Configuration Management, Recipe Management
Ruby	VM Management, Gem Management

There were also entries for various internal applications that I've worked on (including some development) and supported from an operational perspective. Mind you, I was embedded as an operations guy with a specific development group in the organization. Sort of a DevOps-lite role.

What was really interesting about the meeting was how the lines were broken down. Literally lines were drawn as to where operations would stop supporting something and the group I was leaving would take over. Take the Linux example:

Application Management was intended to refer to software that was "standard" as a part of the distro. Things like Apache or MySQL. Code Management refers to our internally developed code (all Rails applications except for the sexy Sinatra webservice I wrote). In the end, the responsibilities were shifted divided like so:

- Operations Team supported up to the installation and configuration of Apache (including vhosts) with the exclusion of Passenger configuration.

- Passenger configuration and internal code would now be managed by the Development group. They would handle deployments themselves via Webistrano.

- MySQL? Passed on to those currently managing the MSSQL database servers.

That's the very definition of a silo'd IT infrastructure. Everything is thrown over the wall. At least the deploys remain with the development team. There is nothing "wrong" with this model of IT governance. It's not agile but many companies use it. Contrast that to the position I'll be starting on the 6th of July.

An explicit group is being formed inside the company called "DevOps". I know that at this point everyone is incredulous. I can hear it now; "You're doing it wrong!". Interestingly enough, I asked the same question during the interview process. We all know that DevOps is model and philosophy and not a title or department. You don't have two development teams - Agile and Waterfall. You have Developers and they use one methodology over the other. The same goes for Operations. The people I was interviewing with were cognizant of this fact as well. The reason the group is being called "DevOps" is strictly for organizational and political reasons. The goal of the team is to actually develop a set of operational and developmental processes,tools and guidelines that embody everything that DevOps represents.

This is being done inside of one division of the company for now with the Director of Development and Director of Operations essentially sharing a brain and heading things up. The work that this group does will establish and codify something that will be used throughout the rest of the company. We'll be doing development of tools, architecting systems and recommending/implementing solutions that will, among other things, define how the organization operates itself from an IT perspective. Additionally we'll be supporting this as any traditional operations team would but, for now, the group has a distinct title. My title is "Systems Architect" which nice and generic enough to apply to both traditional groups ;)

The only remotely distressing part of the whole thing is that I'll probably have to learn some Python. (tongue firmly in cheek). There's a point to be made that distribution vendors have settled on Python as the Lingua Franca of OS management - excluding things like Chef, Puppet and Nagios which have their own respective DSLs that abstract away much of the language they were written in. Yes, my precious Ruby will still be there when it comes to extending, say, Puppet. I'll probably still write quite a few service checks for Nagios in shell (should Nagios be the best fit).

But language wars aside, I think this gives a clear picture of exactly the types of problems the DevOps methodology is trying to solve. Agility and flexibility across all branches of IT produces a leaner, strong and faster organization that can spend less time "in the muck" (as John Willis likes to say) and start making the company more money.

Parsing Nagios Objects in Ruby Redux

When I was with MediaOcean, one of the projects I was working on was a stack of ruby code for parsing/writing nagios configs. This was in parallel with setting up puppet. I actually got pretty far along but then DDS had a RIF and I was back on the market.

I had to put the code aside and just recently pulled it out again. Some/most of it is pretty ugly. I had quite a bit done though.

So here I am "refactoring" (and I use that term VERY VERY loosely) the code base. So I'm happily parsing away - iterating over the cfg_file/cfg_dir entries and build a nice big hash to hold all the object definitions. I dump it to a YAML file for validation when I notice this nice bit of junk in my timeperiod dump:

Oh yeah...that looks right =/

For those who don't know, nagios object definitions are pretty straightforward. Here's an example:

Easy enough to parse, right? Object type named (host). A clear beginning and end denoted by curly braces. Object attributes in a seemingly key/value type markup. Some people smash the opening brace up against the object type definition but that's easily caught.

Iterating over the definition in Ruby is pretty straightforward (assuming a perfect example):

What becomes a problem is timeperiod defintions. The syntax for those is somewhat complicated. Looking back at the YAML fragment above, you can see why. For timeperiods, while the rightmost value is the actual time frame, the left part (usually a day of the week) can actually be a specific day. So if I wanted to create an entry for Christmas in the U.S., I would define it like so:

december 25 00:00-00:00


Things can be even MORE complicated if I wanted to handle non-date holidays:

monday 1 september 00:00-00:00 ; Labor Day (first Monday in September)
thursday -1 november 00:00-00:00 ; Thanksgiving (last Thursday in November)


Ugly, no? I searched high and low for the ability to split starting from the right and didn't find anything native. I had to resort to implementing my own rsplit method for String:

It works but I also can't use it across the board. If I do, I break things that were working (alias, service_description) that can have a space in the value.

I'm not even going to get into trying to convert timeperiod definitions into Date objects yet. That gives me cold sweats.

Code on Github

I'm working on pushing all my code snippets I've developed over the years to Github. Right now I have the Ruby-EWS and Ruby-Downtime stuff uploaded. I'll get the rest over as I can sanitize it.

http://github.com/lusis

Finally getting to dig into Puppet

Well I'm finally getting to dig into Puppet at a professional level. I've been handed the keys to establishing the Puppet infrastructure. Mixed Linux/Solaris clients. Beyond the hassle of building up-to-date RPMs for CentOS/RHEL, everything else is going smoothly.

I've got all of my configuration in version control and handy script to create new modules from templates. I'm already pushing out OS specific configurations as well as OS version specific stuff. Augeas is AMAZING. I don't know how I never found/saw it before.

We're also pushing out Splunk (Cox is a huge Splunk customer) across the board. I'm still undecided about its value but I haven't really looked at it since it first came out. It just seemed, at the time, like a huge overhead for a problem that was already tackled (syslogging).

My next step is deciding how to integrate everything into my Kickstart scripts. There's a TON of options out there. I'm really trying to avoid setting up Cobbler but I might be at the end of what can be done with Kickstart already.

I'm still frustrated with managing current versions of Ruby and gems with RPM but gem2rpm has made some of that easier. I'm not responsible for the Solaris boxen so I have no idea how those guys plan on managing those bad boys.

Sold on unit testing

I'll tell anyone who asks that I'm NOT a programmer. I've always understood the value in unit tests but considered them somewhat limited in scope. I mean it's pretty much impossible to write a unit test for every possible scenario and (IMHO) impossible to write a unit test that simulates the flow of a user from application login to data entry to logout using the entire infrastructure (i.e. client browser -> through load balancer -> through app server -> database -> back).

Because of those reasons, while I understood the need for unit tests for "stupid stuff", the fact that you had to add unit tests for each bug that popped up over the life of a product felt "odd". Of course this wouldn't stop me from bitching at a developer who's lack of a unit test caused me to restore a filesystem because it would ascend to the parent directory if the file or directory it was trying to delete did not exist! (True story).

So here I am writing my ruby nagios library and realizing that I'm duplicating A LOT of code writing quick little scripts to test my output. I decide to write unit tests for each of my classes. I found an interesting little article on about.com talking about learning ruby via unit tests. It was an interesting concept and one that made sense to me. So I start with my contact class.

Basically all of the classes that exist for a specific nagios object (contact,service,host) have their own class. One method they have is called "hashify". It's different for each object type but essentially creates a nested hash like so:

{
"nagiosadmin"=>
{"service_notification_period"=>"24x7",
"host_notification_options"=>"d,u,r",
"service_notifications_enabled"=>nil,
"host_notification_enabled"=>nil,
"pager"=>nil,
"service_notification_commands"=>"notify-service-by-email",
"host_notification_period"=>"24x7",
"alias"=>"Nagios Admin",
"host_notification_commands"=>"notify-host-by-email",
"service_notification_options"=>"w,u,c,r",
"email"=>"root@localhost"},
"johnv"=>
{"service_notification_period"=>"24x7",
"host_notification_options"=>"d,u,r",
"service_notifications_enabled"=>nil,
"host_notification_enabled"=>nil,
"pager"=>nil,
"service_notification_commands"=>"notify-service-by-email",
"host_notification_period"=>"24x7",
"alias"=>"John E. Vincent",
"host_notification_commands"=>"notify-host-by-email",
"service_notification_options"=>"w,u,c,r",
"email"=>"root@localhost"}
}

So I start writing unit tests. I test that hashify returns a Hash. I test that I have a hash called 'johnv'. Then I start testing for each of the attributes for johnv. This is where it all falls apart. Attribute email is always returning nil even though I can see right in the debugger that it's set to 'root@localhost'.

After about 30 minutes of dicking around I finally realize that, being a Ruby beginner who has used some of the higher-level Ruby stuff for quickies without actually getting into the meat of the language (Rails, Ruport), I was totally confused on the usage of ":" in hashes much less anywhere else in Ruby. Two minutes after that, I stopped throwing symbols all over the place in my classes when building a hash ;)

My unit test passed! I continue to write the remaining tests and run them. I get a failure. I look in detail and realize that in my original Contact class, I had a typo in the hashify method that defined a key as 'host_notifications_period' instead of 'host_notification_period'!

So yeah, I'm sold on unit tests and I'm not writing another lick of code until I finish writing them for the classes I have now.

Nagios Configuration Tips Part 1 - cfg_dir

One of the key problems I see with people using Nagios is the fact that they add EVERYTHING into a single file for each type of object. This is obviously fine when you have only a few systems to monitor but starts to become unwieldy when you have 10, 20 or even 100 servers to monitor. This article is to show you what I consider to be a very flexible file system layout for your Nagios configurations. The end result is a configuration structure that allows you to easily jump to the source of a configuration problem and encourages the use of object templating. This is the first in a multi part series of Nagios Configuration Tips.

nagios.cfg
Since we are only concerned about configuration files, I'm only going to paste the relevant lines from the nagios.cfg:

cfg_file=/etc/nagios/objects/commands.cfg
cfg_file=/etc/nagios/objects/contacts.cfg
cfg_file=/etc/nagios/objects/contactgroups.cfg
cfg_file=/etc/nagios/objects/timeperiods.cfg
cfg_dir=/etc/nagios/objects/(organization)

(organization is really just an arbitrary directory to logically group a collection of objects. )

The explicitly defined files are where we keep the more "global" stuff. Objects that are shared across all configs for all organizations/domains or system-level stuff.

commands.cfg
This file is a standard nagios cfg file containing a list of command definitions. In this case, I'm only keeping the stuff that applies to the local system (check_local_*) and the notify-service-by-email/notify-host-by-email defines.
contacts.cfg/contactgroups.cfg/timeperiods.cfg
contacts.cfg has a definition for a nagiosadmin account, contactgroups.cfg has a definition for a testgroup contactgroup and timeperiods.cfg has a definition for 24x7.

And that's it for the base configuration files. Notice that there really isn't much in them. As you'll see, all of the heavy lifting will be done by the stuff in the cfg_dir.

cfg_dir
So now let's look at what we have in our cfg_dir.

For this example, we're going to assume that we have two areas that we need to monitor, systems and processes. Let's also use the fictional company name of widgetcorp. Systems are exactly what they sound like. This is where we monitor things at the host level like reachability, loadavg and disk utilization. Process would be things that we monitor at a higher level like database locks, http connections, jvm usage or even specific business processes like user logins, outstanding orders shipments or even the date of the last warehouse load.

So let's create the following directory structure under /etc/nagios/objects/:

widgetcorp
 /processes
 /systems

Now before we write any configs, let's think about how we want to categorize our these new directories. At widgetcorp, we have three classes of systems - database and application. Let's create those directories under systems:

widgetcorp
 /systems
 /database

Being the sane company that they are, widgetcorp was smart enough to invest in a minimal level of high availability. This environment consists of 4 servers using round-robin DNS to balance between application servers and using linux-ha to provide access to the database servers. Notice that I've not yet defined WHAT application server is running or what dbms is being used. These systems are named app01,app02,dbs01,dbs02.

systems
 /application
  /app01
  /app02
 /database
  /dbs01
  /dbs02

As far as the process monitoring goes, we have two types of "processes" we need to concern ourselves with - application response time and database server availability.

processes
 /database
 /application

Sidebar: One thing you'll note is a particular attitude I have. I consider physical systems "interchangeable". I don't want to tie the fact that I run MySQL on db01.widgetcorp.com to the status of db01.widgetcorp.com as a whole. What if we're operating using MySQL Proxy or operating a Linux-HA MySQL cluster or using HACMP on AIX for DB2? The availability of a single system is really quite independent from the higher level availability of the service that grouping of systems provides. Our application servers would never be configured to talk to JUST db01 but instead would use the name of the mysql proxy server or the VIP assigned to the HA cluster - db.widgetcorp.com. Using service dependencies, you can still tie the polling process of db.widgetcorp to a specific server or uplink.

Back to the layout. We now get to discuss what programs are actually installed on each server because the facts that we need are from those programs.

In the case of app01 and app02, they are both running tomcat and apache with mod_jk. All traffic coming in from the internet is balanced between each apache server on port 80 talking to a localhost-listening tomcat instance on the jk connector port. These details aren't really important for the purposes of this document except to say that our customers don't go to app01.widgetcorp or app02.widgetcorp but instead www.widgetcorp.com.

As for the database, the databases are using Linux-HA and MySQL replication to talk to the database server. Each server has a VIP assigned which is aliased to dbrw.widgetcorp and dbro.widgetcorp. The current MASTER in the replication process is assigned the VIP for dbrw and the SLAVE is assigned the VIP for dbro. When one of the systems fails, the other assumes the role of BOTH as the application performs lookups against dbro while doing actual inserts and updates against dbrw.

All of the above means our directory structure now looks like this:

widgetcorp/
 systems/
  application/
   app01/
   app02/
  database/
   dbs01/
   dbs02/
 processes/
  database/
   dbrw/
   dbro/
  application/
   www/

And that's it for the first part of this post. The next post will get into the actual naming, location and content of the configuration files. Please feel free to leave comments and let me know your thoughts. Please also be aware that I'm intentionally trying to be generic in these examples. Don't get too caught up in the fictional implementation of the company. I'm aware of the limitations of both round-robin DNS as well as the MySQL implementation. I only picked these as high-level examples.

Thanks and I look forward to the comments!

An interesting log entry

A few (okay 2) years ago I wrote a small page on my website documenting how to get container statistics out of Websphere 5.x from the command-line. To this day, awstats shows that, and the page on monitoring DB2 UDB with Nagios as the single biggest hit pages. I like to occasionally take note of where people are coming from to look at the information.

In looking at the logs this morning, I see the standard incoming search result from google for something like "websphere + jdbc + command-line" or "websphere + jdbc + monitoring" but what caught my eye was something in the USERAGENT string:

"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20080211 Fedora/2.0.0.12-1.oc2 (CK-IBM) (CK-IBM) Firefox/2.0.0.12"

Was someone from IBM looking at *MY* notes? Why, yes, they were:

torolab.ibm.com

Thanks for visiting guys. I'm even more honored that it was someone from Toronto.

EDIT: I just noticed the CK part of the IBM thing. I finally remembered why that looked familiar. Does IBM have an internal client customization kit for Firefox? For those not in the know, here's a link to the mozilla wiki entry on the concept/program:

http://wiki.mozilla.org/CCK:Overview

Basically it was the repackager for Netscape that allowed ISP branding, locked settings and a host of other features. I wasn't aware that there was one for Firefox but I haven't really needed one since I left CLA.

Nagios and Ruby

So as I mentioned in my post here, I've been working on a sort of Nagios toolbox in Ruby. I never really found anything at large about it and honestly, most people are doing stuff in shell scripts or with Perl.

I don't honestly blame them but I wanted a project to force me to learn Ruby better and I had a specific need. I could have this written already in Perl or even straight Bash but, again, I wanted to learn Ruby better.

As I said in my other post, here are the design goals starting out:

• Import existing configs (objects only. Not concerned about the operational parameters)
• Parse existing objects (apart from reading the configurations)
• Create objects (hosts,services,contacts, *groups, etc...)
• Write new configs
• Enforce/Support Templating

I'm not writing a new front end. I'm not writing a global configuration suite for Nagios. My goal is object management. Many of the defaults in nagios.cfg work OOB. However, any time I spend in #nagios, the MAJORITY of the questions are related to actually defining what to monitor and how to monitor it. The example included are pretty solid but they're VERY verbose to the point of being confusing. On the flip side, someone like me who can write a .cfg in his sleep, is still using vi to manage entries.

At my new company, they are BIG on automation. I mean EVERYTHING is automated from an SA perspective. The only exception comes with NEW environments. This is still a mishmash of manual processes and checklists with a slathering of the above automation. One of my first things I was tasked with when I walked through the door was to modernize the existing Nagios configuration. This project is part of that. I need to integrate anything I do seamlessly into the existing automation for it to be accepted as process.

So consider this a "declaration of intent to proceed" or somesuch politispeek. Feel free to comment on my lack of Ruby skill and where you think I can change things.

Here's the code I have so far:

http://dev.lusis.org/nagios/ruby/