DevOps and NoSQL - bad naming leads to confusion

I've recently started following a few new topics (where recently means over the past year). Both of them have the potential to be paradigm shifts and, unfortunately, both have somewhat vague names that evoke responses on both sides of the issue.

The one I'm going to focus on right now is DevOps. I intend on doing another post on NoSQL but that all depends on how much free time I can finagle between setting up the nest for baby number 2 and work projects.

Background

I should clarify my background because that plays a large part in how I perceive both of these issues. I'm a systems engineer. No, I don't have a degree in engineering but I wouldn't call the work I've done over the years any less than that. I've been the intermediary between DBAs and Developers. I've spent 20+ hours on my feet in a frigid datacenter racking servers. I've done high-level architecture of disparate system integration. I've done low-level implementation of disparate system integration. I've been up at 4AM to do deploys of new code during the 30 minute maintenance window. I've been the guy getting the pages and been the guy calling people who we're supposed to get the pages.

I've been in big shops and small shops. I've been responsible for systems that pass millions of dollars and systems that are critical to education.

I don't say all this to toot my own horn. It's just background that is relevant to the discussion.

DevOps

So what's this DevOps thing that people keep throwing around? Well there are tons of opinions and all of them are like certain sphincter muscles. Not one is entirely on the money but the background work has been done here:

http://stochasticresonance.wordpress.com/2010/03/26/devops-misnamed/ (which, coincidentally prompted this post)

So what is it? I think at the core it's about closely integrating the "SysOp" silo with the "Developer" silo as a methodology. But why is this important?

SysOps have always been apart from the rest of the IT department in a sense. While many groups have frequent overlapping areas, the operations team has the final responsibility. As I like to put it, they're the folks getting the phone call. Unless the organization is small, most developers aren't even in the loop unless a bug report is filed after an outage. As it was put elsewhere, many times software is thrown "over the wall" to be deployed. But why is this? I think that's key to the whole issue.

Roles, Responsibilities and Titles

I'm not a stickler for titles. I've held many over the years for Administrator to Director. In one interesting case, I was given a title (and the subsequent responsibility) simply for the purpose of interacting with a client who had firm opinions about only interfacing with someone at the same level. This didn't take away any responsibilities; only added to them. Titles, roles and responsibilities are all different things.

In this way, the organizational title for "IT Operations" denotes a clear differentiator from "Developer". There are certain expectations from your operations team. Production stays stable, for instance. Many times the goals of the Operations team are in direct opposition to those of the Development team. Make no mistake, however. The developers are part of revenue generation while those of operations are not. Operations exists as fire fighters. If Operations is doing its job properly, they aren't actually doing much of their primary responsibility. They have quite a bit of downtime.

So why is there a need for a DevOps movement?

I think on one hand, there is an increasing frustration from the end-user (in this case development) in its interaction with operations. Development methodologies are changing rapidly. Some changes are for the better (less bugs, more testing) while others create friction with how a production environment operates (frequent releases). Another aspect is people transitioning from one role to the other. You have people moving into development from an operations background and vice versa. People change. They discover that they enjoy X more than Y. With each of these transitions, a mindset and attitude is brought along. An Ego.

The developer who moves into production operations laments the slow sluggish pace at which things move. The operations guy who moves into development loves the fast and fluid nature of Agile development. Both feel the need to reconcile the two worlds thinking they can impart some sort of wisdom from one side of which the other was not aware.

Additionally, in times where the leanest team that is first to market often wins many people are wearing multiple hats. See the rise of IaaS (Infrastructure as a Service), Amazon Web Services, NoSQL and other technologies where traditional roles are eliminated.

Both sides have a lot to learn from each other and both sides need to understand the constraints each team has. This is where I feel DevOps has the most to offer as an ideal. Integrating operations into development and letting development be a part of operations. The specifics are still up in the air but I think there are some key areas that each side needs to understand about the other. I'll follow those up in the next post to for logical grouping purposes.

As always, comments are welcome!

Code on Github

I'm working on pushing all my code snippets I've developed over the years to Github. Right now I have the Ruby-EWS and Ruby-Downtime stuff uploaded. I'll get the rest over as I can sanitize it.

http://github.com/lusis

Ummm...thank you Google Voice?

I just got this wonderful bit of translation from Google Voice on a voicemail:

I just came across your resume on Monster. Here, and had a position available on the line area thought it'd be a good fit for you sexually a infrastructure design related positions in the offer an area love the chance to discuss with you at your earliest convenience

Mind you, I love infrastructure design but I'd hate to see how I could fit it into my sex life. The job WAS with Cox....

Usability and Performance testing in the Analog world

Right now, our offices are located downtown. The AJC has been in this building since 1972 (I think). We're soon to be moving outside the Perimeter to new offices. Trust me, this is an important fact.

I started as a contractor here back in November of last year. I had an option for parking but at the time I didn't know how long I would be here (contracting can be volatile) and didn't know if I would plan on switching to Marta. Combined with the fact that the company WAS planning on moving offices, I decided not to get a parking pass.

Instead, I opted to use the pay parking lots behind the office. These spots are $3 a day or a monthly option of $40. This was about the same as the AJC parking pass so it was something of a wash. I went permanent with the paper in the beginning of February. Our move is scheduled at the beginning of April and parking passes are no longer available so I *HAVE* to use the parking vendor out back.

Of course this is when the parking company, Central Parking Systems decided to swap out the working payment terminals with new ones. The old ones, while a bit worn around the collar actually worked well. People understood them and things moved along quickly. I'm trying to find a picture of the old ones but the new ones look like the picture above.

Since these have been put in place, the lines to get a parking pass have been super long. I thought this was a familiarity issue. The new terminals are smaller, the screen is less readable and from a usability perspective, it's a pain in the ass. People have to stoop down to get the money in the machine and GOD FORBID you pay with a credit card. It's one of the quick swipe methods and from the angle you can NEVER "quickly remove your card" without also pulling up as you pull it out. This obviously isn't quick enough so you have to start again.

Well this morning I found out there's ANOTHER problem. There were two parking attendants watching people use the machines. Finally one guy spoke up and said the following:

"Folks, this terminal is running really slow and overloading the computer system. If you use X terminal or Y terminal, they're running really fast"

Let me position those other two terminals for you in relation to this one. X terminal is closest but by the time I get to it, pay and get back to my car (which I parked near THIS terminal I would already be done paying if I stayed in line.

The OTHER terminal would require me to get in my car, drive to it and come back to park near where I actually work.

So how slow were these new machines running? They were running so slow that they wouldn't actually take money. When I got up there, I decided to ask the guy how I would pay for a monthly option here. This is where it gets good:

"You can't buy monthly passes here anymore. Just go online to blahblahblahblah.com and register as a vendor. You get much cheaper parking and other rewards."

Are you kidding me? I'm only going to be here for another month. I'm not a "vendor" and I don't want to go to a stupid website to pay for freaking parking when I could pay just fine up until recently.

Where do I start with the screwups?

If they had ANY business intelligence or metrics, they would have known that this particular kiosk is the busiest one in the deck. It's near the best parking and it's centrally located. The fact that this one terminal was causing these kinds of issues is unacceptable. Load testing isn't strictly digital. Look at any major downtown event. The same work goes into capacity planning there from parking to traffic that any major website would undergo preparing for the Christmas holiday. The kiosk was running so slow that it was actually refusing to accept dollar bills. I decided to pay with my credit card and my ticket had printed while the screen still said processing payment.

But even IF the system could keep up with the load generated, having someone actually go onsite and see HOW users were using the damn things would have shown them the ergonomic bottlenecks.

Something as simple as a parking lot was foiled by lack of performance and usability testing. Amazing.

EDIT: I found a picture of what the old terminal looked like. It's not "pretty" but it worked and was ergonomically superior.

Finally getting to dig into Puppet

Well I'm finally getting to dig into Puppet at a professional level. I've been handed the keys to establishing the Puppet infrastructure. Mixed Linux/Solaris clients. Beyond the hassle of building up-to-date RPMs for CentOS/RHEL, everything else is going smoothly.

I've got all of my configuration in version control and handy script to create new modules from templates. I'm already pushing out OS specific configurations as well as OS version specific stuff. Augeas is AMAZING. I don't know how I never found/saw it before.

We're also pushing out Splunk (Cox is a huge Splunk customer) across the board. I'm still undecided about its value but I haven't really looked at it since it first came out. It just seemed, at the time, like a huge overhead for a problem that was already tackled (syslogging).

My next step is deciding how to integrate everything into my Kickstart scripts. There's a TON of options out there. I'm really trying to avoid setting up Cobbler but I might be at the end of what can be done with Kickstart already.

I'm still frustrated with managing current versions of Ruby and gems with RPM but gem2rpm has made some of that easier. I'm not responsible for the Solaris boxen so I have no idea how those guys plan on managing those bad boys.

Increasing frustration with managing Ruby on RHEL/CentOS

I am getting INCREASINGLY frustrated having to manage installations of Ruby on RHEL and its derivatives. It's to the point where I'm ready to shoot myself.

A bit of background:

RHEL5 and its offspring (referred from here on out as RHEL5) ship with a really dated version of Ruby (1.8.5). Yes there are various patches are backported but in the end, it's old. NOTHING works with it. No modern Gems work with it (for all intents and purposes). God forbid you try to run a modern Rails-based application on it.

The MAIN reason is one that plagues not only Ruby but Perl, PHP and Python on the same platform:

language-specific packaging repositories

With Perl, you have CPAN. With PHP, you have PECL/PEAR. With Python you have Eggs. All of these are somewhat antithetical to the OS package management system. Taking it even further, it's the same problem with Debian offshoots as well.

This isn't a problem for most end-users. They have the luxury of installing stuff wily-nilly. If it breaks, you just reinstall. However for those of tasked with maintaining systems in production environments, this is simply unacceptable. There is a requirement to keep systems in a consistent state. Software must be tested and work its way through the system before it gets to production.

OS updates are the same way. You don't simply have a cronjob that does a 'apt-get upgrade' or 'yum -y update'. Any smart company manages those packages using an internal repository. Packages aren't graduated to the production repository until they've been tested and verified not to break the system. A system MUST be maintainable. At one company, we actually managed our OWN application code via RPM. We had 3 repositories (dev, qa, production) and each class of server pointed to those repositories. As we built new RPMs, they were copied into each repository as needed for testing.

Now many people are probably thinking how convoluted that sounds but personally, I have no desire to have the financial future of my company at the hands of one broken upstream package. How would you feel if your banking site went offline because of the same issue? One broken upgrade can put someone out of a job.

So what happens if you say "screw it" and just use CPAN or rubygems or PECL outside of the system package management? Now what happens when you upgrade from something like PHP 5.2 to PHP 5.3? Now you have to rebuild any native extensions you have installed. Oh but look, the ImageMagick native extension requires that's compatible with version X of your interpreter also requires a new version of ImageMagick. But that version happens to be a major release instead of a point release. Let's hope nothing critical to the system requires a specific version of ImageMagick.

The specifics of the previous are somwhat hypothetical but everyone has run into the situation.

It's a difficult beast to tame. The closest I've come to having this process be somewhat simply is CPAN2RPM. However nothing that clean exists for Ruby. The best you can hope for is something like RVM (which in and of itself is pretty brilliant).

I still don't know the best way to manage Ruby on CentOS right now. There are countless repositories out there that are trying to do the same thing.

One thing we did at RBX which I've been doing currently is the concept of a "current" symlink. It's a bit convoluted but it works out for now and is pretty much the way that RVM works. Having this wrapped in a wrapper RPM keeps it managed via native package management but it's far from ideal.

Time to retire the beast

So my primary server at home for the LAN is an aging dual P3 with 1GB of memory (it won't even support any beyond that).

Currently it has the following disk layout:

/dev/mapper/DataVolGroup01-DataVol
184G 87G 88G 50% /data01
/dev/mapper/DataVolGroup00-DataVol00
880G 834G 1.5G 100% /data00

My mythtv backend (a somewhat freshly built X2 Dual Core Processor 4400+) has this disk layout:

/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth

I finally got the new drives I ordered from Amazon back in November - 3x1TB Hitachi Deskstars.

Now the mythbackend looks like this:
/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth
/dev/sdc1 932G 4.2M 932G 1% /myth2

The 880GB on lansrv is actually an external SATA array connected via Multilane to the server (4x500GB in a RAID10). I've been wanting to retire that box for some time as it draws WAY too much power for the service it provides. The mythbackend is becoming the hub for storing our archived records, pictures and music on the network.

However I didn't really have an easy place to dump that data on lansrv temporarily. Now I do. Here's the tentative plan:

1- Migrate all the lansrv data over to new drive on mythbackend.
2- Pull two of the 500GB drives out of the external array and put in the new 1TB drives.
3 - Mirror the new 1TB drives
4 - Rebuild the remaining 500GB drives as a mirrored pair.
5 - Stripe the two mirrors together
6 - Figure out what to do with the spare 1TB and 2 500GB drives.

Most likely the 1TB drive remaining will end up as a hot-spare for the 1TB mirror. Hitachi's aren't called Deathstars for nothing. The other two 500GB drives will probably go somewhere internal to the mythbackend box and become a mirrored pair as well and added to the new LV.

If I can manage to fit an eSATA card with 2 ports in the machine, I might consider moving all 4 500GB drives into a new array and adding it to the volume group.

Who knows? I'll figure something out but for now I had to have somewhere to dump the data temporarily while I shifted hardware and data around.

As an interesting side note, the 4 500GB drives have been running in the multilane enclosure since I worked at CLA. That's at least 5 years. I don't even remember what KIND of drive they were but I'm damn sure going to make note of it when I pull them.