Wednesday, February 24, 2010

Increasing frustration with managing Ruby on RHEL/CentOS

I am getting INCREASINGLY frustrated having to manage installations of Ruby on RHEL and its derivatives. It's to the point where I'm ready to shoot myself.

A bit of background:
RHEL5 and its offspring (referred from here on out as RHEL5) ship with a really dated version of Ruby (1.8.5). Yes there are various patches are backported but in the end, it's old. NOTHING works with it. No modern Gems work with it (for all intents and purposes). God forbid you try to run a modern Rails-based application on it.

The MAIN reason is one that plagues not only Ruby but Perl, PHP and Python on the same platform:

language-specific packaging repositories

With Perl, you have CPAN. With PHP, you have PECL/PEAR. With Python you have Eggs. All of these are somewhat antithetical to the OS package management system. Taking it even further, it's the same problem with Debian offshoots as well.

This isn't a problem for most end-users. They have the luxury of installing stuff wily-nilly. If it breaks, you just reinstall. However for those of tasked with maintaining systems in production environments, this is simply unacceptable. There is a requirement to keep systems in a consistent state. Software must be tested and work its way through the system before it gets to production.

OS updates are the same way. You don't simply have a cronjob that does a 'apt-get upgrade' or 'yum -y update'. Any smart company manages those packages using an internal repository. Packages aren't graduated to the production repository until they've been tested and verified not to break the system. A system MUST be maintainable. At one company, we actually managed our OWN application code via RPM. We had 3 repositories (dev, qa, production) and each class of server pointed to those repositories. As we built new RPMs, they were copied into each repository as needed for testing.

Now many people are probably thinking how convoluted that sounds but personally, I have no desire to have the financial future of my company at the hands of one broken upstream package. How would you feel if your banking site went offline because of the same issue? One broken upgrade can put someone out of a job.

So what happens if you say "screw it" and just use CPAN or rubygems or PECL outside of the system package management? Now what happens when you upgrade from something like PHP 5.2 to PHP 5.3? Now you have to rebuild any native extensions you have installed. Oh but look, the ImageMagick native extension requires that's compatible with version X of your interpreter also requires a new version of ImageMagick. But that version happens to be a major release instead of a point release. Let's hope nothing critical to the system requires a specific version of ImageMagick.

The specifics of the previous are somwhat hypothetical but everyone has run into the situation.

It's a difficult beast to tame. The closest I've come to having this process be somewhat simply is CPAN2RPM. However nothing that clean exists for Ruby. The best you can hope for is something like RVM (which in and of itself is pretty brilliant).

I still don't know the best way to manage Ruby on CentOS right now. There are countless repositories out there that are trying to do the same thing.

One thing we did at RBX which I've been doing currently is the concept of a "current" symlink. It's a bit convoluted but it works out for now and is pretty much the way that RVM works. Having this wrapped in a wrapper RPM keeps it managed via native package management but it's far from ideal.

Friday, February 12, 2010

Time to retire the beast

So my primary server at home for the LAN is an aging dual P3 with 1GB of memory (it won't even support any beyond that).

Currently it has the following disk layout:

/dev/mapper/DataVolGroup01-DataVol
184G 87G 88G 50% /data01
/dev/mapper/DataVolGroup00-DataVol00
880G 834G 1.5G 100% /data00

My mythtv backend (a somewhat freshly built X2 Dual Core Processor 4400+) has this disk layout:

/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth

I finally got the new drives I ordered from Amazon back in November - 3x1TB Hitachi Deskstars.

Now the mythbackend looks like this:
/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth
/dev/sdc1 932G 4.2M 932G 1% /myth2

The 880GB on lansrv is actually an external SATA array connected via Multilane to the server (4x500GB in a RAID10). I've been wanting to retire that box for some time as it draws WAY too much power for the service it provides. The mythbackend is becoming the hub for storing our archived records, pictures and music on the network.

However I didn't really have an easy place to dump that data on lansrv temporarily. Now I do. Here's the tentative plan:

1- Migrate all the lansrv data over to new drive on mythbackend.
2- Pull two of the 500GB drives out of the external array and put in the new 1TB drives.
3 - Mirror the new 1TB drives
4 - Rebuild the remaining 500GB drives as a mirrored pair.
5 - Stripe the two mirrors together
6 - Figure out what to do with the spare 1TB and 2 500GB drives.

Most likely the 1TB drive remaining will end up as a hot-spare for the 1TB mirror. Hitachi's aren't called Deathstars for nothing. The other two 500GB drives will probably go somewhere internal to the mythbackend box and become a mirrored pair as well and added to the new LV.

If I can manage to fit an eSATA card with 2 ports in the machine, I might consider moving all 4 500GB drives into a new array and adding it to the volume group.

Who knows? I'll figure something out but for now I had to have somewhere to dump the data temporarily while I shifted hardware and data around.

As an interesting side note, the 4 500GB drives have been running in the multilane enclosure since I worked at CLA. That's at least 5 years. I don't even remember what KIND of drive they were but I'm damn sure going to make note of it when I pull them.