Friday, March 5, 2010

Finally getting to dig into Puppet

Well I'm finally getting to dig into Puppet at a professional level. I've been handed the keys to establishing the Puppet infrastructure. Mixed Linux/Solaris clients. Beyond the hassle of building up-to-date RPMs for CentOS/RHEL, everything else is going smoothly.

I've got all of my configuration in version control and handy script to create new modules from templates. I'm already pushing out OS specific configurations as well as OS version specific stuff. Augeas is AMAZING. I don't know how I never found/saw it before.

We're also pushing out Splunk (Cox is a huge Splunk customer) across the board. I'm still undecided about its value but I haven't really looked at it since it first came out. It just seemed, at the time, like a huge overhead for a problem that was already tackled (syslogging).

My next step is deciding how to integrate everything into my Kickstart scripts. There's a TON of options out there. I'm really trying to avoid setting up Cobbler but I might be at the end of what can be done with Kickstart already.

I'm still frustrated with managing current versions of Ruby and gems with RPM but gem2rpm has made some of that easier. I'm not responsible for the Solaris boxen so I have no idea how those guys plan on managing those bad boys.

Wednesday, February 24, 2010

Increasing frustration with managing Ruby on RHEL/CentOS

I am getting INCREASINGLY frustrated having to manage installations of Ruby on RHEL and its derivatives. It's to the point where I'm ready to shoot myself.

A bit of background:
RHEL5 and its offspring (referred from here on out as RHEL5) ship with a really dated version of Ruby (1.8.5). Yes there are various patches are backported but in the end, it's old. NOTHING works with it. No modern Gems work with it (for all intents and purposes). God forbid you try to run a modern Rails-based application on it.

The MAIN reason is one that plagues not only Ruby but Perl, PHP and Python on the same platform:

language-specific packaging repositories

With Perl, you have CPAN. With PHP, you have PECL/PEAR. With Python you have Eggs. All of these are somewhat antithetical to the OS package management system. Taking it even further, it's the same problem with Debian offshoots as well.

This isn't a problem for most end-users. They have the luxury of installing stuff wily-nilly. If it breaks, you just reinstall. However for those of tasked with maintaining systems in production environments, this is simply unacceptable. There is a requirement to keep systems in a consistent state. Software must be tested and work its way through the system before it gets to production.

OS updates are the same way. You don't simply have a cronjob that does a 'apt-get upgrade' or 'yum -y update'. Any smart company manages those packages using an internal repository. Packages aren't graduated to the production repository until they've been tested and verified not to break the system. A system MUST be maintainable. At one company, we actually managed our OWN application code via RPM. We had 3 repositories (dev, qa, production) and each class of server pointed to those repositories. As we built new RPMs, they were copied into each repository as needed for testing.

Now many people are probably thinking how convoluted that sounds but personally, I have no desire to have the financial future of my company at the hands of one broken upstream package. How would you feel if your banking site went offline because of the same issue? One broken upgrade can put someone out of a job.

So what happens if you say "screw it" and just use CPAN or rubygems or PECL outside of the system package management? Now what happens when you upgrade from something like PHP 5.2 to PHP 5.3? Now you have to rebuild any native extensions you have installed. Oh but look, the ImageMagick native extension requires that's compatible with version X of your interpreter also requires a new version of ImageMagick. But that version happens to be a major release instead of a point release. Let's hope nothing critical to the system requires a specific version of ImageMagick.

The specifics of the previous are somwhat hypothetical but everyone has run into the situation.

It's a difficult beast to tame. The closest I've come to having this process be somewhat simply is CPAN2RPM. However nothing that clean exists for Ruby. The best you can hope for is something like RVM (which in and of itself is pretty brilliant).

I still don't know the best way to manage Ruby on CentOS right now. There are countless repositories out there that are trying to do the same thing.

One thing we did at RBX which I've been doing currently is the concept of a "current" symlink. It's a bit convoluted but it works out for now and is pretty much the way that RVM works. Having this wrapped in a wrapper RPM keeps it managed via native package management but it's far from ideal.

Friday, February 12, 2010

Time to retire the beast

So my primary server at home for the LAN is an aging dual P3 with 1GB of memory (it won't even support any beyond that).

Currently it has the following disk layout:

/dev/mapper/DataVolGroup01-DataVol
184G 87G 88G 50% /data01
/dev/mapper/DataVolGroup00-DataVol00
880G 834G 1.5G 100% /data00

My mythtv backend (a somewhat freshly built X2 Dual Core Processor 4400+) has this disk layout:

/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth

I finally got the new drives I ordered from Amazon back in November - 3x1TB Hitachi Deskstars.

Now the mythbackend looks like this:
/dev/mapper/mythvg-mythvol
978G 380G 599G 39% /myth
/dev/sdc1 932G 4.2M 932G 1% /myth2

The 880GB on lansrv is actually an external SATA array connected via Multilane to the server (4x500GB in a RAID10). I've been wanting to retire that box for some time as it draws WAY too much power for the service it provides. The mythbackend is becoming the hub for storing our archived records, pictures and music on the network.

However I didn't really have an easy place to dump that data on lansrv temporarily. Now I do. Here's the tentative plan:

1- Migrate all the lansrv data over to new drive on mythbackend.
2- Pull two of the 500GB drives out of the external array and put in the new 1TB drives.
3 - Mirror the new 1TB drives
4 - Rebuild the remaining 500GB drives as a mirrored pair.
5 - Stripe the two mirrors together
6 - Figure out what to do with the spare 1TB and 2 500GB drives.

Most likely the 1TB drive remaining will end up as a hot-spare for the 1TB mirror. Hitachi's aren't called Deathstars for nothing. The other two 500GB drives will probably go somewhere internal to the mythbackend box and become a mirrored pair as well and added to the new LV.

If I can manage to fit an eSATA card with 2 ports in the machine, I might consider moving all 4 500GB drives into a new array and adding it to the volume group.

Who knows? I'll figure something out but for now I had to have somewhere to dump the data temporarily while I shifted hardware and data around.

As an interesting side note, the 4 500GB drives have been running in the multilane enclosure since I worked at CLA. That's at least 5 years. I don't even remember what KIND of drive they were but I'm damn sure going to make note of it when I pull them.

Wednesday, January 27, 2010

Apple predictions

I wanted to get some quick predictions out for the Apple event today in the event that the device is a tablet:

- WiFi
- Bluetooth
- WhisperNet (like Kindle) via Verizon or possibly Clearwire/Sprint (and thus be carrier independent when LTE is rolled out across the board.)
- Wireless HDMI (aka WiDi)
- Wireless Charging

I don't think the device will have ANY external ports whatsoever. Depending on the range of WiDi, you can sit on your couch and broadcast a iTunes show to the TV while doing other stuff on the tablet itself. If you want, you leave the charging pad by the television or somewhere else.

Imagine you're riding on the train watching the latest episode of whatever. You get home and the device picks up where it left off broadcasting to your television.

I think social gaming a la Farmville will be there as well. Garage band on this thing would actually be pretty cool.

Anyway, some of these ideas are my own and others are things I've picked up from various podcasts but if I were building the next gen "device", those are the features I would pick.

Sunday, January 10, 2010

Ugly queries

I've never been a fan of ORMs in general. I love the concept. I think it's awesome but since the early days of hibernate, I've always had problems with automatically generated SQL statements. It's not as bad as it used to be but sometimes you get "gems" in the wild.

Today's gem comes courtesy of the Geokit plugin for Rails. Our newest application at AJC uses Geokit for finding properties within a 10-mile radius of the entered address. We've got a table with about 500k rows of geocoded metro Atlanta property data in MySQL.

Enter the query:


SELECT
*,
(ACOS(least(1,COS(0.593807786352424)
*COS(-1.47255067854929)
*COS(RADIANS(parcels.latitude))
*COS(RADIANS(parcels.longitude))+
COS(0.593807786352424)
*SIN(-1.47255067854929)
*COS(RADIANS(parcels.latitude))
*SIN(RADIANS(parcels.longitude))+
SIN(0.593807786352424)*SIN(RADIANS(parcels.latitude))))*3963.19)
AS distance FROM `parcels`
WHERE (((parcels.latitude>32.5769814714641
AND parcels.latitude<35.4683785285359>-86.1150637195914
AND parcels.longitude<-82.6268142804086))
AND ((ACOS(least(1,COS(0.593807786352424)
*COS(-1.47255067854929)*COS(RADIANS(parcels.latitude))
*COS(RADIANS(parcels.longitude))+ COS(0.593807786352424)
*SIN(-1.47255067854929)
*COS(RADIANS(parcels.latitude))
*SIN(RADIANS(parcels.longitude))+
SIN(0.593807786352424)
*SIN(RADIANS(parcels.latitude))))*3963.19)<= 100))
ORDER BY distance asc
LIMIT 21;


This bad boy is a DOOZY. It's got all the makings of exploiting everything in MySQL that can't be optimized - function results for comparison, ordering by function results and even a LIMIT thrown in for good measure.

And guess what? There's currently not much that can be done to optimize this query. Indexes won't help. The query cache is useless because A) the queries are unique to each source address and B) it'll just get pushed out anyway.

I'm looking around for some tips on this and the most likely solution is using spatial indexes in MySQL but that's a whole other issue.

Tuesday, November 24, 2009

Saving the newspaper industry

I debated whether or not to post this but I figured what the hell. If somehow it gets read by my employer and causes issues, I'll cope.

I've recently become employed as a contractor with a large newspaper in the Atlanta area. I was and am really excited by the opportunity. The technologies are cool and it's a market I never thought I'd be in.

This particular paper has been hit pretty hard by the declining revenue in the industry because they are so large. They're currently in the process of moving out of the cornerstone building they've been in downtown since (I think) 1972. It's a gorgeous building and I would have love to been there during the hey day. These days the building is like 30% occupied or something absurd. One of the paths I take to get lunch takes me through the back past all the old printing and sorting equipment. It's very ghostly and I plan on getting some pictures before we move.

Anyway, I was talking to a coworker about a project they're working on. He needed some feedback about solutions from a system engineering side. During the conversation, we got off on a tangent about the company and recent history. It was legitimate discussion because the information helped me understand the purpose of the application and what it needed to do, how it was being accessed and such.

One thing he mentioned to me was that at the last publisher's meeting, they pointed out that digital was only 10% of the revenue and that they would be focusing more on how to increase print revenue.

I was flabbergasted. I mentioned this to my wife to gauge her thoughts and she came to the same conclusion I did.

Why would you scale back from the growing market to try and preserve the dying one?

Here's the deal. I'm pretty typical (I think) of the modern reader (and I LOVE to read).

Sidebar: I remember when this paper was actually TWO papers - one the conservative publication and the other the liberal one. Up until about 9 years ago, I had a subscription to the paper including Sunday. I got great joy out of sitting down and reading the newspaper. I still do.

Anyway, as I said I'd like to think I'm pretty representative of the transition in society. I'm 35 years old. I'm old enough to remember pre-internet and young enough to be hooked into the new stuff. Some of that might have something to do with IT but that's becoming less and less of a gap. Technology is now so pervasive and accessible that what used to be the domain of the geek is now the domain of the mom. Everyone, for the most part, has access to the Internet from any number of devices not the least of which is the cell phone.

So when I hear a statement like I heard above I have to wonder how anyone could make such a statement about trying to "life-support" the print side of things. In addition to the AMAZING post mortem from John Temple, former editor/president/publisher of the Rocky Mountain News. The market for that is slowly dying - literally. I'm not trying to be grim or hyperbolic. According to this, the largest percentage of newspaper subscribers is in the 65+ range. My age range only accounts for 17%.

What I'm trying to say is that once that subscription base starts passing on, things don't look to good. By the time I'm at that range, there simply won't be any subscription base left. From my perspective (and I'm going to be somewhat crass here), why would I buy a newspaper subscription when I can read the same news from 10 different sources on my cell phone in the morning while I'm taking a dump? I wouldn't. These days I find myself not even bothering to pick up a copy of the paper at the doctor's office or mechanic. I'm either tweeting, facebooking, posting on forums or reading RSS feeds of the aforementioned 10 different sources. I'm pretty damn efficient at it too. The generation below me is even BETTER at it. They'll be doing things when I'm 64 (cue music) I won't understand and I'll be bitching about the good old days of the tubes and the propensity of children to walk across my grass.

Newspapers cannot compete on news. At least as far as my "outside" eyes can see. News is a commodity in a sense. Word travels fast these days and it's only going to get faster. From the perspective of a print paper, the news is already old as soon as it happens in this modern age. 500 people have already Twittered, iReported, blogged , Facebooked, Youtubed and done interviews with newspapers overseas before it's been sent to the presses. Using any random bit of aggregation services and software all of that information has been compiled and presented to me in a way that I wish to consume it*. If it hasn't been, I can easily do so with a single Google search when I want to know more. I can easily participate with a hashtag, url shortening service and a cell phone if I want. I become a part of it. I'm a member of the community around that bit of information.

So what can be done? Where should newspapers go? I thought about this a lot on the way home from work tonight. I had a plan all devised in my head. It was all so clear. Too clear in fact. Obviously someone else has to have seen this as clear as I did. Maybe I got some of the ideas from John Temple's speech. I wasn't sure.

When I finally got downstairs in front of my computer, I did a quick search (on Google of course). I pulled up the copy of the Temple speech. I also searched for one phrase - "How to save the newspapers". The fourth result down** was this article on truliablog.com. There were all of my grand ideas laid bare and done so almost a year ago. This almost entirely sums up what I had in mind and plan on repeating here in condensed form.

So what was my idea that obviously isn't just my idea?

Community. The Eastern Standard Tribe kind.

Back "in the day", some of us geeks were identified by the web communities of which we were a part. We're you a Slashdotter or did you read Kuro5hin? Maybe you spent your time digging and burying. If tech wasn't your thing, were you a Deviant? These days it's much of the same but the communities are even more diverse. People like, I think, having a sense of belonging and it's much more if not entirely acceptable to belong to a community where you've probably never met half the people. You're known by an avatar and a nickname. I have people I consider friends who call me by my character name in World of Warcraft and yet we all know each other's real names and personal information just as much as some of my friends here in town. I went to parties years ago with people I only knew from IRC and was called by my nickname the entire night. I addressed others the same way.

Newspapers can establish that kind of community in the modern age. When there were two separate newspapers in town, you might have an idea about a person based on which one they read. These days, I find myself building relationships and communities with people around the world based on the most niche of ideas - a mailing list for an open source project, a coding language or a forum dedicated to an author I might like. If the author is reasonably current, I might even be able to follow them on Twitter or become a fan on Facebook. It's no less of a community than the one I actually live in which has an equally vivid online presence.

So my plan was for the newspapers to be breeding grounds for those communities. Create the tools. Forums. Aggregation. Whatever is appropriate for making it easy for people to build those communities. They will still be providing the same service they were before which, at the core, is delivery of information. Ebay is nice but Craiglist is better when I want to go local. Building local communities online can even help spur the local economy which can always use help regardless of where "local" is.

Take advantage of the personal capital you have built up in your existing personalities. If you have a reporter that people trust, that person should be right in the thick of the community. Clark Howard is someone who has an amazing community of like-minded people from everywhere that he's broadcast but I would wager he's loved nowhere else in the world more than he is right here in Atlanta where we've been listening to him locally for years. Modern news is interactive. It's no longer a matter of writing an editorial and walking away from it. You'll have to be accountable for what you say because if you aren't, you'll lose trust from the community and, as in any relationship, you may never be able to earn that trust back.

Be open. This somewhat ties to the previous statement but I'm thinking more along the lines of "open to ideas". I'm amazed by how much CNN has integrated iReport into the main "product". After the President's speech to Congress months back, CNN had two people on who represented each side of the issue. The amazing part is that those two people were from the community CNN had provided and weren't talking heads. They were real people that best represented the viewpoints of their community at CNN. Mind you I don't think the person representing one of the sides was particularly articulate but he wasn't a pundit or a news guy so I can't be too critical. Make the community part of the product. When someone feels a sense of ownership, it becomes more personal.

Don't be afraid of Google. Don't buy into the paywall tripe being spread by some business men. Don't cut off your nose to spite your face and all that. While your focus may be the traditional concept of local, there are expats all over the world who would belong in that "local" community. Those people might even be your greatest source of revenue. I can see me living in Michigan years down the road and wishing I could get an Frosted Orange. You get the idea. I still need to know about the community and if Google doesn't know about it, I won't know about it.

Commit to the community. Don't be halfway about it. If you're going to do it, do it. Nothing is more frustrating that getting involved with something (an online game or whatever) only to see it die on the vine because of lack of support from the people who put it out there in the first place. While people will gladly contribute more than you can handle, they'll need some nudging to maintain focus. If you have a sub-community built around Foodies in Atlanta, maybe you should have some prominent Foodies actually be involved in some officially recognized capacity. Moderators are a valuable thing. A forum is only as good as its signal to noise ratio. If appointing community managers is out of the question, at least provide the community with the means to self-police. Don't discount Whuffie in self-moderating environments.

Accept organic. While the communities will need some nudging to help get started, realize that they WILL take on a life of their own. They may grow beyond what you might originally envision. They'll merge. They'll split. The Atlanta Baklava community might realize that while anyone who DOESN'T like baklava is obviously deficient in some capacity, maybe they're better suited in merging with the Atlanta Pastry community as a whole.

So where does the news come out of all this? Where's the news in this modern newspaper? The communities are built FROM the news. You'll still have your traditional reporting "model" but the communities will spring up out of that news. I started following a few new people on Twitter after the #atlflood event and every single picture I saw of the flood came from either Flickr, Twitpic or *ding-ding* ajc.com. There were people watching the #atlflood hashtag who had the same experience I did. I would hope that the newpaper was ready to capitalize on that.

I don't claim to have all the answers. I've only been on the job for two weeks now. The thing is that I REALLY like the job. I enjoy the people I'm working with and I want the company to succeed. Maybe that's what's needed. Less old-school baggage and a fresh perspective from the outside. It's also more selfish than that. I would love to be at the ground level of helping to engineer something as amazing as bringing the newspapers into the modern age.

* Think Manfred Macx in Accelerando
** The first result was from Time magazine. The second was from another newspaper. The third was an article on how to save newspaper clippings.

Friday, November 20, 2009

Exchange Web Services via Ruby

I realized as I went to post an update on this that I don't think I ever posted an actual blog entry about it.

While I was on contract at HSWI, the email was hosted on Exchange. The development team was all on Mac workstations but the front-side was all on Windows. Since I was running Linux full-time, Exchange access wasn't a big deal. I ran Mutt and used LDAP/IMAPS/SSMTP. Digging around the Mail.app on the OS X side however, I realized that the client side was using Exchange Web Services for much of the functionality.

Wanting to do some poking around with Ruby and SOAP, I figured it would be a fun exercise to talk to the Exchange server with Ruby. I also had an itch to scratch thinking I could use it as an address book source for Mutt. I got it working in a day or so after dealing with some broken functionality in either the WSDL from the Exchange side or how SOAP4R tried to translate it.

You can find the code here.

Anyway, I'm no longer with HSWI but over at the AJC, we're also using Exchange in a much greater capacity. Anyway, so I whipped out the "old" code and was depressed to find out it didn't work.

Every attempt gave me a 401 error. It made no sense since I could access OWA, ActiveSync with my DROID (awww yeah) and even access the EWS wsdls on the server.

As I started to poke around online, I started to realize that in some of the more complex configurations, the Exchange server is hidden behind an ISA server or something. I don't do the Microsoft world much anymore and I have no real access to the environment.

What I noticed though, was that the internal IP is different than the external one. That gave me an idea to make sure and test the code externally from home while NOT being on the VPN.

It worked!

As I poked around with THAT information, I remembered an interesting thing that happened the one time I logged. on to a Windows machine at the office. If I fired up I.E. and went to the OWA url, I was logged in directly.

What I'm guessing is that the Exchange server has a different set of criteria for authenticating internally than externally. I'm not up to date on current Exchange and AD implementations so I have no idea what the configuration is that's preventing me from using EWS but still allowing me to use OWA from Firefox under Linux and OS X.

If any MS guys out there have any idea what's causing this, I'd be interested to know so I can either work around it or document it appropriately.

I have a *FEELING* that it's somehow related to NTLM but I don't know how to force my EWS call to bypass it just yet.