tag:blogger.com,1999:blog-9349853014557059902024-03-12T19:52:51.726-04:00LusisStuff from the mind of John E. Vincentlusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.comBlogger97125tag:blogger.com,1999:blog-934985301455705990.post-11378968934569727382011-09-13T22:19:00.001-04:002011-10-18T06:55:07.337-04:00Relocating again.FYI, I'm relocating my blog back to my own domain. All the new posts will go there and eventually I'll get everything pulled in from here.<br />
<br />
New Old Place is here: <a href="http://blog.lusis.org/">http://blog.lusis.org</a>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com1tag:blogger.com,1999:blog-934985301455705990.post-49769916255875085942011-07-21T22:19:00.001-04:002011-07-22T10:26:26.642-04:00Monitoring Sucks - Watch your language<p><em>The following post is a recap of what was discussed in the 07/21/11 #monitoringsucks irc meeting</em></p><p>Before I get any further, I just want to thank everyone who attended, either in virtual person or in spirit. There have been so many awesome discussions going around this topic since we started. I am truly priviledged to interact with each and every one of you. I struggle to count myself a peer and I can only hope that I provide something in return.</p><p>I mentioned to someone today that I’m literally sick of the current landscape. I consider the current crop of monitoring solutions to be technologically bankrupt. The situation is fairly untenable at this point.</p><p>I just installed (after having a total loss of an existing Zenoss setup) Nagios again. I’m not joking when I say that it depressed the everliving hell out of me. The monitoring landscape simply has not kept up with modern developments. At first it was mildly frustrating. Then it was annoying. Now it’s actually a detriment.</p><p>Now that we have that out of the way….</p><h1 id="darmok-and-jalad-and-tanagra">Darmok and Jalad at Tanagra</h1><p>Communication is important. Like Picard and Dathon, we’ve been stranded on a planet with shitty monitoring tools and we’re unable to communicate about the invisibile threat of suck because we aren’t even speaking the same language. I say event, you hear trigger, I mean data point. So the first order of business was to try and agree on a set of terms. It was decided that we would consider these things primitives. Here they are:</p><p>Please read through this before jumping to any conclusions. I promise it will all become clear (as mud).</p><h2 id="metric">metric</h2><p><em>a numeric or boolean data point</em></p><p>The data type of a metric was something of a sticking point. People were getting hung up on data points being various things (a log message, a “status”, a value). We needed something to describe the origin. The single “thing” that triggered it all. That thing is a metric.</p><p>So why numeric <em>OR</em> boolean? It was pretty clear that many people considered, and rightly so I would argue, that a state change is a metric. A good example given by <a href="http://twitter.com/cwebber">Christopher Webber</a> is that of a BGP route going away. Why is this a less valid data point than the amount of disk space in use or the latency from one host to another? Frankly, it’s not.</p><p>But here’s where it gets fuzzy. What about a log message. Surely that’s a data point and thus a metric.</p><p>Yes and No. The <em>presence</em> of a log message is a data point. But it’s a boolean. The log message itself?</p><h2 id="context">context</h2><p><em>metadata about a metric</em></p><p>Now metadata itself is a loaded term but in this scope, the “human readable” attributes are considered context. Going back to our log example. The presence of the log message is a metric. The log message itself is context. Here’s the thing. You want to know if there is an error message in a log file. The type of error, the error message text? That’s context for the metric to use in determining a course of action.</p><p>Plainly speaking, metrics are for machines. Context is for humans. This leads us to….</p><h2 id="event">event</h2><p><em>metric combined with context</em></p><p>This is still up in the air but the general consensus was that this was a passable definition. The biggest problem with a group of domain experts is that they are frequently unable to accept semantic approximation. Take the discussion of Erlang Spawned process:</p><ul><li>It’s sort of like a VM on a VM</li>
<li>NO IT’S NOT.</li>
<li><em>headdesk</em></li>
</ul><p>The fact is that an Erlang spawned process has shades of a virtual machine is irrelevant to the domain expert. We found similar discussions around what we would call the combination of a metric and its context. But where do events come from?</p><h2 id="resource">resource</h2><p><em>the source of a metric</em></p><p>Again, we could get into arguments around what a resource is. One thing that was painfully obvious is that we’re all sick and tired of being tied to the Host and Service model. It’s irrelevant. These constructs are part “legacy” and part “presentation”.</p><p>Any modern monitoring thought needs to realize that metrics no longer come from physical hosts or are service states. In the modern world, we’re taking a holistic view of monitoring that includes not only bare metal but business matters. The number of sales is a metric but it’s not tied to a server. It’s tied to the business as a whole. The source of your metrics is a resource. So now that we have this information - a metric, its context and who generated it - what do we do? We take….</p><h2 id="action">action</h2><p><em>a response to a given metric</em></p><p>What response? It doesn’t MATTER. Remember that these are primitives. The response is determined by components of your monitoring infrastructure. Humans note the context. Graphite graphs it. Nagios alerts on it. ESPER correlates it with other metrics. Don’t confuse scope here. From this point on, whatever happens has is all decided on by a given component. It’s all about perspective and aspects.</p><h1 id="temba-his-arms-wide">Temba, his arms wide</h1><p>I’m sure through much of that, you were thinking “alerting! graphing! correlation!”. Yes, that was pretty much what happened during the meeting as well. Everyone has pretty much agreed (I think) at this point that any new monitoring systems should be modular in nature. As <a href="http://twitter.com/obfuscurity">Jason Dixon</a> put it - “Voltron”. No single system that attempts to do everything will meet everyone’s needs. However, with a common dictionary and open APIs you should be able to build a system that DOES meet your needs. So what are those components? Sadly this part is not as fleshed out. We simply ran out of time. However we did come up with a few basics:</p><h2 id="collection">Collection</h2><p><em>getting the metrics</em></p><p>It doesn’t matter if it’s push or pull. It doesn’t matter what the transport is - async or point-to-point. Somehow, you have to get a metric from a resource.</p><h2 id="event-processing">Event Processing</h2><p><em>taking action</em></p><p>Extract the metric and resource from an event. Do something with it. Maybe you send the metric to another component. Maybe you “present” it or send it somewhere to be presented. Maybe you perform a change on a resource (restarting a service). Essentially the decision engine.</p><h2 id="presentation">Presentation</h2><p>While you might be thinking of graphing here, that’s a type of presentation. You know what else is presentation? An email alert. Stick with me. I know what’s going through your head. No..not that…the other thing.</p><h2 id="analytics">Analytics</h2><p>This is pretty much correlation. We didn’t get a REAL solid defintion here but everyone was in agreement that some sort of analytics is a distinct component.</p><h2 id="the-other-stuff">The “other” stuff</h2><p>As I said, we had to kind of cut “official” things short. There was various discussion around Storage and Configuration. Storage I personally can see as a distinct component but Configuration not so much. Configuration is an aspect of a component but not a component itself.</p><h2 id="logical-groupings">Logical groupings</h2><p>Remember when I said I know what you’re thinking? This is what I think it was.</p><p>You can look at the above items and from different angles they look similar. I mean sending an email feels more like event processing than presentation. You’d probably be right. By that token, drawing a point on a graph is technically processing an event. The fact is many components have a bit of a genetic bond. Not so much parent/child or sibling but more like cousins. In all honesty, if I were building an event processing component, I’d probably handle sending the email right there. Why send it to another component? That makes perfect sense. Graphing? Yeah I’ll let graphite handle that but I can do service restarts and send emails. Maybe you have an intelligent graphing component that can do complex correlation inband. That makes sense too.</p><p>I’m quite sure that we’ll have someone who writes a kickass event processor that happens to send email. I’m cool with that. I just don’t want to be bound to ONLY being able to send emails because that’s all your decision system supports.</p><h1 id="shaka-when-the-walls-fell">Shaka, when the walls fell</h1><p>Speaking personally, I really feel like today’s discussion was VERY productive. I know that you might not agree with everything here. Things are always up for debate. The only thing I ask is that at some point, we’re all willing to say “I know that this definition isn’t EXACTLY how I would describe something but it’s close enough to work”.</p><p>So what are the next steps? I think we’ve got enough information and consensus here for people to start moving forward with some things. One exercise, inspired by something Matt Ray said, that we agreed would be REALLY productive is to take an existing application and map what it does to our primitives and components. Matt plans on doing that with Zenoss since that’s what he knows best.</p><p>Let me give an example:</p><p>Out of the box, Nagios supports Hosts and Services which map pretty cleanly to resources. It is does not only collection but event processing and presentation. It not only supports metrics but also context (Host down is the boolean metric. “Response timeout” is the context. Through something like pnp4nagios, it can support different presentations. It has very basic set of Analytic functionality.</p><p>Meanwhile Graphite is, in my mind, strictly presentation and deals only with metrics. It does support both numeric and boolean metrics. It also has basic resource functionality but it’s not hardwired. It doesn’t really do event handling in the strict sense. Analytics is left to the human mind. It certainly doesn’t support context.</p><p>I’d love to see more of these evaluations.</p><p>Also, I know there are tons of “words” that we didn’t cover - thresholds for instance. While there wasn’t a total consensus, there was some agreement that somethings were attributes of a component but not a primitive itself. It was also accepted that components themselves would be primitives. You correlation engine might aggregate (another word) a group of metrics and generate an event. At that point, your correlation engine is now a resource with its own metrics (25 errors) and context (“number of errors across application servers exceeded acceptable limits”) which could be then sent to an event processor.</p><p>That’s the beauty of the Voltron approach and not binding a resource to a construct like a Host.</p><h1 id="special-note-to-the-aussies">Special note to the Aussies</h1><p>I’m very sorry that we couldn’t get everyone together. I’ve scheduled another meeting where we can start from scratch just like this one, or build on what was discussed already. I’m flexible and willing to wake up at whatever time works best for you guys</p><p>Thanks again to everyone who attended. If you couldn’t be there, I hope you can make the next one.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com9tag:blogger.com,1999:blog-934985301455705990.post-65977637752926657982011-07-12T02:26:00.000-04:002011-07-12T02:26:09.088-04:00Monitoring Sucks - Round 2 - FIGHT<h1 id="monitoring-sucks---round-2---fight">Monitoring sucks - Round 2 - FIGHT</h1><p>Recently there’s been more and more attention to the whole ##monitoringsucks campaign. Peopel are writing blog posts. Code is being written. Discussions are being had. Smart people are doing smart things.</p><h2 id="meetingssuck">meetingssuck</h2><p>I figure it’s time to get the band back together for another official irc session and stop flooding infra-talk with random rants about where things stand. As much as I hate to even consider it, I think having something of an agenda so that it doesn’t turn into a bitch fest.</p><p>I’m open to any ideas that people have.</p><p>Jordan Sissel brought up some things that might be nice to have:</p><ul><li>use cases</li>
<li>primitives</li>
</ul><p>I’d like to also love to get input from others before we settle on the details. This one is going to be a bit more “formal” than the first session. Don’t let that turn you off. Everyone has something valuable to contribute.</p><h2 id="deliverables">Deliverables</h2><p>I can’t believe I’m actually suggesting this part but I’d also love to walk away with something that people can work with. It’s not like people will have homework or a task list.</p><p>This is more of a “this is what we would like to see, if you want to write some code it’s a decent place to start”.</p><h2 id="international-flavor">International flavor</h2><p>I’m also keen on finding a time when we can get as many folks as possible to contribute. This may not be possible with that whole rotation of the earth thing but we’ll see.</p><h1 id="suggestions-comments-and-rude-remarks">Suggestions, Comments and Rude Remarks</h1><p>If you’ve got anything you’d like to see or contribute on the planning side, drop me a message on IRC, email or twitter and I’ll include it (if possible) when I draft up the final “notes” beforehand.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com3tag:blogger.com,1999:blog-934985301455705990.post-47666330448736098912011-06-05T23:30:00.002-04:002011-06-14T00:40:18.935-04:00Why Monitoring Sucks<h1 id="why-monitoring-sucks-and-what-were-doing-about-it">Why Monitoring Sucks (and what we're doing about it)</h1><p>About two weeks ago someone made a tweet. At this point, I don't remember who said it but the gist was that "monitoring sucks". I happened to be knee-deep in frustrating bullshit around that topic and was currently evaluating the same effing tools I'd evaluated at every other company over the past 10 years or so. So I did what seems to be S.O.P for me these days. I started something.</p><h1 id="but-does-monitoring-really-suck">But does monitoring REALLY suck?</h1><p>Heck no! Monitoring is AWESOME. Metrics are AWESOME. I love it. Here's what I don't love: - Having my hands tied with the model of host and service bindings. - Having to set up "fake" hosts just to group arbitrary metrics together - Having to either collect metrics twice - once for alerting and another for trending - Only being able to see my metrics in 5 minute intervals - Having to chose between shitty interface but great monitoring or shitty monitoring but great interface - Dealing with a monitoring system that thinks <strong>IT</strong> is the system of truth for my environment - Perl (I kid...sort of) - Not actually having any real choices</p><p>Yes, yes I know:</p><blockquote><p>You can just combine Nagios + collectd + graphite + cacti + pnp4nagios and you have everything you need!</p></blockquote><p>Seriously? Kiss my ass. I'm a huge fan of the Unix pipeline philosophy but, christ, have you ever heard the phrase "antipattern"?</p><h1 id="so-what-the-hell-are-you-going-to-do-about-it">So what the hell are you going to do about it?</h1><p>I'm going to let smart people be smart and do smart things.</p><p>Step one was getting everyone who had similar complaints together on IRC. That went pretty damn well. Step two was creating a github repo. Seriously. Step two should ALWAYS be "create a github repo". Step three? Hell if I know.</p><p>Here's what I do know. There are plenty of frustrated system administrators, developers, engineers, "devops" and everything under the sun who don't want much. All they really want is for shit to work. When shit breaks, they want to be notified. They want pretty graphs. They want to see business metrics along side operational ones. They want to have a 52-inch monitor in the office that everyone can look at and say:</p><blockquote><p>See that red dot? That's bad. Here's what was going on when we got that red dot. Let's fix that shit and go get beers</p></blockquote><h1 id="about-the-repo">About the "repo"</h1><p>So the plan I have in place for the repository is this. We don't really need code. What we need is an easy way for people to contribute ideas. The plan I have in place for this is partially underway. There's now a <em>monitoringsucks</em> organization on Github. Pretty much anyone who is willing to contribute can get added to the team. The idea is that, as smart people think of smart shit, we can create new repository under some unifying idea and put blog posts, submodules, reviews, ideas..whatever into that repository so people have an easy place to go get information. I'd like to assign someone per repository to be the owner. We're all busy but this is something we're all highly interested in. If we spread the work out and allow easy contribution, then we can get some real content up there.</p><p>I also want to keep the repos as light and cacheable as possible. The organization is under the github "free" plan right now and I'd like to keep it that way.</p><h2 id="blog-posts-repo">Blog Posts Repo</h2><p>This repo serves as a place to collect general information about blog posts people come across. Think of it as hyper-local delicious in a DVCS.</p><p>Currently, by virtue of the first commit, Michael Conigliaro is the "owner". You can follow him on twitter and github as @mconigliaro</p><h2 id="irc-logs-repo">IRC Logs Repo</h2><p>This repo is a log of any "scheduled" irc sessions. Personally, I don't think we need a distinct #monitoringsucks channel but people want to keep it around. The logs in this repo are not full logs. Just those from when someone says "Hey smart people. Let's think of smart shit at this date/time" on twitter.</p><p>Currently <strong>I</strong> appear to be the owner of this repo. I would love for someone who can actually make the logs look good to take this over.</p><h2 id="tools-repo">Tools Repo</h2><p>This repo is really more of a "curation" repo. The plan is that each directory is the name of some tool with two things it in:</p><ul><li>A README.md as a review of the tool</li>
<li>A submodule link to the tool's repo (where appropriate)</li>
</ul><p>Again, I think I'm running point on this one. Please note that the submodule links APPEAR to have some sort of UI issue on github. Every submodule appears to point to Dan DeLeo's 'critical' project.</p><h2 id="metrics-catalog-repo">Metrics Catalog Repo</h2><p>This is our latest member and it already has an official manager! Jason Dixon (@obfuscurity on github/twitter - jdixon on irc) suggested it so he get's to run it ;) The idea here is that this will serves as a set of best practices around what metrics you might want to collect and why. I'm leaving the organization up to Jason but I suggested a per-app/service/protocol directory.</p><h1 id="wrap-up">Wrap Up</h1><p>So that's where we are. Where it goes, I have no idea. I just want to help where ever I can. If you have any ideas, hit me up on twitter/irc/github/email and let me know. It might help to know that if you suggest something, you'll probably be made the person reponsible for it ;)</p><h1 id="update">Update!</h1><p>It was our good friend Sean Porter (@portertech on twitter), that we have to thank for all of this ;) <table style="width:auto;"><tr><td><a href="https://picasaweb.google.com/lh/photo/Zi1k9F_7lBKjcN8dtJlXXQ?feat=embedwebsite"><img src="https://lh5.googleusercontent.com/-O6mNvCvCPyU/TexPV1P9YaI/AAAAAAAAAWk/7ZQ8BkXUyn8/s144/monitoring-sucks.png" height="51" width="144" /></a></td></tr>
<tr><td style="font-family:arial,sans-serif; font-size:11px; text-align:right">From <a href="https://picasaweb.google.com/lusisjv/PublicPhotos?feat=embedwebsite">Public Photos</a></td></tr>
</table></p><h1 id="update2">Update (again)</h1><p>It was kindly pointed out that I never actually included a link to the repositories. Here they are:<br />
</p><p><a href="https://github.com/monitoringsucks">https://github.com/monitoringsucks</a><br />
</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com4tag:blogger.com,1999:blog-934985301455705990.post-37287444080744288022011-05-20T02:01:00.001-04:002011-05-20T13:34:56.048-04:00On Noah - Part 4<h1 id="on-noah---part-4">On Noah - Part 4</h1><p><em>This is the fourth part in a series on Noah. <a href="http://goo.gl/l3Mgt">Part 1</a>, <a href="http://goo.gl/Nj2TN">Part 2</a> and <a href="http://goo.gl/RsZtZ">Part 3</a> are available as well</em></p><p>In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. Part 3 was about the components underneath Noah that make it tick.</p><p>This post is about the "future" of Noah. Since I'm a fan of Fourcast podcast, I thought it would be nice to do an immediate, medium and long term set of goals.</p><h2 id="immediate-future---the-road-to-1.0">Immediate Future - the road to 1.0</h2><p>In the most immediate future there are a few things that need to happen. These are in no specific order.</p><ul><li>General<br />
<ul><li>Better test coverage ESPECIALLY around the watch subsystem</li>
<li>Full code comment coverage</li>
<li>Chef cookbooks/Puppet manifests for doing a full install</li>
<li>"fatty" installers for a standalone server</li>
<li>Documentation around operational best practices</li>
<li>Documentation around clustering, redundancy and hadr</li>
<li>Documentation around integration best practices</li>
<li>Performance testing</li>
</ul></li>
<li>Noah Server<br />
<ul><li>Expiry flags and reaping for Ephemerals</li>
<li>Convert mime-type in Configurations to make sense</li>
<li>Untag and Unlink support</li>
<li>Refactor how you specify Redis connection information</li>
<li>Integrated metrics for monitoring (failed callbacks, expired ephemeral count, that kind of stuff)</li>
</ul></li>
<li>Watcher callback daemon<br />
<ul><li>Make the HTTP callback plugin more flexible</li>
<li>Finish binscript for the watcher daemon</li>
</ul></li>
<li>Other<br />
<ul><li>Finish <a href="http://goo.gl/B65aL">Boat</a></li>
<li>Finish NoahLite LWRP for Chef (using Boat)</li>
<li>A few more HTTP-based callback plugins (Rundeck, Jenkins)</li>
</ul></li>
</ul><p>Now that doesn't look like a very cool list but it's a lot of work for one person. I don't blame anyone for not getting excited about it. The goal now is to get a functional and stable application out the door that people can start using. Mind you I think it's usable now (and I'm already using it in "production").</p><p>Obviously if anyone has something else they'd like to see on the list, let me know.</p><h2 id="medium-rare">Medium Rare</h2><p>So beyond that 1.0 release, what's on tap? Most of the work will probably occur around the watcher subsystem and the callback daemon. However there are a few key server changes I need to implement.</p><ul><li>Server<br />
<ul><li>Full ACL support on every object at every level</li>
<li>Token-based and SSH key based credentialing</li>
<li>Optional versioning on every object at every level</li>
<li>Accountability/Audit trail</li>
<li>Implement a long-polling interface for inband watchers</li>
</ul></li>
<li>Watcher callback daemon<br />
<ul><li>Decouple the callback daemon from the Ruby API of the server. Instead the daemon itself needs to be a full REST client of the Noah server</li>
<li>Break out the "official" callback daemon into a distinct package</li>
</ul></li>
<li>Clients<br />
<ul><li>Sinatra Helper</li>
</ul></li>
</ul><p>Also during this period, I want to spend time building up the ecosystem as a whole. You can see a general mindmap of that <a href="https://github.com/lusis/Noah/wiki/Ecosystem">here</a>.</p><p>Going into a bit more detail...</p><h3 id="tokens-and-keys">Tokens and keys</h3><p>It's plainly clear that something which has the ability to make runtime environment changes needs to be secure. The first thing to roll off the line post-1.0 will be that functionality. Full ACL support for all entries will be enabled and in can be set at any level in the namespace just the same as Watches.</p><h3 id="versioning-and-auditing">Versioning and Auditing</h3><p>Again for all entires and levels in the namespace, versioning and auditing will be allowed. The intention is that the number of revisions and audit entries are configurable as well - not just an enable/disable bit.</p><h3 id="in-band-watches">In-band watches</h3><p>While I've lamented the fact that watches were in-band only in Zookeeper, there's a real world need for that model. The idea of long-polling functionality is something I'd actually like to have by 1.0 but likely won't happen. The intent is simply that when you call say <code>/some/path/watch</code>, you can pass an optional flag in the message stating that you want to watch that endpoint for a fixed amount of time for any changes. Optionally a way to subscribe to all changes over long-polling for a fixed amount of time is cool too.</p><h3 id="agent-changes">Agent changes</h3><p>These two are pretty high on my list. As I said, there's a workable solution with minimal tech debt going into the 1.0 release but long term, this needs to be a distinct package. A few other ideas I'm kicking around are allowing configurable filtering on WHICH callback types an agent will handle. The idea is that you can specify that this invocation only handle http callbacks while this other one handles AMQP.</p><h3 id="sinatra-helper">Sinatra Helper</h3><p>One idea I'd REALLY like to come to fruition is the Sinatra Helper. I envision it working something like this:</p><pre class="sourceCode"><code class="sourceCode ruby"> require <span class="st">'sinatra/base'</span>
<span class="kw">class</span> <span class="dt">MyApp</span> < <span class="dt">Sinatra</span>::<span class="dt">Base</span>
register <span class="dt">Noah</span>::<span class="dt">Sinatra</span>
noah_server <span class="st">"http://localhost:5678"</span>
noah_node_name <span class="st">"myself"</span>
noah_app_name <span class="st">"MyApp"</span>
noah_token <span class="st">"somerandomlongstring"</span>
dynamic_get <span class="st">:database_server</span>
dynamic_set <span class="st">:some_other_variable</span>, <span class="st">"foobar"</span>
watch <span class="st">:this_other_node</span>
<span class="kw">end</span></code></pre><p>The idea is that the helper allows you to register your application very easily with Noah for other components in your environment to be know. As a byproduct, you get the ability to get/set certain configuration parameters entirely in Noah. The watch setting is kind of cool as well. What will happen is if you decide to <code>watch</code> something this way, the helper will create a random (and yes, secure) route in your application that watch events can notify. In this way, your Sinatra application can be notified of any changes and will automatically "reconfigure" itself.</p><p>Obviously I'd love to see other implementations of this idea for other languages and frameworks.</p><h2 id="long-term-changes">Long term changes</h2><p>There aren't so much specific list items here as general themes and ideas. While I list these as long term, I've already gotten an offer to help with some of them so they might actually get out sooner.</p><h3 id="making-noah-itself-distributed">Making Noah itself distributed</h3><p>This is something I'm VERY keen on getting accomplished and would really consider it the fruition of what Noah itself does. The idea is simply that multiple Noah servers themselves are clients of other Noah servers. I've got several ideas about how to accomplish this but I got an interesting follow up from someone on Github the other day. He asked what my plans were in this area and we had several lengthy emails back and forth including an offer to work on this particular issue.</p><p>Obviously there are a whole host of issues to consider. Race conditions in ordered delivery of Watch callbacks (getting a status "down" after a status "up" when it's supposed to be the other way around..) and eventual consistency spring to mind first.</p><p>The general architecture idea that was offered up is to use <a href="https://github.com/derekcollison/nats">NATS</a> as the mechanism for accomplishing this. In the same way that there would be AMQP callback support, there would be NATS support. Additional Noah servers would only need to know one other member to bootstrap and everything else happens using the natural flows within Noah.</p><p>The other part of that is how to handle the Redis part. The natural inclination is to use the upcoming Redis clustering but that's not something I want to do. I want each Noah server to actually include its OWN Redis instance "embedded" and not need to rely on any external mechanism for replication of the data. Again, the biggest validation of what Noah is designed to do is using only Noah itself to do it.</p><h3 id="move-off-redisswappable-persistence">Move off Redis/Swappable persistence</h3><p>If NATS says anything to me, it says "Why do you even need Redis?". If you recall, I went with Redis because it solved multiple problems. If I can find a persistence mechanism that I can use without any external service running, I'd love to use it.</p><h3 id="zeromq">ZeroMQ</h3><p>If I were to end up moving off Redis, I'd need a cross platform and cross language way to handle the pubsub component. NATS would be the first idea but NATS is Ruby only (unless I've missed something). ZeroMQ appears to have broad language and platform support so writing custom agents in the same vein as the Redis PUBSUB method should be feasible.</p><h3 id="nanite-style-agents">Nanite-style agents</h3><p>This is more of a command-and-control topic but a set of high-performance specialized agents on systems that can watch the PUBSUB backend or listen for callbacks would be awesome. This would allow you really integrate Noah into your infrastructure beyond the application level. Use it to trigger a puppet or chef run, reboot instances or do whatever. This is really about bringing what I wanted to accomplish with Vogeler into Noah.</p><h3 id="the-paxos-question">The PAXOS question</h3><p>A lot of people have asked me about this. I'll state right now that I can only make it through about 20-30% of any reading about Paxos before my brain starts to melt. However in the interest of proving myself the fool, I think it would be possible to implement some Paxos like functionality on top of Noah. Remember that Noah is fundamentally about fully disconnected nodes. What better example of a network of unreliable processors than ones that never actually talk to each other. The problem is that the use case for doing it in Noah is fairly limited so as not to be worth it.</p><p>The grand scheme is that Noah helps enable the construction of systems where you can say "This component is free to go off and operate in this way secure in the knowledge that if something it needs to know changes, someone will tell it". I did say "grand" didn't I? At some point, I may hit the limit of what I can do using only Ruby. Who knows.</p><h2 id="wrap-up---part-4">Wrap up - Part 4</h2><p>Again with the recap</p><ul><li>Get to 1.0 with a stable and fixed set of functionality</li>
<li>Nurture the Noah ecosystem</li>
<li>Make it easy for people to integrate Noah into thier applications</li>
<li>Get all meta and make Noah itself distributed using Noah</li>
<li>Minimize the dependencies even more</li>
<li>Build skynet</li>
</ul><p><em>I'm not kidding on that last one. Ask me about Parrot AR drones and Noah sometime</em></p><p>If you made it this far, I want to say thank you to anyone who read any or all of the parts. Please don't hesitate to contact me with any questions about the project.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-4118735644535613492011-05-18T22:14:00.000-04:002011-05-18T22:14:43.753-04:00On Noah - Part 3<h1 id="on-noah---part-3">On Noah - Part 3</h1><p><em>This is the third part in a series on Noah. <a href="http://goo.gl/l3Mgt">Part 1</a> and <a href="http://goo.gl/Nj2TN">Part 2</a> are available as well</em></p><p>In Part 1 and 2 of this series I covered background on Zookeeper and discussed the similarities and differences between it and Noah. This post is discussing the technology stack under Noah and the reasoning for it.</p><h2 id="a-little-back-story">A little back story</h2><p>I've told a few people this but my original intention was to use Noah as a way to learn Erlang. However this did not work out. I needed to get a proof of concept out much quicker than the ramp up time it would take to <a href="http://learnyousomeerlang.com/">learn me some Erlang</a>. I had this grandiose idea to slap mnesia, riak_core and webmachine into a tasty ball of Zookeeper clonage.</p><p>I am not a developer by trade. I don't have any formal education in computer science (or anything for that matter). The reason I mention this is to say that programming is hard work for me. This has two side effects:</p><ul><li>It takes me considerably longer than a working developer to code what's in my head</li>
<li>I can only really learn a new language when I have an itch to scratch. A real world problem to model.</li>
</ul><p>So in the interest of time, I fell back to a language I'm most comfortable with right now, Ruby.</p><h2 id="sinatra-and-ruby">Sinatra and Ruby</h2><p>Noah isn't so much a web application as it is this 'api thing'. There's no proper front end and honestly, you guys don't want to see what my design deficient mind would create. I like to joke that in the world of MVC, I stick to the M and C. Sure, APIs have views but not in the "click the pretty button sense".</p><p>I had been doing quite a bit of glue code at the office using <a href="http://www.sinatrarb.com">Sinatra</a> (and EventMachine) so I went with that. Sinatra is, if you use sheer number of clones in other languages as an example, a success for writing API-only applications. I also figured that if I wanted to slap something proper on the front, I could easily integrate it with <a href="http://www.padrinorb.com">Padrino</a>.</p><p>But now I had to address the data storage issue.</p><h2 id="redis">Redis</h2><p>Previously, as a way to learn Python at another company, I wrote an application called <a href="https://github.com/lusis/vogeler">Vogeler</a>. That application had a lot of moving parts - CouchDB for storage and RabbitMQ for messaging.</p><p>I knew from dealing with CouchDB on CentOS5 that I wasn't going to use THAT again. Much of it would have been overkill for Noah anyway. I realized I really needed nothing more than a key/value store. That really left me with either Riak or Redis. I love Riak but it wasn't the right fit in this case. I needed something with a smaller dependency footprint. Mind you Riak is VERY easy to install but managing Erlang applications is still a bit edgy for some folks. I needed something simpler.</p><p>I also realized early on that I needed some sort of basic queuing functionality. That really sealed Redis for me. Not only did it have zero external dependencies, but it also met the needs for queuing. I could use <code>lists</code> as dedicated direct queues and I could use the built-in <code>pubsub</code> as a broadcast mechanism. Redis also has a fast atomic counter that could be used to approximate the ZK sequence primitive should I want to do that.</p><p>Additionally, Redis has master/slave (not my first choice) support for limited scaling as well as redundancy. One of my original design goals was that Noah behave like a traditional web application. This is a model ops folks understand very well at this point.</p><h2 id="eventmachine">EventMachine</h2><p>When you think asynchronous in the Ruby world, there's really only one tool that comes to mind, EventMachine. Noah is designed for asynchronous networks and is itself asynchronous in its design. The callback agent itself uses EventMachine to process watches. As I said previously, this is simply using an EM friendly Redis driver that can do <code>PSUBSCRIBE</code> (using em-hiredis) and send watch messages (using em-http-request since we only support HTTP by default).</p><h2 id="ohm">Ohm</h2><p>Finally I slapped <a href="http://ohm.keyvalue.org">Ohm</a> on top as the abstraction layer for Redis access. Ohm, if you haven't used it, is simply one of if not the best Ruby library for working with Redis. It's easily extensible, very transparent and frankly, it just gets the hell out of your way. A good example of this is converting some result to a hash. By default, Ohm only returns the id of the record. Nothing more. It also makes it VERY easy to drop past the abstraction and operate on Redis directly. It even provides helpers to get the keys it uses to query Redis. A good example of this is in the Linking and Tagging code. The following is a method in the Tag model:</p><pre class="sourceCode"><code class="sourceCode ruby"> <span class="kw">def</span> members=(member)<br /> <span class="dv">self</span>.key[<span class="st">:members</span>].sadd(member.key)<br /> member.tag! <span class="dv">self</span>.name <span class="kw">unless</span> member.tags.member?(<span class="dv">self</span>)<br /> <span class="kw">end</span></code></pre><p>Because Links and Tags are a one-to-many across multiple models, I drop down to Redis and use <code>sadd</code> to add the object to a Redis set of objects sharing the same tag.</p><p>It also has a very handy feature which is how the core of Watches are done. You can define hooks at any phase of Redis interaction - before and after saves, creates, updates and deletes. the entire Watch system is nothing more than calling these post hooks to format the state of the object as JSON, add metadata and send the message using <code>PUBLISH</code> messages to Redis with the Noah namespace as the channel.</p><h2 id="distribution-vectors">Distribution vectors</h2><p>I've used this phrase with a few people. Essentially, I want as many people as possible to be able to use the Noah server component. I've kept the Ruby dependencies to a minimum and I've made sure that every single one works on MRI 1.8.7 up to 1.9.2 as well as JRuby. I already distribute the most current release as a war that can be deployed to a container or run standalone. I want the lowest barrier to entry to get the broadest install base possible. When a new PaaS offering comes out, I pester the hell out of anyone I can find associated with it so I can get deploy instructions written for Noah. So far you can run it on Heroku (using the various hosted Redis providers), CloudFoundry and dotcloud.</p><p>I'm a bit more lax on the callback daemon. Because it can be written in any language that can talk to the Redis pubsub system and because it has "stricter" performance needs, I'm willing to make the requirements for the "official" daemon more stringent. It currently ONLY works on MRI (mainly due to the em-hiredis requirement).</p><h2 id="doing-things-differently">Doing things differently</h2><p>Some people have asked me why I didn't use technology A or technology B. I think I addressed that mostly above but I'll tackle a couple of key ones.</p><h3 id="zeromq">ZeroMQ</h3><p>The main reason for not using 0mq was that I wasn't really aware of it. Were I to start over and still be using Ruby, I'd probably give it a good strong look. The would still be the question of the storage component though. There's still a possible place for it that I'll address in part four.</p><h3 id="nats">NATS</h3><p>This was something I simply had no idea about until I started poking around the CloudFoundry code base. I can almost guarantee that NATS will be a part of Noah in the future. Expect much more information about that in part four.</p><h3 id="mongodb">MongoDB</h3><p>You have got to be kidding me, right? I don't trust my data (or anyone else's for that matter) to a product that doesn't understand what durability means when we're talking about databases.</p><h3 id="insert-favorite-data-store-here">Insert favorite data store here</h3><p>As I said, Redis was the best way to get multiple required functionality into a single product. Why does a data storage engine have a pubsub messaging subsystem built in? I don't know off the top of my head but I'll take it.</p><h2 id="wrap-up---part-3">Wrap up - Part 3</h2><p>So again, because I evidently like recaps, here's the take away:</p><ul><li>The key components in Noah are Redis and Sinatra</li>
<li>Noah is written in Ruby because of time constraints in learning a new language</li>
<li>Noah strives for the server component to have the broadest set of distribution vectors as possible</li>
<li>Ruby dependencies are kept to a minimum to ensure the previous point</li>
<li>The lightest possible abstractions (Ohm) are used.</li>
<li>Stricter requirements exist for non-server components because of flexibility in alternates</li>
<li>I really should learn me some erlang</li>
<li>I'm not a fan of MongoDB</li>
</ul><p>If you haven't guessed, I'm doing one part a night in this series. Tomorrow is part four which will cover the future plans for Noah. I'm also planning on a bonus part five to cover things that didn't really fit into the first four.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-7771972765897556432011-05-17T22:38:00.000-04:002011-05-17T22:38:25.570-04:00On Noah - Part 2<h1 id="on-noah---part-2">On Noah - Part 2</h1><p><em>This is the second part in a series on Noah. Part 1 is available</em> <a href="http://goo.gl/l3Mgt">here</a></p><p>In part one of this series, I went over a little background about ZooKeeper and how the basic Zookeeper concepts are implemented in Noah. In this post, I want to go over a little bit about a few things that Noah does differently.</p><h2 id="noah-primitives">Noah Primitives</h2><p>As mentioned in the previous post, Noah has 5 essential data types, four of which are what I've interchangeably refered to as either Primitives and Opinionated models. The four primitives are Host, Service, Application and Configuration. The idea was to map some common use cases for Zookeeper and Noah onto a set of objects that users would find familiar.</p><p>You might detect a bit of Nagios inspiration in the first two.</p><dl><dt>Host</dt>
<dd>Analogous to a traditional host or server. The machine or instance running the operating system. Unique by name.
</dd>
<dt>Service</dt>
<dd>Typically mapped to something like HTTP or HTTPS. Think of this as the listening port on a Host. Services must be bound to Hosts. Unique by service name and host name.
</dd>
<dt>Application</dt>
<dd>Apache, your application (rails, php, java, whatever). There's a subtle difference here from Service. Unique by name.
</dd>
<dt>Configuration</dt>
<dd>A distinct configuration element. Has a one-to-many relationship with Applications. Supports limited mime typing.
</dd>
</dl><p>Hosts and Services have a unique attribute known as <code>status</code>. This is a required attribute and is one of <code>up</code>,<code>down</code> or <code>pending</code>. These primitives would work very well integrated into the OS init process. Since Noah is curl-friendly, you could add something globally to init scripts that updated Noah when your host is starting up or when some critical init script starts. If you were to imagine Noah primitives as part of the OSI model, these are analagous to Layers 2 and 3.</p><p>Applications and Configurations are intended to feel more like Layer 7 (again, using our OSI model analogy). The differentiation is that your application might be a Sinatra or Java application that has a set of Configurations associated with it. Interestingly enough, you might choose to have something like Tomcat act as both a Service AND an Application. The aspect of Tomcat as a Service is different than the Java applications running in the container or even Tomcat's own configurations (such as logging).</p><p>One thing I'm trying to pull off with Configurations is limited mime-type support. When creating a Configuration in Noah, you can assign a <code>format</code> attribute. Currently 3 formats or types are understood:</p><ul><li>string</li>
<li>json</li>
<li>yaml</li>
</ul><p>The idea is that, if you provide a type, we will serve that content back to you in that format when you request it (assuming you request it that way via HTTP headers). This should allow you to skip parsing the JSON representation of the whole object and instead use it directly. Right now this list is hardcoded. I have a task to convert this.</p><p>Hosts and Services make a great "canned" structure for building a monitoring system on top of Noah. Applications and Configurations are a lightweight configuration management system. Obviously there are more uses than that but it's a good way to look at it.</p><h2 id="ephemerals">Ephemerals</h2><p>Ephemerals, as mentioned previously, are closer to what Zookeeper provides. The way I like to describe Ephemerals to people is a '512 byte key/value store with triggers' (via Watch callbacks). If none of the Primitives fit your use case, the Ephemerals make a good place to start. Simply send some data in the body of your post to the url and the data is stored there. No attempt is made to understand or interpret the data. The hierarchy of objects in the Ephemeral namespace is completely arbitrary. Data living at <code>/ephemerals/foo</code> has no relationship with data living at <code>/ephemerals/foo/bar</code>.</p><p>Ephemerals are also not browseable except via a Linking and Tagging.</p><h2 id="links-and-tags">Links and Tags</h2><p>Links and Tags are, as far as I can tell, unique to Noah compared to Zookeeper. Because we namespace against Primitives and Ephemerals, there existed the need to visualize objects under a custom hierarchy. Currently Links and Tags are the only way to visualize Ephemerals in a JSON format.</p><p>Tags are pretty standard across the internet by now. You might choose to tag a bunch of items as <code>production</code> or perhaps group a set of Hosts and Services as <code>out-of-service</code>. Tagging an item is a simple process in the API. Simply <code>PUT</code> the name of the tag(s) to the url of a distinct named item appended by <code>tag</code>. For instance, the following JSON posted to <code>/applications/my_kick_ass_app/tag</code> with tag the Application <code>my_kick_ass_app</code> with the tags <code>sinatra</code>, <code>production</code> and <code>foobar</code>:</p><pre class="sourceCode"><code class="sourceCode json">{<span class="dt">"tags"</span>:[<span class="st">"sinatra"</span>, <span class="st">"production"</span>, <span class="st">"foobar"</span>]}</code></pre><p>Links work similar to Tags (including the act of linking) except that the top level namespace is now replaced with the name of the Link. The top level namespace in Noah for the purposes of Watches is <code>//noah</code>. By linking a group of objects together, you will be able to (not yet implemented), perform operations such as Watches in bulk. For instance, if you wanted to be informed of all changes to your objects in Noah, you would create a Watch against <code>//noah/*</code>. This works fine for most people but imagine you wanted a more multi-tenant friendly system. By using links, you can group ONLY the objects you care about and create the watch against that link. So <code>//noah/*</code> becomes <code>//my_organization/*</code> and only those changes to items in that namespace will fire for that Watch.</p><p>The idea is also that other operations outside of setting Watches can be applied to the underlying object in the link as well. The name Link was inspired by the idea of symlinking.</p><h2 id="watches-and-callbacks">Watches and Callbacks</h2><p>In the first post, I mentioned that by nature of Noah being "disconnected", Watches were persistent as opposed to one-shot. Additionally, because of the pluggable nature of Noah Watches and because Noah has no opinion regarding the destination of a fired Watch, it becomes very easy to use Noah as a broadcast mechanism. You don't need to have watches for each interested party. Instead, you can create a callback plugin that could dump the messages on an ActiveMQ Fanout queue or AMQP broadcast exchange. You could even use multicast to notify multiple interested parties at once.</p><p>Again, the act of creating a watch and the destination for notifications is entirely disconnected from the final client that might use the information in that watch event.</p><p>Additionally, because of how changes are broadcast internally to Noah, you don't even have to use the "official" Watch method. All actions in Noah are published post-commit to a pubsub queue in Redis. Any language that supports Redis pubsub can attach directly to the queue and PSUBSCRIBE to the entire namespace or a subset. You can write your own engine for listening, filtering and notifying clients.</p><p>This is exactly how the Watcher daemon works. It attaches to the Redis pubsub queue, makes a few API calls for the current registered set of watches and then uses the watches to filter messages. When a new watch is created, that message is like any other change in Noah. The watcher daemon sees that and immediately adds it to its internal filter. This means that you can create a new watch, immediately change the watched object and the callback will be made.</p><h2 id="wrap-up---part-two">Wrap up - Part Two</h2><p>So to wrap up:</p><ul><li>Noah has 5 basic "objects" in the system. Four of those are opinionated and come with specific contracts. The other is a "dumb" key/value store of sorts.</li>
<li>Noah provides Links and Tags as a way to perform logical grouping of these objects. Links replace the top-level hierarchy.</li>
<li>Watches are persistent. The act of creating a watch and notifying on watched objects is disconnected from the final recipient of the message. System A can register a watch on behalf of System B.</li>
<li>Watches are nothing more than a set of filters applied to a Redis pubsub queue listener. Any language that supports Redis and its pubsub queue can be a processor for watches.</li>
<li>You don't even have to register any Watches in Noah if you choose to attach and filter yourself.</li>
</ul><p>Part three in this series will discuss the technology stack under Noah and the reasoning behind it. A bit of that was touched on in this post. Part four is the discussion about long-term goals and roadmaps.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-45918593766510026072011-05-16T23:16:00.001-04:002011-05-16T23:20:45.949-04:00On Noah - Part 1<h1 id="on-noah---part-1">On Noah - Part 1</h1><p><em>This is the first part in a series of posts going over Noah</em></p><p>As you may have heard (from my own mouth no less), I've got a smallish side project I've been working on called <a href="https://github.com/lusis/Noah">Noah</a>.</p><p>It's a project I've been wanting to work on for a long time now and earlier this year I got off my ass and started hacking. The response has been nothing short of overwhelming. I've heard from so many people how they're excited for it and nothing could drive me harder to work on it than that feedback. To everyone who doesn't run away when I talk your ear off about it, thank you so much.</p><p>Since I never really wrote an "official" post about it, I thought this would be a good opportunity to talk about what it is, what my ideas are and where I'd like to see it go in the future.</p><h2 id="so-why-noah">So why Noah?</h2><p><em>fair warning. much of the following may be duplicates of information in the Noah wiki</em></p><p>The inspiration for Noah came from a few places but the biggest inspiration is <a href="http://goo.gl/WGCxY">Apache Zookeeper</a>. Zookeeper is one of those things that by virtue of its design is a BUNCH of different things. It's all about perspective. I'm going to (yet again) paste the description of Zookeeper straight from the project site:</p><pre><code>ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
</code></pre><p>Now that might be a bit confusing at first. Which is it? Is it a configuration management system? A naming system? It's all of them and, again, it's all about perspective.</p><p>Zookeeper, however, has a few problems for my standard use case.</p><ul><li>Limited client library support</li>
<li>Requires persistent connections to the server for full benefit</li>
</ul><p>By the first, I mean that the only official language bindings are C and Java. There's contributed Python support and Twitter maintains a Ruby library. However all of these bindings are "native" and must be compiled. There is also a command-line client that you can use for interacting as well - one in Java and two C flavors.</p><p>The second is more of a showstopper. Zookeeper uses the client connection to the server as in-band signaling. This is how watches (discussed in a moment) are communicated to clients. Persistent connections are simply not always an option. I can't deploy something to Heroku or AppEngine that requires that persistent connection. Even if I could, it would be cost-prohibitive and honestly wouldn't make sense.</p><p>Looking at the list of features I loved about ZK, I thought "How would I make that work in the disconnected world?". By that I mean what would it take to implement any or all of the Zookeeper functionality as a service that other applications could use?</p><p>From that thought process, I came up with Noah. The name is only a play on the concept of a zookeeper and holds no other real significance other than irritation at least two people named Noah when I talk about the project.</p><p>So working through the feature list, I came up with a few things I <strong>REALLY</strong> wanted. I wanted Znodes, Watches and I wanted to do it all over HTTP so that I could have the broadest set of client support. JSON is really the defacto standard for web "messaging" at this point so that's what I went with. Basically the goal was "If your language can make HTTP requests and parse JSON, you can write a Noah client"</p><h2 id="znodes-and-noah-primitives">Znodes and Noah primitives</h2><p>Zookeeper has a shared hierarchical namespace similar to a UNIX filesystem. Points in the hierarchy are called <code>znodes</code>. Essentially these are arbitrary paths where you can store bits of data - up to 1MB in size. These znodes are unique absolute paths. For instance:</p><pre class="sourceCode"><code class="sourceCode mediawiki"> /
/systems
/foo
/bar
/networks
/kansas
/router-1
/router-2</code></pre><p>Each fully qualified path is a unique znode. Znodes can be ephemeral or persistent. Zookeeper also has some primitives that can be applied to znodes such as 'sequence`.</p><p>When I originally started working on Noah, so that I could work with a model, I created some base primitives that would help me demonstrate an example of some of the use cases:</p><ul><li>Host</li>
<li>Service</li>
<li>Application</li>
<li>Configuration</li>
</ul><p>These primitives were actual models in the Noah code base with a strict contract on them. As an example, Hosts must have a status and can have any number of services associated with them. Services MUST be tied explicity to a host. Applications can have Configurations (or not) and Configurations can belong to any number of Applications or not. Additionally, I had another "data type" that I was simply calling Ephemerals. This is similar to the Zookeeper znode model. Originally I intended for Ephemerals to be just that - ephemeral. But I've backed off that plan. In Noah, Ephemerals can be either persistent or truely ephemeral (not yet implemented).</p><p>So now I had a data model to work with. A place to store information and flexibility to allow people to use the predefined primitives or the ephemerals for storing arbitrary bits of information.</p><h2 id="living-the-disconnected-life">Living the disconnected life</h2><p>As I said, the model for my implementation was "disconnected". When thinking about how to implement Watches in a disconnected model, the only thing that made sense to me was a callback system. Clients would register an interest on an object in the system and when that object changed, they would get notified by the method of their choosing.</p><p>One thing about Watches in Zookeeper that annoys me is that they're one-shot deals. If you register a watch on a znode, once that watch is triggered, you have to REREGISTER the watch. First off this creates, as documented by the ZK project, a window of opportunity where you could miss another change to that watch. Let's assume you aren't using a language where interacting with Zookeeper is a synchronous process:</p><ul><li>Connect to ZK</li>
<li>Register watch on znode</li>
<li>Wait</li>
<li>Change happens</li>
<li>Watch fires</li>
<li>Process watch event</li>
<li>Reregister watch on znode</li>
</ul><p>In between those last two steps, you risk missing activity on the znode. In the Noah world, watches are persistent. This makes sense for two reasons. The first is that the latency between a watch callback being fired and proccessed could be much higher than the persistent connection in ZK. The window of missed messages is simply much greater. We could easily be talking 100's of milliseconds of latency just to get the message and more so to reregister the watch.</p><p>Secondly, the registration of Watches in Noah is, by nature of Noah's design and as a byproduct, disconnected from the consumer of those watches. This offers much greater flexibility in what watches can do. Let's look at a few examples.</p><p>First off, it's important to understand how Noah handles callbacks. The message format of a callback in Noah is simply a JSON representation of the changed state of an object and some metadata about the action taken (i.e. delete, create, update). Watches can be registered on distinct objects, a given path (and thus all the children under that path) and further refined down to a given action. Out of the box, Noah ships with one callback handler - http. This means that when you register a watch on a path or object, you provide an http endpoint where Noah can post the aforementioned JSON message. What you do with it from there is up to you.</p><p>By virtue of the above, the callback system is also designed to be 'pluggable' for lack of a better word. While the out of the box experience is an http post, you could easily write a callback handler that posted the message to an AMQP exchange or wrote the information to disk as a flat file. The only requirement is that you represent the callback location as a single string. The string will be parsed as a url and broken down into tokens that determine which plugin to call.</p><p>So this system allows for you to distribute watches to multiple systems with a single callback. Interestingly enough, this same watch callback system forms the basis of how Noah servers will share changes with each other in the future.</p><h2 id="wrap-up---part-1">Wrap up - Part 1</h2><p>So wrapping up what I've discussed, here are the key take aways:</p><ul><li>Noah is a 'port' of specific Zookeeper functionality to a disconnected and asynchronous world</li>
<li>Noah uses HTTP and JSON as the interface to the server</li>
<li>Noah has both traditional ZK-style Ephemerals as well as opinionated Primitives</li>
<li>Noah uses a pluggable callback system to approximate the Watch functionality in Zookeeper</li>
<li>Clients can be written in any language that can speak HTTP and understand JSON (yes, even a shell script)</li>
</ul><h2 id="part-2-and-beyond">Part 2 and beyond</h2><p>In part two of this series we'll discuss some of the additions to Noah that aren't a part of Zookeeper such as Tags and Links. Part 3 will cover the underlying technology which I am intentionally not discussing at this point. Part 4 will be a roadmap of my future plans for Noah.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-25098136072514800832011-04-22T02:51:00.001-04:002011-04-22T09:47:52.864-04:00Who owns my availability?Hey did you know EC2 had problems today? Yeah nothing major just a total effing collapse of the EBS system at US-EAST-1.<br />
<br />
You know what that means....<br />
<br />
<blockquote>"Hey guys, can anyone tell me who owns my availability?"</blockquote><blockquote>"Internet learns lesson of putting "all eggs in the EC2 basket". Buy your own machines, brothers."</blockquote><br />
I could go on....but I won't. I'm also going to stop short of posting a CeeLo video at this point.<br />
<br />
Your stupid little comments mean nothing. I especially find it hilarious that someone from Twitter would make a comment availability. I also find the short-lived memory of some people hilarious (paraphrasing here):<br />
<br />
<blockquote>"Thank god we're hosted on Joyent/Linode/My mom's basement"</blockquote><br />
Please. Your attempt to curry favor and free service with your provider are transparent and frankly, makes you look stupid.<br />
<br />
Yo Netflix/SimpleGeo/JRandomDude I'm happy for you and and all. I'ma let you finish but....<br />
<br />
So who DOES own my availability?<br />
Here's a hint; it's not always that simple.<br />
<br />
Yes, the ultimate responsibility for those impacted lies with those who were impacted but let's look at a few facts (or excuses - if you're being a dick about it):<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><b>Not everyone has the resources of a Netflix</b></span><br />
Comparing anyone else's EC2 usage to Netflix is simply retarded. It's a lot like working with an ex-Google employee (I've worked with a few). They have some awesome ideas and learned some great stuff there but guess what? About 85% of it is USELESS to anyone except someone the size of Google. What works at Google doesn't work at my company.<br />
<br />
It's not even a matter of scaling down the concept. It's simply NOT possible. Yeah let me just go buy a shipping container and build a datacenter in a box. Hardware failure? Replace the box with one off the shelf. Oh wait, not everyone has a warehouse of replacement servers. People have trouble getting a few spare hard drives to swap out.<br />
<br />
Telling someone that they should just do what Netflix does makes you look stupid. Not them.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><b>WE used Joyent/Linode/GoGrid/My mom's basement</b></span><br />
Really? Really? I'm not being an AWS fanboy here but here is a simple fact: No other 'cloud' provider comes even REMOTELY close to the feature set of AWS. No one. Not only does no one come close but Amazon is CONSTANTLY iterating on new stuff to widen the gap even more.<br />
<br />
It's not like your provider hasn't had a major outage in recent memory. And comparing an effing VPS provider to Amazon? You seriously just don't get it.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><b>You should have designed around this possibility</b></span><br />
Well no shit, sherlock. Guess what, it was rejected. Why? Who knows? Who cares? It's irrelevant. Sometimes the decision isn't ours to make. In the REAL world, people have to balance risk vs. reward.<br />
<br />
Here's a tidbit of information. At EVERY single company I've been at where I was involved with architecting a solution from the ground up, we never had redundancy built in from the get go. Did I find it appalling. Absolutely but the choice wasn't mine. I did the best I could to prevent anything that would make adding it TOO difficult later on but we didn't have our DR site online from day one. We sometimes had to accrue a little technical debt. The best we could do was to minimize it as much as possible.<br />
<br />
Designing around failure is not the same as designing for the worse case scenario. Sometimes you just have to accept that "if component X has Y number of failures, we're going to have an outage". If you have the ability to deal with it now (resources/money/whatever), then that's awesome. Sometimes you just have to accept that risk.<br />
<br />
Oh sure I'd love to use (insert buzzword/concurrent/distributed language of the day) here. But I can't. It would be totally awesome if everything were designed from the ground up to handle that level of failure but it's not.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><b>And another thing</b></span><br />
The thing that bothers me most is the two-faced attitude around it all. <br />
<br />
On one hand people are telling you it's stupid to host your own hardware. On the other hand they'll laugh at you when your provider has an outage and tell you that you should have built your own. <br />
<br />
On one hand they'll tell you it's stupid to use some non-traditional new-fangled language and on the other hand laugh at you when you could have avoided all these problems if you had just used non-traditional new-fangled language.<br />
<br />
On one hand they'll tell you that you should use insert-traditional-RDBMS here and on the other hand say that it's your fault for not rearchitecting your entire codebase around some NoSQL data store.<br />
<br />
Not everyone has the same options. I hate the phrase "hindsight is 20/20". Why? Because it's all relevant. Sometimes you don't know that something is the wrong choice till it bites you in the ass. Hindsight in technology is only valuable for about a year. Maybe 6 months. Technology moves fast. It's easy to say that someone should have used X when you don't realize that they started working on things six months before X came along. If you have that kind of foresight, I'd love to hire you to play the stock market for me.<br />
<br />
Not everyone has the luxury of switching midstream. You have to make the most what technology is available. If you keep chasing the latest and greatest, you'll never actually accomplish anything.<br />
<br />
Are these excuses? Absolutely but there's nothing inherently wrong with excuses. You live and learn. So to those affected by the outage (still on-going mind you), take some comfort. Learn from your mistakes. The worst thing you could do at this point would be to <b>NOT</b> change anything. At a minimum, if you aren't the decision maker, you should document your recommendations and move on. If you are the decision maker, you need to..you know...decide if the risk of this happening again is acceptable.lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-19134754874853225202011-04-15T04:06:00.001-04:002011-04-15T04:07:37.036-04:00Sinatra, Noah and CloudFoundry - the dirty detailsSo via some magical digital god, my signup for Cloud Foundry got processed. Obviously my first thought was to try and get Noah up and running. Cloud Foundry is a perfect fit for Noah because I have access to Redis natively. I have a working setup now but it took a little bit of effort.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><u>Getting set up</u></span><br />
As with everything these days, my first action was to create a gemset. I'll not bore you with that process but for the sake of this walkthrough, let's use a 1.9.2 gemset called '<i>cfdev</i>'.<br />
<br />
The VMC getting started guide has most of the information you'll need but I'm going to duplicate some of it here for completeness:<br />
<br />
<br />
<pre><span class="Apple-tab-span" style="white-space: pre;"> </span>gem install vmc
<span class="Apple-tab-span" style="white-space: pre;"> </span>vmc target api.cloudfoundry.com
<span class="Apple-tab-span" style="white-space: pre;"> </span>vmc login
</pre><br />
<br />
And we're ready to rock. The VMC command line help is very good with the exception that the optional args aren't immediately visible.<br />
<br />
<br />
<pre>vmc help options</pre><br />
<br />
will give you a boatload of optional flags you can pass in. One that was frequently used during the demos at launch was '<i>-n</i>'. I would suggest you NOT use that for now. The prompts are actually pretty valuable.<br />
<br />
So in the case of Noah, we know we're going to need a Redis instance. Because everything is allocated dynamically CloudFoundry makes heavy use of environment variables to provide you with important settings you'll need.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><u>First Attempt</u></span><br />
If you watched the demo (or read the quickstart Sinatra example), there's a demo app called 'env' that they walk you through. You're going to want to use that when troubleshooting things. My first task was to duplicate the env demo so I could take a gander at the variables I would need for Redis. For the record, the steps I'm documenting here might appear out of order and result in some wasted time. I'm one of those guys who reads the instructions 2 days after I've broken something so you have an idea of what I did here:<br />
<br />
<br />
<pre><span class="Apple-tab-span" style="white-space: pre;"> </span>vmc help
<span class="Apple-tab-span" style="white-space: pre;"> </span>vmc services
<span class="Apple-tab-span" style="white-space: pre;"> </span>vmc create-service redis redis-noah
<span class="Apple-tab-span" style="white-space: pre;"> </span>vmc services
</pre><br />
<br />
At this point, I now have a named instance of redis. The reason I felt safe enough doing this now is that I noticed in the help two service commands - 'bind-service' and 'unbind-service'. I figured it was easy enough to add the service to my app based on those options.<br />
<br />
So go ahead and create the env app per the getting started documentation. If you followed my suggestion and DIDN'T disable prompts, you'll get the option to bind you app to a service when you push the first time. If you're running without prompts (using the '-n' option), you'll probably want to do something like this:<br />
<br />
<br />
<pre>vmc push myenvapp --url ohai-env.cloudfoundry.com
vmc bind-service my_redis_service myenvapp
</pre><br />
<br />
If you visit the url you provided (assuming it wasn't taken already?) at /env, you'll get a big dump of all the environment variables. The ones that you'll need be using most are probably going to be under `<i>VCAP_SERVICES</i>`. What you'll probably also notice is that `<i>VCAP_SERVICES</i>` is a giant JSON blob. Now you may also notice that there's a nice `VMC_REDIS` env variable there. It's pretty useless primarily because there's also a GIANT warning in the env output that all `<i>VMC_</i>` environment variables are deprecated but also because your redis instance requires a password to access which means you need to traverse the JSON blob ANYWAY.<br />
<br />
So if we paste the blog into an IRB session we can get a better representation. I wish I had done that first. Instead, I reformatted it with jsonlint dutifully wrote the following madness:<br />
<script src="https://gist.github.com/921341.js?file=1.rb">
</script><br />
which I spent a good 30 minutes troubleshooting before I realized that it's actually an array. It should have been this:<br />
<script src="https://gist.github.com/921341.js?file=2.rb">
</script><br />
<br />
So now that I had all the variables in place, I went about converting my <a href="https://github.com/lusis/Noah/wiki/Demo-Instance">heroku Noah demo </a>. That demo uses a Gemfile and a rackup file so I figured it would work just fine here. No such luck. This is where things get hairy.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><u>Sinatra limitations</u></span><br />
The short of it is that Sinatra application support on CF right now is a little bit of a CF. It's very basic and somewhat brute force. If you're running a single file sinatra application, it will probably work. However if you're running anything remotely complex, it's not going to work without considerable effort. Noah is even more of a special case because it's distributed as a gem. This actually has some benefit as I'll mention farther down. However it's not really "compatible" with the current setup on Cloud Foundry. Here's the deal:<br />
<br />
If you look <a href="https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/sinatra/plugin.rb#L21">here</a>, You'll see that the way your sinatra application is start is by calling ruby (with or without bundler depending) against what it detects as your main app file. This is done <a href="https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/common.rb#L442-454">here</a> which leads us all the way to this file:<br />
<br />
`<a href="https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/manifests/sinatra.yml">https://github.com/cloudfoundry/vcap/blob/master/cloud_controller/staging/manifests/sinatra.yml</a>`<br />
<br />
Essentially for sinatra applications, the first .rb file it comes across with '<i>require sinatra</i>', is considered the main app file. Bummer. So <i>config.ru</i> is out. The next step is to rename it to a '.rb' file and try again. This is where I spent most of my troubleshooting. There's a gist of the things I tried (including local testing) here:<br />
<br />
`<a href="https://gist.github.com/920552">https://gist.github.com/920552</a>`<br />
<br />
Don't jump to the solution just yet because it's actually incomplete. This troubleshooting led to another command you'll want to remember:<br />
<br />
<br />
<pre>vmc files myapp logs/stderr.log</pre><br />
<br />
I found myself typing it a lot during this process. For whatever reason, possibly due to bundler or some other vcap magic I've not discovered yet what works at home does not work on Cloud Foundry exactly the same. That's fine, it's just a matter of knowing about it. It also didn't help that I wasn't getting any output at all for the entire time I was trying to figure out why config.ru didn't work.<br />
<br />
Thanks to Konstantin Haase for his awesome suggestion in #sinatra. The trick here was to mimic what rackup does. Because the currently released Noah gem has a hard requirement on rack 1.2.1, his original suggestion wasn't an exact fit but I was able to get something working:<br />
<br />
<a href="https://gist.github.com/921292">https://gist.github.com/921292</a><br />
<br />
<span class="Apple-style-span" style="font-size: large;"><u>So what did we do?</u></span><br />
Ensure that the wrapper file is picked up first by making sure it's the ONLY rb file uploaded with `require sinatra` at the top.<br />
Because of a bug in rack 1.2.1 with <i><b>Rack::Server.new</b></i>, I HAD to create a file called config.ru. The fix in rack 1.2.2 actually honors passing all the options into the constructor without needing the config.ru file.<br />
Explicitly connect to redis before we start the application up.<br />
<br />
The last one was the almost as big of a pain in the ass as getting the application to start up.<br />
<br />
I think (and I'm not 100% sure) that you are prohibited from setting environment variables inside your code. Because of the convoluted way I had to get the application started, I couldn't use my sinatra configuration block properly (`<i>set :redis_url, blahblahblah</i>`). I'm sure it's possible but I'm not an expert at rack and sinatra. I suppose I could have used Noah::App.set but at this point I was starting to get frustrated. Explicitly setting it via Ohm.connect worked.<br />
<br />
I'm almost confident of this environment variable restriction because you can see options in 'vmc help' that allow you to pass environment variables into your application. That would work fine for most cases except that I don't know what the redis values are outside of the app and they're set dynamically anyway.<br />
<br />
<span class="Apple-style-span" style="font-size: large;"><u>So where can things improve?</u></span><br />
First off, this thing is in beta. I'm only adding this section because it'll serve as a punch list of bugs for me to fix in vcap ;)<br />
<br />
<br />
<ul><li>Sinatra support needs to be more robust.</li>
</ul><br />
You can see that the developers acknowledged that in the staging plugin code. There are TODOs listed. It's obvious that a sinatra application of any moderate complexity wasn't really tested and that's fine. The building blocks are there and the code is opensource. I'll fix it myself (hopefully) and submit a pull request.<br />
<br />
<ul><li>Allow override of the main app file from VMC.</li>
</ul><br />
It appears from the various comments that the node.js support suffers some of the same brute force detection routines. An option to pass in what the main applictation file is would solve some of that.<br />
<br />
<ul><li>Document the environment variable restrictions.</li>
</ul><br />
I didn't see any documentation anywhere about that restriction (should it exist). I could be doing something wrong too. It's worth clarifying.<br />
<br />
<ul><li>Better error reporting for failed startups</li>
</ul><br />
I'm not going to lie but I spent a LONG time troubleshooting the fact that the app simply wasn't starting up. The default output when a failure happens during deploy is the staging.log file. All this EVER contained was the output from bundler. It should include the output of stderr.log and stdout.log as well. Also an explicit message should be returned if the main app file can't be detected. That would have solved much of my frustration up front.<br />
<br />
That's just the stuff I ran into to get things going. The first item is the biggest one. If you're writing a monolithic single-file sinatra app, the service will work GREAT. If you aren't, you'll have to jump through hoops and wrapper scripts for now. Supporting rackup files for Sinatra and Rack apps will go a long way to making things even more awesome.<br />
<br />
One pleasant surprise I found was that, despite what I was told, I didn't need to include every gem in my Gemfile. Because Noah itself has its deps, Bundler pulls those in for me.<br />
<br />
I've created a git repo with the code as well as a quickstart guide for getting your own instance running. You can find it here:<br />
<br />
<a href="https://github.com/lusis/noah-cloudfoundry-demo">https://github.com/lusis/noah-cloudfoundry-demo</a>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com3tag:blogger.com,1999:blog-934985301455705990.post-84200974664136855342011-04-14T07:36:00.002-04:002011-04-14T07:36:30.636-04:00Operational Primitives"Infrastructure as code". I love the phrase. Where devops is a word that is sadly open to so much (mis)interpretation, "Infrastructure as code" is pretty clear. Treat your infrastructure as code. Programmable. Testable. Deployable.<br />
<br />
But when you start to really think about that concept, there's a deep dive you can take, navigating various programming and computer science constructs and applying those to your infrastructure.<br />
<br />
I've been working pretty heavily on getting the first API stable release of <a href="http://goo.gl/BQCvs">Noah</a> out the door. It's been a challenge with the schedule I have to work on it - which is essentially "when everyone else in the house is asleep and I'm awake'. Last night, I came to a fork in the road where I needed to make a decision. This decision would lock me into an API path that I was unwilling to change for a while. Nobody wants to use a service or tool with a constantly changing API. I needed to shit or get off the pot, to use a creative euphemism. With the announcements of both <a href="https://github.com/bmizerany/doozer">Doozer</a> and <a href="https://github.com/jtuple/riak_zab">riak_zab</a>, it was clear that I wasn't the only person attempting to tackle the ZooKeeper space.<br />
<br />
Since Github lacks any facility for soliciting project feedback (hint hint, @github), I decided to create a Wufoo form and tweet it out. I don't have a very big audience but I was hoping it would at least get to the people who were likely to use Noah. The form was fairly simple with one question on something that I had pretty summarily dismissed early on - <b>HATEOAS</b> (<i>hypermedia as the engine of application state</i>).<br />
<br />
<span class="Apple-style-span" style="font-size: large;">A small HATEOAS diversion</span><br />
<span class="Apple-style-span" style="font-size: large;"><br />
</span><br />
The HATEOAS debate is a lot like Linux vs. GNU/Linux. It's fairly esoteric but there's some meat to the matter. My problem with it was simply that, despite what Roy Fielding and others intended, REST had taken on a new definition and it wasn't the strict HATEOAS one. Additionally, I found it VERY difficult to map HATEOAS concepts to JSON. JSON is a great format but a rich document structure is not (rightly so) part of the format. It's intended to be simple, easily read and cleanly mapped to machine readable format. It also felt like extra work on the part of the API consumer. The concepts that we use when reading a website (click this link, read this list, click this link) are simple not necessary when you have a contextually relevant (or descriptive) URL scheme. True, as a human I don't make changes in the URL bar to navigate a site (I use the links provided by the site) but when it comes to dealing with an API, I don't exhibit the same usage patterns as a web browser. I'm making distinct atomic transactions (DELETE this resource, PUT this resource) at a given endpoint. These simply aren't the same as filling out forms and are only tangentially related. I'm simply not willing to force someone to parse a JSON object to tell them how to create a new object in the system. The API for Noah is fairly simple as it is. Objects in the system have only two or three required attributes for a given operation and normally one of those attributes is directly inferable from the URL.<br />
<br />
But based on the poll results thus far, I wanted to give the idea fair consideration which led me to think about what types of objects Noah had in its system.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Primitives</span><br />
<span class="Apple-style-span" style="font-size: large;"><br />
</span><br />
For those who aren't familiar or simple don't know, there's a term in computer science and programming called "Primitive". It essentially means a basic data type in a language from which other complex data types are created. A building block if you will. Some easily grokable examples of primitives are Characters and Integers. Some languages actually have ONE primitive like Object and everything is built on top of that. You could get into a semantic argument about a lot of this so I'm going to leave it at that.<br />
<br />
But back to the phrase "Infrastucture as code". If we start looking at how we "program" our infrastructure, what are the "primitives" that our language supports. I inadvertently created some of these in Noah. I've been calling them the "opinionated models" but really in the infrastructure programming language of Noah, they're primitives.<br />
<br />
When this hit me last night, I immediately pulled out the tablet and went to work on a mind map. I laid out what I had already implemented as primitives in Noah:<br />
<br />
<br />
<ul><li>Host</li>
<li>Service</li>
<li>Application</li>
<li>Configuration</li>
</ul><br />
<br />
I then started to think about other concepts in Noah. Were <i>Ephmerals</i> really a primitive. Not really. If anything Ephemerals are more similar to ruby's <i>BasicObject</i>. The only real attribute <i>Ephemerals</i> have are a path (similar to the <i>object_id</i>).<br />
<br />
So what else would be our modern operational primitives? Remember that we're talking about building blocks here. I don't want to abstract out too much. For instance you could simply say that a "Resource" is the only real operational primitive and that everything else is built on top of that. Also consider that languages such as Python have some richer primitives built-in like tuples.<br />
<br />
One interesting thought I had was the idea that "<i>State</i>" was a primitive. Again, in the world of operations and infrastructure, one of your basic building blocks is if something is available or not - up or down. At first glance it would appear that this maps pretty cleanly to a <i>Boolean</i> (which is a primitive in most languages) however I think it's a richer primitive than that.<br />
<br />
In the world of operations, State is actually quaternary (if that's the right word) rather than binary. There are two distinct areas between up and down that have dramatically different implications on how you interact with it:<br />
<br />
<br />
<ul><li>Up</li>
<li>Down</li>
<li>Pending Up</li>
<li>Pending Down</li>
</ul><br />
<br />
Currently in Noah, we simple have Up, Down and Pending but something that is in the State of shutting down is grossly different than something in the state of Starting up. Look at a database that is queiscing connections. It's in a state of "Pending Down". It's still servicing existing requests. However a database in the state of "Pending Up" is NOT servicing any requests.<br />
<br />
So I'm curious what other thoughts people have. What else are the basic building blocks of modern operations when viewed through the lens of "infrastructure as code"?<br />
<br />
<i>For the record, I'm still pretty confident that Noah still has a place in the Doozer, riak_zab, ZooKeeper world. All three of those still rely on the persistent connection for signaling and broadcast whereas Noah is fundamentally about the disconnected and asynchronous world.</i>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com7tag:blogger.com,1999:blog-934985301455705990.post-45593710534583641262011-04-05T05:24:00.000-04:002011-04-05T05:24:30.586-04:00It does not follow and Wheaton's Law<blockquote>"I'm not a smart guy".</blockquote>I say this quite a bit. I don't say it to fish for compliments or as a chance to have my ego boosted. I say it because I realize that, out of the vast corpus of computer science knowledge that exists, the part that I DO know is a blade of grass on a football field.<br />
<br />
<blockquote>"I'm not a developer"</blockquote><br />
I say this a lot too. This is not meant as a slight to developers. It's meant as a compliment. There are <b>REAL</b> developers out there and I'm just pretending (after a fashion). I have never worked a professional gig as a developer. I've had honest discussions with people who want to pay me lots of money to be a developer. The best way I can explain it to them is that it would be unfair to you, as an employer, to hire me for a developer position because you would be unhappy with the results. In general it takes me twice as long to solve a development problem as it takes a real developer.<br />
<br />
There are lots of factors to this; education, regular skill use and a general affinity for picking up concepts. I never graduated college and I pretty much suck at math. That's not to say I couldn't learn it but there are some things I know I'll never be as good at as someone else and that's fine by me. I'm not settling for mediocrity I just know my limitations. I'll still take a stab at it.<br />
<br />
There are, however, some <b>REALLY</b> smart people out there. I used to follow a bunch of them on Twitter because they would link to or drop ideas that really made me want to go research something. I noticed an interesting trend though about some of them. They had a tendency to be dicks. Not just the occasional "Only an idiot would do X" but outright vitriol. Was it trolling? In some cases, sure, but I honestly got the impression that they actually looked down on people who didn't who use a certain technology or chose any path different than they would have chosen.<br />
<br />
At the other extreme, you have the folks who make snide remarks or drop a non sequitur about a given technology presumably in an attempt to make the in-crowd giggle and the rest of us poor saps wonder what the hell we're doing wrong. I mean these are smart people, right? If they know something I don't about a given technology, then by god, I'd love to know what it is. I'd love to learn why they feel that way. In the end, though, all you hear is giggling in the background and wonder what the big joke was.<br />
<br />
When the hell did we, the people who were typically on the outside of the in-crowd, turn into the people who gave us the most shit growing up? It's like a fucking geek Stockholm Syndrome thing that's gone off the deep end but instead of just sympathizing with our abuser, we're the abuser and we relish it.<br />
<br />
I'm guilty of this behavior. I'm the first in line to criticize MongoDB, for instance. The difference? I'll actually sit down with you and tell you WHY I don't like MongoDB and why I feel it's a bad choice in many situations.<br />
<br />
What I'm asking is that, as one of the people on the outside, educate me. As much as I think Ted Dziuba is a big troll, at least he takes the time to write it down and trys to defend his position. Ben Bleything had an awesome tweet today:<br />
<br />
<blockquote>I guess what I meant is, I don't have the experience to form that opinion, I'd like to learn from you.</blockquote><br />
That's my attitude exactly. "<i>Put up or shut up</i>" is a bit harsh but in the broadest terms, that's what needs to happen. If you think X is superior to Z then say why. There are some of us who could benefit from it.<br />
<br />
<i>Sidebar on Semantics</i><br />
<i><br />
</i><br />
Additionally, let's make sure we're also on the same page in terms of semantics. If we're talking about queues, clarify if you're talking about data structures versus a message queue because there's a big f'ing difference in my mind.<br />
<br />
When I hear queue, I don't think data structure. I think of a message queue in the product sense. That's just my background. I think about things like guaranteed delivery and message durability.lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-12437796928970339002011-03-08T02:01:00.000-05:002011-03-08T02:01:20.842-05:00Ad-Hoc Configuration, Coordination and the value of changeFor those who don't know, I'm currently in Boston for DevOps Days. It's been amazing so far and I've met some wonderful people. One thing that was REALLY awesome was the open space program that Patrick set up. You won't believe it works until you've tried it. It's really powerful.<br />
<br />
In one of our open spaces, the topic of ZooKeeper came up. At this point I made a few comments, and at the additional prodding of everyone went into a discussion about ZooKeeper and Noah. I have a tendency to monopolize discussions around topics I'm REALLY passionate about so many thanks for everyone who insisted I go on ;)<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Slaughter the deviants!</span><br />
The most interesting part of the discussion about ZooKeeper (or at least the part I found most revealing) was that people tended to have trouble really seeing the value in it. One of the things I've really wanted to do with Noah is provide (via the wiki) some really good use cases about where it makes sense.<br />
<br />
I was really excited to get a chance to talk with Alex Honor (one of the co-founders of DTO along with Damon Edwards) about his ideas after his really interesting blog post around <a href="http://dev2ops.org/blog/2011/2/16/peanut-butter-in-my-chocolate-convergence-vs-ad-hoc-control.html">ad-hoc configuration.</a> If you haven't read it, I suggest you do so.<br />
<br />
Something that often gets brought up and, oddly, overlooked at the same time is the where ad-hoc change fits into a properly managed environment (using a tool like puppet or chef).<br />
<br />
At this point, many of you have gone crazy over the thought of polluting your beautifully organized environment with something so dirty as ad-hoc changes. I mean, here we've spent all this effort on describing our infrastructure as code and you want to come in and make a random, "undocumented" change? Perish the thought!<br />
<br />
However, as with any process or philosophy, strict adherence with out understanding WHEN to deviate will only lead to frustration. Yes, there is a time to deviate and knowing when is the next level of maturity in configuration management.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">So when do I deviate</span><br />
Sadly, knowing when it's okay to deviate is as much a learning experience as it was getting everything properly configured in the first place. To make it even worse, that knowledge is most often specific to the environment in which you operate. The whole point of the phrase ad-hoc is that it's..well...ad-hoc. It's 1 part improvisation/.5 parts stumbling in the dark and the rest is backfilled with a corpus of experience. I don't say this to sound elitist.<br />
<br />
So, really, when do I deviate. When/where/why and how do I deviate from this beautifully described environment? Let's go over some use cases and point out that you're probably ALREADY doing it to some degree.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Production troubleshooting</span><br />
The most obvious example of acceptable deviation is troubleshooting. We pushed code, our metrics are all screwed up and we need to know what the hell just happened. Let's crank up our logging.<br />
<br />
At this point, changing your log level, you've deviated from what your system of record (your CM tool) says you should be. Our manifests, our cookbooks, our templates all have us using a loglevel of ERROR but we just bumped up one server to DEBUG. so we could troubleshoot. That system is now a snowflake. Unless you change that log level back to ERROR, you know have one system that will, until you do a puppetrun of chef-client run is different than all the other servers of the class/role.<br />
<br />
Would you codify that in the manifest? No. This is an exception. A (should be) short-lived exception to the rules you've defined.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Dynamic environments</span><br />
Another area where you might deviate is in highly elastic environments. Let's say you've reached the holy grail of elasticity. You're growing and shrinking capacity based on some external trigger. You can't codify this. I might run 20 instances of my app server now but drop back down to 5 instances when the "event" has passed. In a highly elastic environment, are you running your convergence tool after every spin up? Not likely. In an "event" you don't want to have to take down your load balancer (and thus affect service to the existing intstances) just to add capacity. A bit of a contrived example but you get the idea.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">So what's the answer?</span><br />
I am by far not the smartest cookie in the tool shed but I'm opinionated so that has to count for something. These "exception" events are where I see additional tools like Zookeeper (or my pet project Noah) stepping in to handle things.<br />
<br />
Distributed coordination, dynamically reconfigurable code, elasticity and environment-aware applications.<br />
These are all terms I've used to describe this concept to people. Damon Edwards provided me with the last one and I really like it.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Enough jibber-jabber, hook a brother up!</span><br />
So before I give you the ability to shoot yourself in the foot, you should be aware of a few things:<br />
<br />
<br />
<ul><li>It's not a system of record</li>
</ul><br />
Your DDCS (dynamic distributed coordination service as I'll call it because I can't ever use enough buzzwords) is NOT your system of record. It can be but it shouldn't be. Existing tools provide that service very well and they do it in an idempotent manner.<br />
<br />
<br />
<ul><li>Know your configuration</li>
</ul><br />
This is VERY important. As I said before, much of this is environment specific. The category of information you're changing in this way is more "transient" or "point-in-time". Any given atom of configuration information has a specific value associated with it. Different levels of volatility. Your JDBC connection string is probably NOT going to change that often. However, the number of application servers might be at different amounts of capacity based on some dynamic external factor.<br />
<br />
<br />
<ul><li>Your environment is dynamic and so should be your response</li>
</ul><br />
This is where I probably get some pushback. Just as one of the goals of "devops" was to deal with, what Jesse Robbins described to day as misalignment of incentive, there's an internal struggle where some values are simply fluctuating in near real time. This is what we're trying to address.<br />
<br />
<br />
<ul><li>It is not plug and play</li>
</ul><br />
One thing that Chef and Puppet do very well is that you can, with next to no change to your systems, predefine how something should look or behave and have those tools "make it so".<br />
<br />
With these realtime/dynamic configuration atoms your application needs to be aware of them and react to them intelligently.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Okay seriously. Get to the point</span><br />
So let's take walk through a scenario where we might implement this ad-hoc philosophy in a way that gives us the power we're seeking.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">The base configuration</span><br />
<br />
<ul><li> application server (fooapp) uses memcached, two internal services called "lookup" and "evaluate" and a data store of somekind.</li>
<li>"lookup" and "evaluate" are internally developed applications that provide private REST endpoints for providing a dictionary service (lookup) and a business rule parser of some kind (evaluate).</li>
<li>Every component's base configuration (including the data source that "lookup" and "evaluation" use) is managed, configured and controlled by puppet/chef.</li>
</ul><br />
<br />
In a standard world, we store the ip/port mappings for "lookup" and "evaluate" in our CM tool and tags those. When we do a puppet/chef client run, the values for those servers are populated based on the ip/port information our EXISTING "lookup"/"evaluate" servers.<br />
<br />
This works. It's being done right now.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">So where's the misalignment?</span><br />
What do you do when you want to spin up another "lookup"/"evaluate" server? Well you would probably use a bootstrap of some kind and apply, via the CM tool, the changes to those values. However this now means that for this to take effect across your "fooapp" servers you need to do a manual run of your CM client. Based on the feedback I've seen across various lists, this is where the point of contention exists.<br />
<br />
What about any untested CM changes (a new recipe for instance). I don't want to apply that but if I run my CM tool, I've now not only pulled those unintentional changes but also forced a bounce of all of my fooapp servers. So as a side product of scaling capacity to meet demand, I've now reduced my capacity at another point just to make my application aware of the new settings.<br />
<br />
<span class="Apple-style-span" style="font-size: x-large;">Enter Noah</span><br />
This is where the making your application aware of its environment and allowing it to dynamically reconfigure itself pays off.<br />
<br />
Looking at our base example now, let's do a bit of architectural work around this new model.<br />
<br />
<br />
<ul><li>My application no longer hardcodes a base list of servers prodviding "lookup" and "evaluate" services.</li>
<li>My application understands the value of a given configuration atom</li>
<li>Instead of the hardcoded list, we convert those configuration atoms akin to something like a singleton pattern that points to a bootstrap endpoint.</li>
<li>FooApp provides some sot of "endpoint" where it can be notified of changes to the number/ip addresses or urls available a a given of our services. This can also be proxied via another endpoint.</li>
<li>The "bootstrap" location is managed by our CM tool based on some more concrete configuration - the location of the bootstrap server.</li>
</ul><br />
<br />
Inside our application, we're now:<br />
<br />
<br />
<ul><li>Pulling a list of "lookup"/"evaluate" servers from the bootstrap url (i.e. http://noahserver/s/evaluate)</li>
<li>Registering a "watch" on the above "path" and providing an in-application endpoint to be notified when they change.</li>
<li>validating at startup if the results of the bootstrap call provide valid information (i.e. doing a quick connection test to each of the servers provided by the bootstrap lookup or a subset thereof)</li>
</ul><br />
<br />
If we dynamically add a new transient "lookup" server, Noah fires a notification to the provided endpoint with the details of the change. The application will receive a message saying "I have a new 'lookup' server available". It will run through some sanity checks to make sure that the new "lookup" server really does exist and works. It then appends the new server to the list of existing (permanent servers) and start taking advantage of the increase in capacity.<br />
<br />
That's it. How you implement the "refresh" and "validation" mechanisms is entirely language specific. This also doesn't, despite my statements previously, have to apply to transient resources. The new "lookup" server could be a permanent addition to my infra. Of course this would have been captured as part of the bootstrapping process if that were the case.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Nutshell</span><br />
And that's it in a nutshell. All of this is availalbe in Noah and Zookeeer right now. Noah is currently restricted to http POST endpoints but that will be expanded. Zookeeper treats watches as ephemeral. Once the event has fired, you must register that same watch. With Noah, watches are permanent.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Takeaway</span><br />
I hope the above has made sense. This was just a basic introduction to some of the concepts and design goals. There are plenty of OTHER use cases for ZooKeeper alone. So the key take aways are:<br />
<br />
<br />
<ul><li>Know the value of your configuration data</li>
<li>Know when and where to use that data</li>
<li>Don't supplant your existing CM tool but instead enhance it.</li>
</ul><br />
<br />
<span class="Apple-style-span" style="font-size: large;">Links</span><br />
<a href="http://goo.gl/iTPQD">Noah</a><br />
<a href="http://goo.gl/WGCxY">ZooKeeper</a><br />
<a href="http://goo.gl/oVgbx">Hadoop Book</a> (which has some AMAZING detail around ZooKeeper, the technology and use caseslusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com4tag:blogger.com,1999:blog-934985301455705990.post-17999952197825022742011-02-25T23:28:00.000-05:002011-02-25T23:28:29.473-05:00Thank YouIn case you hadn't heard, today Amazon went all Top Gun today and gave the world <a href="http://aws.amazon.com/cloudformation/">Cloud Formation</a>. This, of course, gave rise to tweets and one-offs from pundits all over the world stating that it was the death knell of tools like Chef and Puppet.<br />
<br />
Amazon had usurped yet another business model with the stroke of its mighty hand!<br />
<br />
Let's ignore for a moment the fact that:<br />
<ul><li>Amazon had the Chef and Puppet folks in beta</li>
<li>Chef and Puppet are on the block to be supported as part of CloudFormation</li>
<li>CloudFormation is actually nothing like Chef and Puppet and serves an entirely different purpose</li>
</ul>I was pretty heads down at the office today (as was everyone else) so I didn't get a chance to catch up a bit until tonight. That's when I saw some of the most ignorant tweets from some supposedly smart people that I've ever seen. I ended up having to prune quite a bit from my Twitter list.<br />
<br />
These were obviously inspired by the CloudFormation announcement and discussions around how it relates to existing CM tools. There were gems like this:<br />
<br />
<blockquote>"process of orchestration, policy, governance, stacks, cross clouds, billback, etc. way too complex for some scripts"</blockquote><br />
<blockquote>"Scripts also wouldn't cover complexity of trying to mng a variety of clouds, all w/differing APIs & Front ends"</blockquote><br />
<blockquote>"You heard it here first. All you need for cloud automation, orchestration and provisioning is some Perl and you're golden! #DevFlOps"</blockquote><br />
Now maybe I'm taking these a bit out of context. Maybe I was just being a pissy bastard but these really got me riled up. Mind you not so riled up that I ran downstairs because <a href="http://xkcd.com/386/">"someone was wrong on the internet"</a>. I put my son to bed, fell asleep and when I woke up, I was still pissed off about it. I figured an hour of sleeping on it was enough justification so here I am.<br />
<br />
<h1>Thank You</h1>Before I get into the bitching and moaning though, I want to say "Thank you" to some people.<br />
<br />
To Mark Burgess, Luke Kanies, Adam Jacob, Damon Edwards and any other system administrator who got so fed up with the bullshit to write the tools that we're using today, THANK YOU.<br />
<br />
<i>Thank you for not accepting that we had to manage systems the way we always had. Thank you for stepping outside the comfort zone and writing amazing code. Thank you for taking 20 minutes to actually think about it when you we're only given 10 minutes to get it done. Thank you.</i><br />
<br />
To Patrick Debois, John Allspaw, Andrew Clay Shafer and everyone who has promoted the idea of what we call "devops" today, THANK YOU.<br />
<br />
<i>Thank you for pushing the idea into the mainstream with a phrase that so accurately captures what is trying to be accomplished. Thank you for being innovative and being open and sharing about it.</i><br />
<br />
To everyone else who's blog posts, newsgroup postings, tweets, emails, books, irc discussions that I've had the extreme pleasure of learning from over these past 17 years in this business, THANK YOU.<br />
<br />
<i>Thank you for sharing. Thank you for saying it even if you thought no one was reading or listening. Thank you for challenging me to learn more and inspiring me to grow as a person and as, what I'll always be at heart, a system administrator.</i><br />
<br />
To everyone above and those who I didn't mention, thank you. I thank you because it's ideas like "opensource" and "devops" and "configuration management" that free us up as individuals to think and achieve more as individuals personally and professionally. It frees us up to spend time with our families instead of answering a page at 2AM troubleshooting a stupid issue that should have never happened in the first place.<br />
<br />
These things are more valuable than gold.<br />
<br />
<h1>And to the haters...</h1><br />
<object style="height: 390px; width: 640px;"><param name="movie" value="http://www.youtube.com/v/pc0mxOXbWIU?version=3"><param name="allowFullScreen" value="true"><param name="allowScriptAccess" value="always"><embed src="http://www.youtube.com/v/pc0mxOXbWIU?version=3" type="application/x-shockwave-flash" allowfullscreen="true" allowScriptAccess="always" width="640" height="390"></object> <br />
<br />
<br />
Seriously.<br />
<br />
To the vendors who write stupid applications that require me to have X11 installed on a freaking server against ALL best practices forcing me to click through a goddamn powerpoint to install your "enterprise" software, FU.<br />
<br />
<i> I don't need your shit and I'm luckily at a point in my career where I don't have to put up with it anymore.</i><br />
<br />
To the virtualization companies who require me to have a goddamn Windows VM to manage my Linux servers because, after how many f'ing years?, you can't write a Linux port even though your product IS BASED ON LINUX? FU.<br />
<br />
<i>Don't worry. I can Xen and KVM like mofo. I can go to Amazon or GoGrid or Rackspace or any other provider if I don't need to host it in house. And guess what? I can do it all from the same platform I'm deploying without jumping through any hoops.</i><br />
<br />
To the networking vendors who give me a choice between telnet or some overpriced java gui to do configuration of your gear, FU.<br />
<br />
<i>"Oh sorry about the downtime. Because we have to drop and recreate rule sets just to add a new rule, we used copy/paste from Wordpad into HyperTerminal durdurdur".</i><br />
<br />
To the pundits who think that "devops" is just a bunch of perl scripts that can't "cover the complexity of blah blah blah"...I think you know the drill by now.<br />
<br />
<i>Really?</i> A bunch of scripts can't cover the complexity of the various cloud providers? Interesting. I guess <a href="https://github.com/geemus/fog">fog</a> or <a href="http://code.google.com/p/jclouds/">jclouds</a> or <a href="http://incubator.apache.org/libcloud/">libcloud</a> are just toys then.<br />
<br />
Oh wait, what's this? You mean I can use the same commands in my CM tool regardless of where my systems are hosted? I mean Chef's command-line tool uses Fog. Any provider Fog supports, Chef will support.<br />
<br />
But really I feel for you all. I do. You're in a losing battle. Here's the thing. People like me. People like those I mentioned above. The up and coming decision makers? We won't settle for your shitty mediocrity anymore. We won't be beholden to doing it your way. When we pick a vendor or a product or a provider, we're going to go with the ones that provide us the flexibility to manage our infrastructure in the way that's best for our company. Not for you.<br />
<br />
We've tasted what's it like to do things the "right way" and we won't take anything less.lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-85113542934629996812011-01-14T12:42:00.000-05:002011-01-14T12:42:48.251-05:00Follow up to "No Operations Team Left Behind"<i>Jim Bird over at the <a href="http://swreflections.blogspot.com/">swreflections</a> blog, recently posted an article entitled <a href="http://swreflections.blogspot.com/2011/01/what-i-like-and-dont-like-about-devops.html">"What I like (and don't like) about DevOps"</a>. I've attempted to post a comment but something about my comment is making Blogger go batshit so I'm posting it here instead along with some additional notes. Jim, for the record I don't think it's anything on the Blogger side. My comment is triggering an HTTP post too large error.</i><br />
<br />
Here's my original comment:<br />
<br />
<blockquote>As the author of one of your links, I should probably qualify a few things that weren't originally clear. I don't think that DevOps and ITIL are mutually exclusive and I don't think that anything about DevOps inherently subverts any existing policy. The point of my original post was that the enthusiasm that so many of us have can cause a negative reaction. I've often told people that you can get to the point where you can do things like continuous deployment without actually "flipping the switch". I clarified some of this in a presentation I made to the local Atlanta devops user group:</blockquote><blockquote>http://devops-culture-hurdles.heroku.com/</blockquote><blockquote>One thing that's not clear in the slides regarding "boogeymen" is that very little of the regulation from things like HIPPA and SOX impose specific technical requirements. Much of the policy is around auditability and accountability. The problem is that companies use a checklist approach to addressing those regulations because it's most cost-effective. If,for instance, the requirement is that all user access and actions are logged why is it not acceptable to simply eliminate that user access altogether and use an automated tool instead?</blockquote><blockquote><b>Auditor:</b> "<i>Show me who logged on to the server and what they did</i>"</blockquote><blockquote><b>Me</b>: "<i>I can do you one better. No one logs onto the servers. Here's an exact list of every single configuration change applied to the server and when.</i>"</blockquote><blockquote>In fact, Tools like puppet, chef, mcollective, run-deck and the like actually encourage MORE security, auditability and accountability. By approaching your infrastructure as code, using configuration management tools and automation you can eliminate most if not all of the cases where, for instance, a person needs to physically log in to a server. You get disaster recovery built in because you've now codified in "code" how to define your infrastructure and you can "compile" that infrastructure into a finished product. You attack the root cause and not just bandaid it.</blockquote><blockquote>I think companies like WealthFront (originally Kaching) are a good example of what's possible in a regulated industry. It will be interesting to see how Facebook deals with the additional regulation should they ever go public. </blockquote><br />
Sadly my original post has been used as "See? DevOps isn't for <i>REAL</i> enterprises" fodder. That was not my intention. The intention was simply this:<br />
<br />
<blockquote>Do not let the "cool" factor of DevOps cloud the practical factor of DevOps. </blockquote><br />
Yes, continuous deployment and fully automated environments are freaking awesome and they are truly laudable goals but they aren't the only reason to adopt these practices. Using configuration management is a no-brainer. Automated testing is a no-brainer. Having teams work more closely together SHOULD be a no-brainer. You can implement 100% of the capabilities that allow you to do those things and never actually do them. If you do flip that switch, don't belittle another person who can't flip that switch for whatever reason.<br />
<br />
<b>THAT</b> was the point of my original post.lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-10581216900229691962011-01-05T06:04:00.000-05:002011-01-05T06:04:07.471-05:00Chef and Encrypted Data Bags - Revisted<p>In my previous post <a href="http://lusislog.blogspot.com/2010/12/chef-and-encrypted-data-bags.html">here</a> I described the logic behind wanting to store data in an encrypted form in our <a href="http://wiki.opscode.com/display/chef/Data+Bags">Chef data bags</a>. I also described some general encryption techniques and gotchas for making that happen.</p><p>I've since done quite a bit of work in that regard and implemented this at our company. I wanted to go over a bit of detail about how to use my solution. Fair warning, this is a long post. Lot's of scrolling.</p><h1>A little recap</h1><p>As I mentioned in my previous post, the only reliable way to do the encryption of data bag items in an automated fashion is to handle key management yourself outside of Chef. I mentioned two techniques:</p><ul><li>storing the decryption key on the server in a flat file</li>
<li>calling a remote resource to grab the key</li>
</ul><p>Essentially the biggest problem of this issue is key management and, in an optimal world, how to automate it reliably. For this demonstration, I've gone with storing a flat text file on the server. As I also said in my previous post, this assumes you tightly control access to that server. We're going with the original assumption that if a malicious person gets on your box, you're screwed no matter what.</p><h1>Creating the key file</h1><p>I used the knife command to handle my key creation for now:</p><pre><code>knife ssh '*:*' interactive
echo "somedecryptionstringblahblahblah" > /tmp/.chef_decrypt.key
chmod 0640 /tmp/.chef_decrypt.key
</code></pre><h1>Setting up the databags and the rake tasks</h1><p>One of the previous things I mentioned is knowing when and what to encrypt. Be sensible and keep it simple. We don't want to throw out the baby with the bath water. The Chef platform has lots of neat search capabilities that we'd like to keep. In this vein, I've created a fairly opinionated method for storing the encrypted data bag items.</p><p>We're going to want to create a new databag called "passwords". The format of the data bag is VERY simple:</p><script src="https://gist.github.com/742575.js?file=svnpass.json"></script><br />
<p>We have an "id" that we want to use and the plaintext value that we want to encrypt.</p><h3>Rake tasks</h3><p>In my local chef-repo, I've created a 'tasks' folder. In that folder, I've added the following file:</p><script src="https://gist.github.com/742575.js?file=encrypt_databag_item.rake"></script><br />
<p>As you can see, this requires a rubygem called <a href="https://github.com/pluginaweek/encrypted_strings">encrypted_strings</a>. I've done a cursory glance over the code and I can't see anything immediately unsafe about it. It only provides an abstraction to the native OpenSSL support in Ruby with an additional String helper. However I'm not a cryptographer by any stretch so you should do your own due diligence.</p><p>At the end of your existing Rakefile, add the following:</p><pre><code>load File.join(TOPDIR, 'tasks','encrypt_databag_item.rake')
</code></pre><p>If you now run <em>rake -T</em> you should see the new task listed:</p><pre><code>rake encrypt_databag[databag_item] # Encrypt a databag item in the passwords databag
</code></pre><p>If you didn't already create a sample data bag and item, do so now:</p><pre><code>mkdir data_bags/passwords/
echo '{"id":"supersecretpassword","data":"mysupersecretpassword"}' > data_bags/passwords/supersecretpassword.json
</code></pre><p>Now we run the rake task:</p><pre><code>rake encrypt_databag[supersecretpassword]
Found item: supersecretpassword. Encrypting
Encrypted data is <some ugly string>
Uploading to Chef server
INFO: Updated data_bag_item[supersecretpassword_crypted.json]
</code></pre><p>You can test that the data was uploaded successfully:</p><pre><code>knife data bag show passwords supersecretpassword
{
"data": "<some really ugly string>",
"id": "supersecretpassword"
}
</code></pre><p>Additionally, you should have in your 'data_bags/passwords' directory a new file called 'supersecretpassword_crypted.json'. The reason for keeping both files around is for key management. Should you need to change your passphrase/key, you'll need the original file around to reencrypt with the new key. You can decided to remove the unencrypted file if you want as long as you have a way of recreating it.</p><h1>Using the encrypted data</h1><p>So now that we have a data bag item uploaded that we need to use, how do we get it on the client?<br />
That will require two cookbooks:<br />
<ul><li><a href="https://github.com/lusis/lusis-cookbooks/tree/master/databag_decrypt">databag_decrypt</a></li>
<li>A cookbook which needs the decrypted data. <a href="https://github.com/lusis/lusis-cookbooks/tree/master/test_decrypt">example</a></li>
</ul>The general idea is that, in any cookbook you need decrypted data, you essentially do three things:<br />
<ul><li>include the decryption recipe</li>
<pre><code>include_recipe "databag_decrypt::default"
</code></pre><li>assign the crypted data to a value via databag search<br />
<pre><code>password = search(:passwords, "id:supersecretpassword").first</pre></code></li>
<li>assign the decrypted data to a value for use in the rest of the recipe<br />
<pre><code>decrypted_password = item_decrypt(password[:data])</pre></code></li>
</ul><p>From there, it's no different that any other recipe. <a href="https://gist.github.com/765444">Here's an example</a> of how I use it to securely store Amazon S3 credentials as databag items:</p><pre><code>include_recipe "databag_decrypt::default"
s3_access_key = item_decrypt(search(:passwords, "id:s3_access_key").first[:data])
s3_secret_key = item_decrypt(search(:passwords, "id:s3_secret_key").first[:data])
s3_file erlang_tar_gz do
bucket "our-packages"
object_name erlang_file_name
aws_access_key_id s3_access_key
aws_secret_access_key s3_secret_key
checksum erl_checksum
end
</code></pre><h1>Changing the key</h1><p>Should you need to change the key, you'll need to jump through a few hoops:<br />
<ul><li>Update the passphrase on each client. Ease depends on your method of key distribution</li>
<li>Update the passphrase in the rake task</li>
<li>Reencypt all your data bag items.</li>
</ul>The last one can be a pain in the ass. Since Chef currently doesn't support multiple items in a data bag json file, I created a small helper script in my chef-repo called <a href="https://gist.github.com/710759">'split-em.rb'</a>.<br />
I store all of my data bag items in large json files and use split-em.rb to break them into individual files. Those file I upload with knife:</p><pre><code>bin/split-em.rb -f data_bags/passwords/passwords.json -d passwords -o
Parsing data for svnpass into file data_bags/passwords/svnpass.json
Parsing data for s3_access_key into file data_bags/passwords/s3_access_key.json
Parsing data for s3_secret_key into file data_bags/passwords/s3_secret_key.json
#Run the following command to load the split bags into the passwords in chef
for i in svnpass s3_access_key s3_secret_key; do knife data bag from file passwords $i.json; done
</code></pre><p>You could then run that through the rake task to reupload the encrypted data:</p><pre><code>for i in svnpass s3_access_key s3_secret_key; do rake encrypt_databag[$i]; done
</code></pre><h1>Limitations/Gotchas/Additional Tips</h1>Take note of the following, please.<br />
<h1>Key management</h1><p>The current method of key management is somewhat cumbersome. Ideally, the passphrase should be moved outside of the rake task. Additionally, the rekey process should be made a distinct rake task. I imagine a workflow similar to this:<br />
<ul><li>rake accepts a path to the encryption key</li>
<li>additional rake task to change the encryption key in the form of oldpassfile/newpassfile.</li>
<li>Existing data is decrypted using oldpassfile, reencrypted using new passfile and sent back to the chef server.</li>
</ul><br />
Optimally, the rake task would understand the same attributes that the decryption cookbook does so it can handle key managment on the client for you. I'd also like to make the cipher selection configurable as well an integrate it into the above steps.</p><h1>Duplicate work</h1><p>Seth Falcon at Opscode is already in the process of adding official support for encrypted data bags to Chef. His method involves converting the entire databag sans "id" to YAML and encrypting it. I wholeheartedly support that effort but that would obviously require a universal upgrade to Chef as well. The purpose of my cookbook and tasks is to work with the existing version.</p><h1>AWS IAM</h1><p>If you're an Amazon EC2 user, you should start using IAM <strong>NOW</strong>. Stop putting your master credentials in to recipes and limit your risk. I've created a 'chef' user who I give limited access to certain AWS operations. You can see the policy file <a href="https://gist.github.com/766146">here</a>. It gives the chef user read-only access to 'my_bucket' and 'my_other_bucket'.<br />
If you wanted to get REALLY sneaky, you could use fake two-factor authentication to store your key in S3:<br />
<ul><li>Encrypt data bag items with "crediential B" password except for one item "s3_credentials"</li>
<li>s3_credentials (crendential A) is encrypted with a passphrase and managed similar to this article</li>
<li>Use transient credentials to access S3 and grab a passphrase file (credential B)</li>
<li>Decrypt data with secondary credentials</li>
</ul>You would have to heavily modify the cookbook to do this. I think the current implementation is fine.</p><h1>File-based passphrases</h1><p>I'm not a big fan of the file-based passphrase method. While we agreed that you should consider yourself screwed if someone gets on the box, that still leaves poorly coded applications running as an attack vector. Imagine you have an application that must run as root. Now it can read the passphrase. Should that application become remotely exploitable, the passphrase file is vulnerable. I'm leaning to the method of a private server that allows RESTful access to grab the key. I've already added support in the cookbook for a passphrase type of 'url'.</p><h1>Wrapup</h1><p>I think that covers anything. I'd love some feedback on what people think. We've already implemented this in a limited scope for using IAM credentials in our cookbooks. I can easily revoke those should they get compromised without having to generate all new master keys.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com1tag:blogger.com,1999:blog-934985301455705990.post-59699042840204804612010-12-15T00:47:00.000-05:002010-12-15T00:47:01.026-05:00Chef and encrypted data bags.<p>As part of rolling out Chef at the new gig, we had a choice - stand up our own Chef server and maintain it or use the Opscode platform. From a cost perspective, the 50 node platform cost was pretty much break even with standing up another EC2 instance of our own. The upshot was that I didn't have to maintain it.</p><p> </p><p>However, part of due diligence was making sure everything was covered from a security perspective. We use quite a few hosted/SaaS tools but this one had the biggest possible security risk. The biggest concern is dealing with sensitive data such as database passwords and AWS credentials. The Opscode platform as a whole is secure. It makes heavy use of SSL not only for transport layer encryption but also for authentication and authorization. That wasn't a concern. What was a concern was what should happen if a copy of our CouchDB database fell into the wrong hands or a "site reliability engineer" situation happened. That's where the concept of "encrypted data bags" came from for me.</p><p> </p><p><span style="font-size: x-large;">Atlanta Chef Hack Day</span></p><p>I had the awesome opportunity to stop by the Atlanta Chef Hack day this past weekend. I couldn't stay long and came in fairly late in the afternoon. However I happened to come in right at the time that <a href="http://twitter.com/#!/botchagalupe" target="_blank">@botchagalupe (John Willis)</a> and <a href="http://twitter.com/#!/schisamo" target="_blank">@schisamo (Seth Chisamore)</a> brought up encrypted data bags. Of course, Willis proceeded to turn around and put me on the spot. After explaining the above use case, we all threw out some ideas but I think everyone came to the conclusion that it's a tough nut to crack with a shitload of gotchas.</p><p> </p><p>Before I left, I got a chance to talk with <a href="http://twitter.com/#!/sfalcon" target="_blank">@sfalcon (Seth Falcon)</a> about his ideas. While he totally understood the use cases and mentioned that other people had asked about it as well, he had a few ideas but nothing that stood out as the best way.</p><p> </p><p>So what are the options? I'm going to list a few here but I wanted to discuss a little bit about the security domain we're dealing with and what inherent holes exist.</p><p> </p><p><span style="font-size: x-large;">Reality Checks</span></p><ul><li>Nothing is totally secure.</li></ul><p> Deal with it. Even though it's a remote chance in hell, your keys and/or data are going to be decrypted somewhere at some point in time. The type of information we need to read, unfortunately, can't use a one-way encryption algo like MD5 or SHA because we NEED to know what the data actually is. I need that MySQL password to provide to my application server to talk to the database. That means it has to be decrypted and during that process and during usage of that data, it's going to exist in a possible place that it can be snagged.</p><ul><li>You don't need to encrypt <strong>everything</strong></li></ul><p> You need to understand what exactly needs to be encrypted and why. Yes, there's the "200k winter coats to troops" scenario and every bit of information you expose provides additional material for an attack vector but really think about what you need to encrypt. Application database account usernames? Probably not. The passwords for those accounts? Yes. Consider the "value" of the data you're considering encrypting.</p><ul><li>Don't forget the "human" factor</li></ul><p> So you've got this amazing library worked out, added it to your cookbooks and you're only encrypting what you need to really encrypt. Then some idiot puts the decryption key on the wiki or the master password is 5 alphabetical characters. As we often said when I was a kid, "Smooth move, exlax"</p><ul><li>There might be another way</li></ul><p style="text-align: left;"> There might be another way to approach the issue. Make sure you've looked at all the options.</p><p> </p><p><span style="font-size: x-large;">Our Use Case</span></p><p>So understanding that, we can narrow down our focus a bit. Let's use the use case of our applications database password because it's a simple enough case. It's a single string.</p><p> </p><p>Now in a perfect world, Opscode would encrypt each CouchDB database with customer specific credentials (like say an organizational level client cert) and discards the credentials once you've downloaded them.</p><p> </p><p><span style="font-size: medium;">That's our first gotcha</span> - What happens when the customer loses the key? All that data is now lost to the world. </p><p> </p><p>But let's assume you were smart and kept a backup copy of the key in a secure location. There's another gotcha inherent in the platform itself - Chef Solr. If that entire database is encrypted, unless Opscode HAS the key, they can't index the data with Solr and all those handy searches you're using in your recipes to pull in all your users is gone. Now you'll have to manage the map/reduce views yourself and deal with the performance impact where you don't have one of those views in place.</p><p> </p><p>So that option is out. The Chef server has to be able to see the data to actually work.</p><p> </p><p><span style="font-size: medium;">What about a master key?</span> That has several problems.</p><p> </p><p>You have to store the key somewhere accessible to the client (i.e. the client chef.rb or in an external file that your recipes can read to decrypt those data bag items).</p><ul><li>How do you distribute the master key to the clients?</li><li>How do you revoke the master key to the clients and how does that affect future runs? See the previous line - how do you then distribute the updated key?</li></ul><p> </p><p>I'm sure someone just said "<em>I'll put it in a data bag</em>" and then promptly smacked themselves in the head. Chicken - meet Egg. Or is it the other way around?</p><p> </p><p>You could have the Chef client ASK you for the key (remember apache SSL startups where the startup script required a password? Yeah, that sucked.</p><p> </p><p> </p><p><span style="font-size: x-large;">Going the Master Key Route</span></p><p>So let's assume that we want to go this route and use a master key. We know we can't store in with Opscode because that defeats the purpose. We need a way to distribute the master key to the clients so they can decrypt the data so how do we do it?</p><p> </p><p>If you're using Amazon, you might say "<em>I'll store it in S3 or on an EBS volume</em>". That's great! Where do you store the AWS credentials? "<em>In a data ba...oh wait. I've seen this movie before, haven't I?</em>"</p><p> </p><p>So we've come to the conclusion that we must store the master key somewhere ourselves locally available to the client. Depending on your platforming, you have a few options:</p><ul><li>Make it part of the base AMI</li><li>Make it part of your kickstart script</li><li>Make it part of your vmware image</li></ul><p> </p><p>All of those are acceptable but they don't deal with updating/revocation. Creating new AMIs is a pain in the ass and you have to update all your scripts with new AMI ids when you do that. Golden images are never golden. Do you really want to rekick a box just to update the key?</p><p> </p><p>Now we realize we have to make it dynamic. You could make it a part of a startup script in the AMI, first boot of the image or the like. Essentially, <em>"when you startup, go here and grab this key</em>". Of course now you've got to maintain a server to distribute the information and you probably want two of them just to be safe, right? Now we're spreading our key around again.</p><p> </p><p><span style="font-size: large;">This is starting to look like an antipattern.</span></p><p> </p><p>But let's just say we got ALL of that worked out. We have a simple easy way for clients to get and maintain the key. It works and your data is stored "securely" and you feel comfortable with it.</p><p> </p><p>Then your master key gets compromised. No problem, you think.<em> I'll just use my handy update mechanism to update the keys on all the clients and...shit...now I've got to re-encrypt <strong>EVERYTHING</strong> and re-upload my data bags. Where the hell is the plaintext of those passwords again?</em> This is getting complicated, no?</p><p> </p><p>So what's the answer? Is there one? Obviously, if you were that hypersensitive to the security implications you'd just run your own server anyway. You still have the human factor and backups can still be stolen but that's an issue outside of Chef as a tool. You just move the security up the stack a bit. You've got to secure the Chef server itself. But can you still use the Opscode platform? I think so. With careful deliberation and structure, you can reach a happy point that allows you to still automate your infrastructure with Chef (or some other tool) and host the data off-site.</p><p> </p><p><span style="font-size: x-large;">Some options</span></p><p><span style="font-size: large;">Certmaster</span></p><p> </p><p><a href="https://fedorahosted.org/certmaster/" target="_blank">Certmaster</a> spun out of the Func project. It's essentially an SSL certificate server at the base. It's another thing you have to manage but it can handle all the revocation and distribution issues.</p><p><span style="font-size: large;">Riak</span></p><p> </p><p>This is one idea I came up with tonight. The idea is that you run a very small <a href="http://www.basho.com/Riak.html" target="_blank">Riak</a> instance on all the nodes that require the ability to decrypt the data. Every node is a part of the same cluster and this can all be easily managed with Chef. It will probably have a single bucket containing the master key. You get the fault tolerance built in and you can pull the keys as part of your recipe using basic Chef resources. Resource utilization on the box should be VERY low for the erlang processes. You'll have a bit more network chatter as the intra-cluster gossip goes on though. Revocation is still an issue but that's VERY easily managed since it's a simple HTTP put to update. And while the data is easily accessible to anyone who can get access to the box, you should consider yourself <a href="http://www.imdb.com/title/tt0208092/quotes" target="_blank">"proper f'cked" </a>if that happens anyway.</p><p> </p><p>But you still have the issue of re-encrypting the databags should that need to happen. My best suggestion is to store the encrypted values in a single data bag and add a rake task that does the encryption/revocation for you. Then you minimize the impact of something that simply should not need to happen that often.</p><p> </p><p>Another option is to still use Riak but store the credentials themselves (as opposed to a decryption key) and pull them in when the client runs. The concern I have there is how that affects idempotence and would it cause the recipe to be run every single time just because it can't checksum properly? You probably get around this with a file on the filesystem telling Chef to skip the update using "not_if". </p><p> </p><p><span style="font-size: x-large;">Wrap Up</span></p><p> </p><p>As you can see, there's no silver bullet here. Right now I have two needs, storing credentials for S3/EBS access and storing database passwords. That's it. We don't use passwords for user accounts at all. You can't even use password authentication with SSH on our servers. If I don't have your pubkey in the users data bag, you can't log in. </p><p> </p><p>The AWS credentials are slowly becoming less of an issue. With the <a href="http://aws.amazon.com/iam/" target="_blank">Identity Access beta </a>product, I can create limited use keys that can only do certain things and grant them access to specific AWS products. I can make it a part of node creation to generate that access programatically. That means I still have the database credentials issue though. For that, I'm thinking that the startup script for an appserver, for instance, will just have to pull the credentials from Riak (or whatever central location you choose) and update a JNDI string. It spreads your configuration data out a bit but these things shouldn't need to change to often and with proper documented process you know exactly how to update it.</p><p> </p><p>One thing that this whole thing causes is that it begins to break down the ability to FULLY automate everything. I don't like running the knife command to do things. I want to be able to programatically run the same thing that Knife does from my own scripts. I suppose I could simply popen and run the knife commands but shelling out always feels like an anti-pattern to me.</p><p> </p><p>I'd love some feedback on how other people are addressing the same issues!</p><p> </p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-48139493973262287242010-12-02T03:13:00.001-05:002010-12-02T03:14:15.054-05:00Automating EBS Snapshot validation with @fog - Part 2<p><em>This is part 2 in a series of posts I'm doing - You can read part 1 <a href="http://goo.gl/sQIrm" target="_blank">here</a></em></p><p>Getting started</p><p>I'm not going to go into too much detail on how to get started with Fog. There's plenty of documentation on the github repo (protip: read the test cases) and Wesley a.k.a @geemus has done some awesome screencasts. I'm going to assume at this point that you've at least got Fog installed, have an AWS account set up and have Fog talking to it. The best way to verify is to create your .fog yaml file, start the fog command line tool and start looking at some of the collections available to you.</p><p>For the purpose of this series of posts, I've actually created a small script that you can use to spin up two ec2 instances (m1.small) running CentOS 5.5, create four (4) 5GB EBS volumes and attach them to the first instance. In addition to the fog gem, I also have awesome_print installed and use it in place of prettyprint. This is, of course, optional but you should be aware.</p><p><span style="color: #ff0000;"><em><strong>WARNING: The stuff I'm about to show you will cost you money. I tried to stick to minimal resource usage but please be aware you need to clean up after yourself. If, at any time, you feel like you can't follow along with the code or something isn't working - terminate your instances/volumes/resources using the control panel or command-line tools. PLEASE DO NOT JUST SIMPLY RUN THESE SCRIPTS WITHOUT UNDERSTANDING THEM. </strong></em></span></p><p>The setup script</p><p>The full setup script is available as gist on github - <a href="https://gist.github.com/724912#file_fog_ebs_demo_setup.rb" target="_blank">https://gist.github.com/724912#file_fog_ebs_demo_setup.rb</a></p><p>Things to note:</p><p></p><ul><li>Change the key_name to a valid key pair you have registered with EC2</li><li>There's a stopping point halfway down after the EBS volumes are created. You should actually stop there and read the comments.</li><li>You can run everything inside of an irb session if you like.</li></ul><p></p><p>The first part of the setup script does some basic work for you - it reads in your fog configuration file (~/.fog) and creates an object you can work with (AWS). As I mentioned earlier, we're creating two servers - hdb and tdb. HDB is the master server - say your production MySQL database. TDB is the box which will be running as the validation of the snapshots.</p><p><script src="https://gist.github.com/724912.js?file=setup.rb"></script>In the Fog world, there are two big concepts - models and collections. Regardless of cloud provider, there are typically at least two models available - Compute and Storage. Collections are data objects under a given model. For instance in the AWS world, you might have under the Compute model - servers, volumes, snapshots or addresses. One thing that's nice about Fog is that, once you establish your connection to your given cloud, most of your interactions are the same across cloud providers. In the example above, I've created a connection with Amazon using my credentials and have used that Compute connection to create two new servers - hdb and tdb. Notice the options I pass in when I instantiate those servers.</p><ul><li>image_id</li><li>key_name</li></ul><p>If I wanted to make these boxes bigger, I might also pass in 'flavor_id'. If you're running the above code in an irb session, you might see something like the following when you instantiate those servers: <img style="vertical-align: middle;" src="http://lh6.ggpht.com/_DB1q19qzGOg/TPdKHe3JJsI/AAAAAAAAAQ8/5u0UDnG3F3k/s800/2010-12-02--1291267770_735x483_scrot.png" alt="" width="735" height="483" /> Not all of the fields may be available depending on how long it takes Amazon to spin up the instance. The above shot is after the instance was up and running. For instance, when you first created 'tdb', you'll probably see "state" as pending for quite some time. Fog has a nice helper method for all models call 'wait_for'. In my case I could do:</p><p> <span style="font-family: 'andale mono', times;">tdb.wait_for { print "."; ready?}</span> </p><p>And it would print dots across the screen until the instance is ready for me to log in. At the end, it will tell you the amount of time you spent waiting. Very handy. You have direct access to all of the attributes above via the instance 'tdb' or 'hdb'. You can use 'tdb.dns_name' to get the dns name for use in other parts of your script for example. In my case, after the server 'hdb' is up and running, I now want to create the four 5GB EBS volumes and attach them to the instance:</p><script src="https://gist.github.com/724912.js?file=create_ebs.rb"></script> <p> </p><p>I've provided four device names (sdi through sdl) and I'm using the "volumes" collection to create them (AWS.volumes.new). As I mentioned earlier, all of the attributes for 'hdb' and 'tdb' are accessible by name. In this case, I have to create my volumes in the same availability zone as the hdb instance. Since I didn't specify where to create it when I started it, Amazon has graciously chosen 'us-east-1d' for me. As you can see, I can easily access that as '<span style="font-family: 'andale mono', times;">hdb.availability_zone</span>' and pass it to the volume creation section. I've also specified that the volume should be 5GB in size.</p><p>At the point where I've created the volume with '.new' it hasn't actually been created. I want to bind it to a server first so I simply set the volume.server attribute equal to my server object. Then I 'save' it. If I were to log into my running instance, I'd probably see something like this in the 'dmesg' output now:</p><p><span style="font-family: 'andale mono', times;"> sdj: unknown partition table</span></p><p><span style="font-family: 'andale mono', times;"> sdk: unknown partition table</span></p><p><span style="font-family: 'andale mono', times;"> sdl: unknown partition table</span></p><p><span style="font-family: 'andale mono', times;"> sdi: unknown partition table</span></p><p>As you can see from the comments in the full file, you should stop at this point and setup the volumes on your instance. In my case, I used mdadm and created a RAID0 array using those four volumes. I then formatted them, made a directory and mounted the md0 device to that directory. If you look, you should now have an additional 20GB of free space mounted on /data. Here I might make this the data directory for mysql (which is the case in our production environment). Let's just pretend you've done all that. I simulated it with a few text files and a quick 1GB dd. We'll consider that the point-in-time that we want to snapshot from. Since there's no actual constant data stream going to the volumes, I can assume for this exercise that we've just locked mysql, flushed everything and frozen the XFS filesystem. Let's make our snapshots. In this case I'm going to be using Fog to do the snapshots but in our real environment we're using the ec2-consistent-snapshot script from Aelastic. First let's take a look at the state of the hdb object:</p><p><img style="vertical-align: middle;" src="http://lh4.ggpht.com/_DB1q19qzGOg/TPdKHMxsJjI/AAAAAAAAAQ4/KbJsUT6B7C0/s800/2010-12-02--1291267751_739x591_scrot.png" alt="" width="739" height="591" /></p><p>Notice that the '<span style="font-family: 'andale mono', times;">block_device_mapping</span>' attribute now consist of an array of hashes. Each hash is a subset of the data about the volume attached to it. If you aren't seeing this, you might have to run '<span style="font-family: 'andale mono', times;">hdb.reload</span>' to refresh the state of the object. To create our snapshots, we're going to iterate over the block_device_mapping attribute and use the 'snapshots' collection to make those snapshots:</p><p><script src="https://gist.github.com/724912.js?file=snapshotting.rb"></script></p><p>One thing you'll notice is that I'm being fairly explicity here. I could shorthand and chain many of these method calls but for clarity, I'm not. </p><p>And now we have 4 snapshots available to us. The process is fairly instant but sometimes it can lag. As always, you should check the status via the .state attribute of an object to verify that it's ready for the next step. Here's a shot of our snapshots right now:</p><p><img style="vertical-align: middle;" src="http://lh5.ggpht.com/_DB1q19qzGOg/TPdKHdio4fI/AAAAAAAAARA/E7nWnpwbhY0/s640/2010-12-02--1291267799_569x866_scrot.png" alt="" width="421" height="640" /></p><p>That's the end of Part 2. In the next part, we'll have a full fledged script that does the work of making the snapshots usable on the 'tdb' instance.</p><p> </p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-25293918050135771182010-12-02T01:35:00.001-05:002010-12-02T01:35:00.032-05:00Automating EBS Snapshot validation with @fog - Part 1<p><strong><span style="font-size: x-large;">Background</span></strong></p><p>One thing that's very exciting about the new company is that I'm getting to use quite a bit of Ruby and also the fact that we're entirely hosted on Amazon Web Services. We currently leverage EBS, ELB, EC2 S3 and CloudFront for our environment. The last time I used AWS in a professional setting, they didn't even have Elastic IPs much less EBS with snapshots and all the nice stuff that makes it viable for a production environment. I did, however, manage to keep abreast of changes using my own personal AWS account.</p><p><strong><span style="font-size: large;">Fog</span></strong></p><p>Of course the combination of Ruby and AWS really means one thing - Fog. And lot's of it.</p><p>When EngineYard announced the sponsorship of the project, I dove headlong into the code base and spent what time I could trying to contribute code back. The half-assed GoGrid code in there right now? Sadly, some of it is mine. Time is hard to come by these days. Regardless, I'm no stranger to Fog and when I had to dive into the environment and start getting it documented and automated, Fog was the first tool I pulled out and when the challenge of verifying our EBS snapshots (of which we're currently at a little over 700), I had no choice but to automate it.</p><p><strong><span style="font-size: large;">Environment</span></strong></p><p>A little bit about the environment:</p><ul><li>- A total of 9 EBS volumes are snapshotted each day</li><li>- 8 of the EBS volumes are actually raid0 mysql data stores across two DB servers (so 4 disks on one/4 disks on another)</li><li>- The remaining EBS volume is a single mysql data volume</li><li>- Filesystem is XFS and backups are done using the Aleastic ec2-consistent-snapshot script (which currently doesn't support tags)</li></ul><p>The end result of this is to establish a rolling set of validated snapshots. 7 daily, 3 weekly, 2 monthly. Fun!</p><p><strong><span style="font-size: large;">Mapping It Out</span></strong></p><p>Here was the attack plan I came up with:</p><ul><li>- Identify snapshots and groupings where appropriate (raid0, remember?)</li><li>- create volumes from snapshots</li><li>- create an m1.xlarge EC2 instance to test the snapshots</li><li>- attach volume groups to the test instance</li><li>- assemble the array on the test instance</li><li>- start MySQL using the snapshotted data directory</li><li>- run some validation queries using some timestamp columns in our schema</li><li>- stop MySQL, unmount volume, stop the array</li><li>- detach and destroy the volumes from the test instance</li><li>- tag the snapshots as "verified"</li><li>- roll off any old snapshots based on retention policy</li><li>- automate all of the above!</li></ul><p>I've got lots of code samples and screenshots so I'm breaking this up into multiple posts. Hopefully part 2 will be up some time tomorrow</p><p> </p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-59981295584659884532010-11-09T01:12:00.002-05:002010-11-09T01:12:26.112-05:00Fix it or Kick It and the ten minute maxim<p>One of the things I brought up in my <a href="http://devops-culture-hurdles.heroku.com/#1" target="_blank">presentation</a> to the Atlanta DevOps group was the concept of "Payment". One of the arguments that people like to trot out when you suggest an operational shift is that "We can't afford to change right now". My argument is that you CAN'T afford to change. It's going to cost you more in the long run. The problem is that in many situations, the cost is detached from the original event.</p><p>Take testing. Let's assume you don't make unit testing an enforced part of your development cycle. There are tons of reasons people do this but much of it revolves around time. We don't have time to write tests. We don't have time to wait for tests to run. We've heard them all. Sure you get lucky. Maybe things go out the door with no discernible bugs. But what happens 3 weeks down the road when the same bug that you solved 6 weeks ago crops up again? It's hard to measure the cost when it's so far removed from the origination.</p><p>Configuration management is the same way. I'm not going to lie. Configuration management is a pain in the ass especially if you didn't make it a core concept from inception. You have to think about your infrastructure a bit. You'll have to duplicate work initially (i.e. templating config files). It's not easy but it pays off in the long run. However as with so many things, the cost is detached from the original purchase. </p><p><span class="Apple-style-span" style="font-size: large;">Fix it?</span></p><p>Walk with me into my imagination. A scary place where a server has started to misbehave. What's your initial thought? What's the first thing you do? You've seen this movie and done this interview:</p><p></p><ul><li>log on to the box</li><li>perform troubleshooting</li><li>think</li><li>perform troubleshooting</li><li>call vendor support (if it's an option)</li><li>update trouble ticket system</li><li>wait</li><li>troubleshoot</li><li>run vendor diag tools</li></ul><p></p><p>What's the cost of all that work? What's the cost of that downtime? Let's be generous. Let's assume this is a physical server and you paid for 24x7x4 hardware support and a big old RHEL subscription. How much time would you spend on each task? What's the turn around time to getting that server back into production?</p><p>Let's say that the problem was resolved WITHOUT needing replacement hardware but came in at the four hour mark. That's three hours that the server was costing you money instead of making you money. Assuming a standard SA salary of $75k/year in Georgia, that works out to $150. That's just doing a base salary conversion not calculating all the other overhead associated with staffing an employee. What if that person consulted with someone else during that time, a coworker at the same rate, for two of those hours. $225. Not too bad, right? Still a tangible cost. Maybe one you're willing to eat.</p><p>But let's assume the end result was to wipe and reinstall. Let's say it takes another hour to get back to operational status. Woops. Forgot to make that tweek to Apache that we made a few weeks ago. Let's spend an hour troubleshooting that.</p><p>But we're just talking man power at this point. This doesn't even take into account end-user productivity, loss of customers from degraded performance or any host of issues. God forbid that someone misses something that causes problems to other parts of the environment (like not setting the clock and inserting invalid timestamps into the database or something. Forget that you shouldn't let your app server handle timestamps). Now there's cleanup. All told your people spent 5 hours to get this server back into production while you've been running in a degraded state. What does that mean when our LOB is financial services and we have an SLA and attached penalties? I'm going to go easy on you and let you off with 10k per hour of degraded performance.</p><p>Get ready to credit someone $50k or worse cut a physical check.</p><p><span class="Apple-style-span" style="font-size: large;">Kick it!</span></p><p>Now I'm sure everyone is thinking about things like having enough capacity to maintain your SLA even with the loss of one or two nodes but be honest. How many companies actually let you do that? Companies will cut corners. They roll the dice or worse have a misunderstanding of HA versus capacity planning.</p><p>What you should have done from the start was kick the box. By kicking the box, I mean performing the equivalent of a kickstart or jumpstart. You should, at ANY time, be able to reinstall a box with no user interaction (other than the action of kicking it) and return it to service in 10 minutes. I'll give you 15 minutes for good measure and bad cabling. My RHEL/CentOS kickstarts are done in 6 minutes on my home network and most of that time is the physical hardware power cycling. With virtualization you don't even have a discernible bootup time.</p><p><span class="Apple-style-span" style="font-size: large;">Unit testing for servers</span></p><p>I'll go even farther. You should be wiping at least one of your core components every two weeks. Yes. Wiping. It should be a part of your deploy process in fact. You should be absolutely sure that should you ever need to reinstall under duress that you can get that server back into service in an acceptable amount of time. Screw the yearly DR tests. I'm giving you a world where you can perform bi-monthly DR tests as a matter of standard operation. All it takes is a little bit of up front planning.</p><p><span class="Apple-style-span" style="font-size: large;">The 10 minute maxim</span></p><p>I have a general rule. Anything that has to be done in ten minutes can be afforded twenty minutes to think it through. Obviously, it's a general rule. The guy holding the gun might not give you twenty minutes. And twenty minutes isn't a hard number. The point is that nothing is generally so critical that it has to be SOLVED that instant. You can spend a little more time up front to do things right or you can spend a boatload of time on the backside trying to fix it.</p><p>Given the above scenario, you would think I'm being hypocritical or throwing out my own rule. I'm not. The above scenario should have never happened. This is a solved problem. You should have spent 20 minutes actually putting the config file you just changed into puppet instead of making undocumented ad-hoc changes. You should have spent an hour when bringing up the environment to stand up a CM tool instead of just installing the servers and doing everything manually. That's the 10 minute maxim. Take a little extra time now or take a lot of time later.</p><p>You decide how much you're willing to spend.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-89411550582486668162010-11-08T15:12:00.002-05:002010-11-08T15:12:11.913-05:00Transitions<p>I haven't had a chance to mention this but those of you who I'm connected with on LinkedIn are aware that I'm starting with a new company on Wednesday. I'm taking a few days to get some house work done and then diving in. I don't like switching companies in general but I'm really excited about this opportunity. In addition to having almost a blank slate, I'm working with a much smaller team and a chance to contribute back to the community. It's also a chance for me to work in the Atlanta startup scene; something I've been hoping to do for a few years now.</p><p>So what about the previous company? Well they're looking to back fill my position. Please feel free to contact me if you're interested. I can put you in touch with the right people. Fair warning, it's a challenging place to work. They'll tell you the same thing. I've blogged about working at a "traditional" company before right here so you can go back and glean information from that.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-86444669446357944572010-11-02T23:56:00.000-04:002010-11-02T23:56:04.553-04:00Using Hudson and RVM for Ruby unit testing<p>As with everything lately, something popped up on Twitter that prompted a blog post. In this case, @wakaleo was looking for any stories/examples for his Hudson book. I casually mentioned I could throw in some notes about how we use Hudson on the Padrino project.</p><p><span style="font-size: large;">Prerequisites</span></p><p>Here's what you'll need:</p><ul><li><a href="http://hudson-ci.org/" target="_blank">Hudson</a></li><li><a href="http://rvm.beginrescueend.com/" target="_blank">RVM</a></li><li><a href="http://wiki.hudson-ci.org/display/HUDSON/Ruby+metrics+plugin" target="_blank">Hudson Ruby Metrics plugin</a> (if you want code coverage reports)</li></ul><p>I'll leave you to get Hudson working. There are prebuilt packages for every distro under the sun. If you can't get past this step, you'll need to rethink a few things.</p><p><strong><span style="font-size: large;">Setting up RVM</span></strong></p><p>Once you have it installed, log in as your Hudson user and set up RVM.</p><p><em><strong>RVM Protip </strong>- If there are any gems (like say Bundler) that you ALWAYS install, edit </em><span style="font-family: 'andale mono', times;">.rvm/gemsets/default.gems</span><em> and </em><span style="font-family: 'andale mono', times;">.rvm/gemsets/global.gems</span><em> and add them there. In my examples, I did not do that.</em></p><p>You'll want to go ahead and install all the VMs you plan on testing against. We use 1.8.7, 1.9.1, 1.9.2, JRuby, RBX and REE:</p><p><span style="font-family: 'andale mono', times; font-size: x-small;">for i in 1.8.7 1.9.1 1.9.2 jruby ree rbx; do rvm install ${i}; done</span></p><p>This will take a while. When it's done, we can now dive into configuring our job in Hudson</p><p><strong><span style="font-size: large;">What is the Matrix?</span></strong></p><p>So you've got Hudson running and RVM all set up? Open the Hudson console and create a new job of type "<strong>Build multi-configuration project</strong>". From the job configuration screen, you'll want to set some basics - repository, scm polling and the like. The key to RVM comes under "<strong>Configuration Matrix</strong>"</p><p> </p><p>The way any user-defined variables work in Hudson, whether a build parameter or matrix configuration, is that you provide a "key" and then a value for that key. The value for that key is accessible to your build steps as a sigil variable. So if your key is <span style="font-family: 'andale mono', times;">my_funky_keyname_here</span>, you can reference <span style="font-family: 'andale mono', times;">$my_funky_keyname_here</span> in your build steps to get that value. With a configuration matrix, each permutation of the matrix provides the value for that key in the given permutation. So if I have:</p><p style="text-align: center;">foo as one axis with 6 values (1, 2, 3 ,4 ,5 ,6) and bar with 3 values (1, 2, 3)</p><p>each combination of foo and bar will be available to my build steps as $foo and $bar. The first run will have $foo as 1 and $bar as 1. Second run will have $foo as 2 and $bar as 1. On an on until the combinations are exhausted.</p><p>This makes for some REALLY powerful testing matrices. In our case, however, we only need one axis - <span style="font-family: 'andale mono', times;">rubyvm</span></p><p><em><strong>Hudson Protip -</strong> Don't get creative with your axis or parameter names. In our case, we'll be performing shell script steps. Don't call your axis "HOME" because that will just confuse things. Just don't do it.</em></p><p>So now we've added an axis called 'rubyvm' and provided it with values '1.8.7 1.9.1 1.9.2 jruby rbx ree'. As explained, this means that our build steps will iterate over each value of 'rubyvm' for us and repeat our build steps.</p><p><strong><span style="font-size: large;">Configuring your job</span></strong></p><p>Now that you've got your variables in place, you can write the steps for your job. This took me a little bit of time to work out the best flow. There were some things with how RVM operates with the shell that caught me off-guard initially (the rvm command being a function alias versus an executable). I've broken the test job into three steps:</p><ul><li>Create my gemset, install bundler and run bundle install/bundle check</li><li>Run my unit tests</li><li>Destroy my gemset</li></ul><p>In addition to taking advantage of the variable provided by the configuration matrix, we're also going to take advantage of some variables exposed by Hudson in a given job run - $BUILD_NUMBER. Using these two bits of information, we can build a gemset name for RVM that is unique to that run and that ruby vm.</p><p>Step 1:</p><p><p><span style="font-family: 'andale mono', times;">#!/bin/bash -l</span></p><p><span style="font-family: 'andale mono', times;">rvm use $rubyvm@padrino-$rubyvm-$BUILD_NUMBER --create</span></p><p><span style="font-family: 'andale mono', times;">gem install bundler</span></p><p><span style="font-family: 'andale mono', times;">bundle install</span></p><p><span style="font-family: 'andale mono', times;">bundle check</span></p><p>This uses the --create option of RVM to create our gemset. If our build number is 99 and our ruby vm is ree, we're creating a gemset called padrino-ree-97 for ree. Pretty straightforward.</p><p>Next we install bundler and then run the basic bundler tasks. All operations are performed in the workspace for your hudson project. This is typically the root directory of your SCM repository. If the root of your repo doesn't contain your Gemfile and Rakefile, you'll probably want to make your first step a 'cd' to that directory.</p><p>The reason for using a full shebang line is to make sure that RVM instantiates properly.</p><p>Step 2:</p><p><p><span style="font-family: 'andale mono', times;">#!/bin/bash -l</span></p><p><span style="font-family: 'andale mono', times;">rvm use $rubyvm@padrino-$rubyvm-$BUILD_NUMBER</span></p><p><span style="font-family: 'andale mono', times;">rake test</span></p><p>Each build step is a distinct shell session. For that reason we need to "use" the previously created gemset. Then we run our rake tasks.</p><p>Step 3:</p><p><p><span style="font-family: 'andale mono', times;">#!/bin/bash -l</span></p><p><span style="font-family: 'andale mono', times;">rvm use $rubyvm@global</span></p><p><span style="font-family: 'andale mono', times;">rvm --force gemset delete padrino-$rubyvm-$BUILD_NUMBER</span></p><p>This is the "cleanup" step. This cleans up our temporary gemsets that we created for the test run. My understanding was the each step was "independent". Should the middle step fail, the final step would still be executed. This doesn't appear to be the case anymore. For this reason, you'll probably want to occasionally go in and clean up gemsets from failed builds. If your build passes, the gemset will clean itself up. There's probably justification for some sort of "cleanup" job here but I haven't gotten around to trying to pass variables as artifacts to other build steps.</p><p>Now you can run the job and watch as Hudson gleefully executes your test cases against each ruby vm. How many of those run concurrently is dependent on how many workers you have configured globally in Hudson.</p><p><em><strong>Unit Testing Protip</strong> - One thing you'll find out early on is how concurrent your unit tests REALLY are. In the case of Padrino, ALL of our unit tests were using a hardcoded path (/tmp/sample_project) for testing. My first major step once I got added to the project was to refactor ALL of our tests to make that dynamic so that we could run more than one permutation at a time. You can see an example of how I did that <a href="https://github.com/padrino/padrino-framework/blob/master/padrino-gen/test/test_project_generator.rb" target="_blank">here</a>. Essentially I created an instance variable for our temp directory using UUID.new.generate. It was the quickest way to resolve the problem. If your tests aren't capable of running in parallel, that's one way to address it.</em></p><p>One thing to be aware of: if you have intensive unit tests and your hudson server isn't very powerful, you simply may not have the capacity to run multiple tests at the same time. I had to spin up some worker VMs on other machines around the house to serve as Hudson slave nodes. Our unit tests were actually taking LONGER when we tried to run them in parallel because of the strain of compiling native extension gems and actually running the tests.</p><p><strong><span style="font-size: large;">Optional profit! step</span></strong></p><p>Code coverage is important. However it makes NO sense to run code coverage tasks on EVERY VM permutation. You only need to run it once (unless you have some VM dependent code in your application). What I've done is take advantage of "Post build actions" to kick off a second job I've defined. This job does nothing but runs our code coverage rake tasks. Steps 1 and 3 are the same as above without the rubyvm variable. Step 2 is different:</p><p><p><span style="font-family: 'andale mono', times;">#!/bin/bash -l</span></p><p><span style="font-family: 'andale mono', times;">rvm use 1.8.7@padrino-rcov-$rubyvm-$BUILD_NUMBER</span></p><p><span style="font-family: 'andale mono', times;">bundle exec rake hudson:coverage:clean</span></p><p><span style="font-family: 'andale mono', times;">bundle exec rake hudson:coverage:unit</span></p><p>We've broken the coverage tests into a unique rake task so they don't impact normal testing. This creates a code coverage report that's visible in Hudson under that project's page. Currently we don't run the coverage report job unless the primary job finishes.</p><p><strong><span style="font-size: large;">Wrap up</span></strong></p><p>That's pretty much it in a nutshell. I'm looking to move Hudson to a more powerful VM here at the house as soon as the hardware comes in. I should be able to then run all the tests across all VMs at one time. Screenshots for each of the steps described in this post are available <a href="http://picasaweb.google.com/lusisjv/RVMAndHudsonRubyBuildMatrix?feat=directlink" target="_blank">here</a></p></p></p></p></p><p> </p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-18790873233230703132010-10-28T05:25:00.000-04:002010-10-28T05:25:35.369-04:00Designed for Developers - Why people keep asking you to use Github<p>I'll be the first to admit that I'm a Github fanboy. The shocker is that my love of Github has nothing to do with the DVCS underneath. While Git plays a major part of what makes github so great, the bigger reason github is so successful is this:</p><p><span style="font-size: large;">Github is designed for developers</span></p><p>What do I mean by that? Let's compare a series of screenshots from various code hosting sites:</p><p> </p><table style="width: 194px;"><tbody><tr><td style="height: 194px; background: url(http://picasaweb.google.com/s/c/transparent_album_background.gif) no-repeat left;" align="center"><a href="http://picasaweb.google.com/lusisjv/CodeHostingSolutionsComparison?feat=embedwebsite"><img style="margin: 1px 0 0 4px;" src="http://lh5.ggpht.com/_DB1q19qzGOg/TMksZgT2EZE/AAAAAAAAALY/XggULu12it4/s160-c/CodeHostingSolutionsComparison.jpg" alt="" width="160" height="160" /></a></td></tr><tr><td style="text-align: center; font-family: arial,sans-serif; font-size: 11px;"><a style="color: #4d4d4d; font-weight: bold; text-decoration: none;" href="http://picasaweb.google.com/lusisjv/CodeHostingSolutionsComparison?feat=embedwebsite">Code Hosting Solutions comparison</a></td></tr></tbody></table><p>I want you to take a look at the screenshots very carefully especially the "project" pages. What's the one thing you notice about Github compared to the others (excluding BitBucket). What's the focus of the project?</p><p><span style="font-size: large;">It's all about the code</span></p><p>You'll see quite clearly that with all the sites except for BitBucket, the focus of the project is the code itself. Not only is the focus of the project the code but everything about the code is about the community. I can "watch" a developer or project. I can easily see from the first page how to download the codebase. However the biggest part of what makes Github a success is one button:</p><p>Fork</p><p>From the start of a project page, not only can I easily browse the code and am provided with the information I need to checkout the code but I'm invited with a single button to become a contributor to that project. Immediately, I'm a potential contributor to that project. If I change something and push the code back to my fork, I can push one button and send a message to the project maintainers asking them to merge the changes back in. As a project maintainer, I have an easy way to evaluate the impact of the change and communicate with the requester and other team members about said change. At the bottom of the pull request page, I'm provided the information on how to easily merge those changes into my main tree.</p><p><span style="font-size: large;">Designed for Developers</span></p><p>I've been on a bit of a tear lately about usability in developer-targeted products. The latest target of my ire has been Atlassian. Let me clarify that I think Atlassian makes some wonderful products. Confluence is one of the best wikis out there. JIRA is a great issue tracking system for Developers.</p><p>However, Atlassian has some "duds" in my opinion. The biggest thorn in my side these days is Bamboo. Bamboo is Atlassian's Continuous Integration server. Like most Atlassian products, its primary target is Java developers. Everything about Bamboo is designed around the Java development toolchain - Maven, Ant and the like. But I don't have a problem with that. What I have a problem with is the over-complication. I grabbed the latest beta of Bamboo at the recommendation of one of the Bamboo developers who heard my rant on Twitter one day. He asked for some feed back and I provided it in a very detailed email. I'm happy to say that the new interface for adding build plans in Bamboo is much simpler than previous versions. I can't do screenshots of our company Bamboo install but previous versions had a VERY complicated multitab build plan configuration.</p><p>One point I mentioned in my email is that Bamboo felt like it lacked a focus. Jira was very clearly about Issues. That was the "unit of work". Confluence was very clearly about being a wiki. That was its "unit of work". Bamboo didn't have a singular focus. It was a CI server but what was the unit of work? A build plan? Test results? Fisheye integration? It wasn't clear.</p><p>Compare that with Hudson which had a very clear focus. The strength in Hudson is that it performs tasks. Those tasks are typically centered around CI but they don't have to be. In Hudson I can define a job that does nothing more than list directories. I don't even need to back it with a VCS. Bamboo, sadly, in the beta version still hasn't gotten this part right. I can't define a build plan without having a repository somewhere. It still assumes that I want to define all my work inside of an ant script. Using the "shell" builder is still VERY limiting. </p><p>You can see some sample comparison shots between the two <a href="http://picasaweb.google.com/lusisjv/BambooVsHudson?feat=directlink" target="_blank">here</a>. I'll try to actually setup a repo that Bamboo can use and do a deeper comparison later. </p><p><span style="font-size: large;">So what's the focus of Google Code, Launchpad...</span></p><p>Going back to code hosting and comparing Github to the others, I think it's clear that they lack a focus. They try to do too much. They "feel" like they were designed by project managers and targeted at them. Maybe it was a faulty assumption that to effectively manage a large project, you had have all of the extra stuff. I don't know. Launchpad and others DO some things better than Github. Issue tracking is one. Github issue tracking is a pretty weak area for them. However here's where Github understands its focus and strengths.</p><p>Where Github lacks, it makes up for in integration. Github doesn't TRY to be the project manager's tool. It doesn't try to be a good issue tracker. What it DOES do is say "I suck at this. My focus is on the code and making working with and contributing to the code dead simple. I'll add hooks for the other stuff"</p><p>And they do. Github has a boatload of service hooks for everything from issue tracking to project management to irc and IM. They even have a "generic" hook that will submit JSON to a url for you so you can write your own receiver.</p><p><span style="font-size: large;">About BitBucket, backend technology and focus</span></p><p>I haven't mentioned much about BitBucket. The main reason is that at this point, BitBucket is simple attempting to feature copy from Github except using Mercurial in the background. Sadly, this isn't enough I think. If my only reason for using BitBucket is the DVCS tool then I honestly might as well use Github. I'll get more engagement there. See this quote from Mark Philips from Basho about why the moved from BitBucket to Github:</p><blockquote><p style="text-align: left;">Why? There are several reasons, the primary of which is that GitHub,</p><p style="text-align: left;">the application, lends itself to more collaboration when developing</p><p style="text-align: left;">open source software. Again, this was a decision made on the basis of</p><p style="text-align: left;">community development; technically-speaking we were satisfied with</p><p style="text-align: left;">what Bitbucket offered.</p></blockquote><p>The issue wasn't the technology. Mercurial and Git are pretty much at feature parity (as is Bazaar). One thing mercurial doesn't do out of the box is cherry picking but it's supported with extra configuration. Mercurial has <em><span style="font-family: 'courier new', courier;">hg incoming</span></em> which let's you see what people are working on. Git has staging. Mercurial has better Windows support than Git. It's really six in one, half dozen in the other.</p><p>However what BitBucket DOESN'T have is the community. You see, BitBucket was playing catchup to Github. Simply copying the social aspects of Github isn't enough. Github has too much momentum precisely because they had the focus right from the start - code is king.</p><p>As a developer, my key focus is my code. It's what says the most about me. As a developer who wants to attract other developers, the best way to do that is showing the code and making that contribution as easy as possible. Github gets that.</p><p>That's why people keep asking you to switch to Github.</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com5tag:blogger.com,1999:blog-934985301455705990.post-53256513453212660502010-10-22T22:03:00.000-04:002010-10-22T22:03:36.010-04:00Potato Candy - A family recipe<p>With Halloween right around the corner and Thanksgiving beyond that, it's getting about the time of year when I get to make Potato Candy. Yes, candy made from potatoes.</p><p>I don't know the real story behind it. Ever since I was a little pile of baby fat, it's something the kids in my family have eaten. My uncle only made it for Thanksgiving and I think Christmas get-togethers. I've tried to find a bit of history about it over the years but nothing ever concrete. My uncle's family is Irish so that's as stereotypical of a reason as any. What I did seem to track down is that it's pretty unique to the Southeast. We do things weird here, ya'll.</p><p>Not long after I married my wife (a Michigan native), her aunt was putting together a family cookbook. Now that I was part of the family I got to contribute a few things. I had my mom and step-mom provide a few entries but I reserved one for myself - Potato Candy. Since the "secret" is out and because freaking @jtimberman got me thinking about candy, I figured I'd add it here for all my interweb friends.</p><p><span style="font-size: medium;">Ingredients</span></p><ul><li>1 Potato about the size of your fist. Seriously. Don't get it any bigger. If you've got big hands, find someone with normal sized hands and compare.</li><li>1 jar of peanut butter. Creamy not Crunchy. The last thing you want to deal with when making this stuff is nuts. Trust me.</li><li>2 bags of powdered sugar. Yes, you will probably use ALL of it.</li><li>Wax paper and plenty of counter top space</li></ul><p>Peel and boil the potato as you would to make mashed potatoes. When it gets sufficiently soft, mash that bastard up. No lumps. Again, trust me. As smooth as you can get it.</p><p>Dump it into a large mixing bowl and reach for the strongest and sturdiest spoon/stirring instrument you can find. Start folding in the first bag of powdered sugar.</p><p>This is where it gets fun. As the powdered sugar gets mixed in, this thing is going to get thick and heavy very quickly. It's going to be VERY hard to mix. Did you trust me on the sturdy spoon part? You should have. Don't even think about putting this in an electric mixer. It will burn out the motor. I've literally broken 1/4 inch dowel wooden spoons in this stuff. Your arm is going to hurt. You're going to have to put your back into it.</p><p>When you physically can't mix it ANYMORE put it aside for a minute. Spread out a nice sized area on the counter with wax paper and cover it in powdered sugar. This crap is sticky and you're going to need to manipulate it. Once you've gotten the workspace ready, start spreading the "mash" on the wax paper. Usually about 1/4 to 1/2 inch thick is good. You'll probably screw it up the first time around. I did.</p><p>Open the jar of peanut butter and start spreading it on the mash. Peaks are okay but you really want to get a good layer on there.</p><p><span style="font-size: medium;">Now, the hard part</span></p><p>Somehow you're going to need to roll from one end of this beast to the other. Like a jelly roll. It's really hard and don't feel too bad if it isn't pretty. The end result is still good. You'll probably want to cover your hands in powdered sugar.</p><p>Once you've got it rolled up, flatten it back out. Stick it in the fridge overnight. The next day, cut it into smallish 1.5inch slices and enjoy.</p><p>As I said earlier, I've tried to do some research each year. The best picture I can find outside of making some myself is <a href="http://www.flickr.com/photos/38843845@N00/1244127037/">this one</a>.</p><p>You probably won't be able to eat more than one or two pieces. It's REALLY rich and really thick. If you give it to kids, do it early in the day so they have time to burn it off.</p><p>Enjoy ya'll!</p>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com0tag:blogger.com,1999:blog-934985301455705990.post-22635867104299051172010-10-21T23:17:00.004-04:002010-10-21T23:49:05.740-04:00PyCon DevOps piggy backSo I had a random idea the other night and like any other random idea I immediately sent it to Twitter.<br /><br />This of course brought feedback which is the whole point, right?<br /><br />The idea was to have a Velocity style conference in the South East. We all know my love for <a href="http://sweetteamanifesto.com/">Atlanta</a> and my half-disdain/half-jealousy of the West coast. So I threw the idea out on twitter and immediately got my first reply from <a href="http://twitter.com/#!/heckj">Joe Heck</a> with a bit of reality thrown in:<div><br /></div><div><blockquote>@lusis nice idea. critical mass with either be easy or impossible to get. You might consider riffing on existing conferences ... PyCon2011</blockquote></div><div><br /></div><div>Awesome idea so I headed off to to read up on how PyCon does that kind of thing. I shot off an email to the pycon-organizers mailing list and got some really nice responses. I also got a private tweets from people on the list as well.</div><div><br /></div><div>The end result is this. If I want to hitchhike on the back of PyCon for a devops-related conference, here are the requirements/suggestions:</div><div><br /></div><div><ul><li>Involve Python in some way</li><li>Will need to take advantage of the Open Spaces system</li></ul></div><div><br /></div><div>This essentially means unless I (or someone else) is giving a full blown talk on Python and DevOps, it will be an ad-hoc thing. We can't reserve the spaces until the day of the conference. I'm also not sure how big the spaces are. I think this is the same place LISA was held years ago so you might be able to snag a dividable room segment?</div><div><br /></div><div>So what does everyone think? I'm considering giving a talk on the state of devops toolchains in Python (func, cobbler, fabric, kokki, overmind, whatever else) but I don't know that I'm ready for that yet after a single LUG presentation ;)</div><div><br /></div><div>I know that Mitchell H. of Vagrant fame was considering heading into town for it. Vagrant isn't just for Rubyists ;)</div><div><br /></div><div>I'm open to ideas. I'd love to just have the conference I sent the tweet about but when I really think about it, I don't think I can pull something like that off in this amount of time.</div><div><br /></div><div>Many thanks to the pycon-organizers folks for the input - Doug Hellmann, Vern Ceder and Jesse Noller. Also to <a href="http://twitter.com/#!/ponderings">Dean Goodmanson</a> for his feedback via Twitter.</div>lusishttp://www.blogger.com/profile/15354716270000450238noreply@blogger.com1