Want Some Campfire With That Bluepill

This is a seriously old post. I’d probably not bother reading it, but I keep it here for nostalgia.

Custom Triggers With the Bluepill Monitoring Gem

We use Chef for automating our server infrastructure, but I’ll show you the manual way of installing Bluepill (either locally or remotely) so you can play with it.

1	gem install bluepill

That wasn’t hard, was it? Dependencies will be resolved, and installed too. Now we’ve got it installed, let’s take a little look at the commands we can give it.

Check the status of all applications and processes

1	sudo bluepill status

Start all applications and stop all applications, or a single group.

1 2 3

sudo bluepill start sudo bluepill stop sudo bluepill start resque

Quit Bluepill.

1	$ sudo bluepill quit

You configure Bluepill with a configuration file. Here’s one we use to start multiple resque queues. It’s a work in progress but does what we want for now. I won’t explain the config file because there’s plenty of blogs (and of course the Bluepill README) that go into detail here.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

queues = {} queues["comms"] = 1 queues["rules"] = 2 queues["patients"] = 4 queues["import"] = 1 @app_name = "pharmmd" @owner_name = "rails" @rails_env = "production" @rake_command = "bundle exec rake " Bluepill.application("#{@app_name}_resque", :log_file => "/var/log/bluepill.log") do |app| app.uid = app.gid = @owner_name queues.each do |queue, count| count.times.each do |idx| app.process("resque_worker_#{queue}_#{idx}") do |process| process.group = 'resque' process.pid_file = "/var/run/resque/#{@app_name}/worker_#{queue}_#{idx}.pid" process.working_dir = "/data/#{@app_name}/current" process.start_command = @rake_command + "RACK_ENV=#{@rails_env} RAILS_ENV=#{@rails_env} " + "QUEUE=#{queue} " + "resque:work" process.stop_command = <<-EOF kill -QUIT sleep_count=0 while [-e /proc/]; do sleep 1 let "sleep_count+=1" if [$sleep_count -eq 60]; then kill -TERM fi if [$sleep_count -ge 70]; then kill -KILL fi done EOF process.start_grace_time = 5.seconds process.stop_grace_time = 75.seconds process.restart_grace_time = 80.seconds process.daemonize = true process.checks :mem_usage, :below => 150.megabytes, :every => 1.minute, :times => 3 end end end end

What I wanted to add was a notification system that let me know when things had gone horribly wrong. Specifically I wanted Campfire notifications, as that’s where all the cool techies at PharmMD hang out.

For notifications, Bluepill allows us to write simple ‘Triggers’. These triggers can be added to a process in the config file, like zoh:

1	process.checks :some_trigger, :every => 1.minute

So let’s add a Campfire notifier. First things first, I’ll be using the tinder gem to talk to Campfire, have a look at their README for details on how it works (or the excellent tinder homepage). Install tinder.

1	gem install tinder

At the top of the Bluepill config file (anywhere above the application block) add the following:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

require 'rubygems' require 'uri' require 'tinder' class Campfire < Bluepill::Trigger PARAMS = [:times, :within, :retry_in, :rails_env] attr_accessor *PARAMS def initialize(process, options = {}) options.reverse_merge!(:times => 5, :within => 1, :retry_in => 5) options.each_pair do |name, val| instance_variable_set("@#{name}", val) if PARAMS.include?(name) end super end def notify(transition) @logger.info "I so just got triggered with a transition!" begin if transition.to == "down" && transition.from == "up" @logger.info "BluePill: #{@rails_env} worker(#{process.name}) went from #{transition.from_name} to #{transition.to_name}. Restarting them." campfire = Tinder::Campfire.new 'pharmmd', :token => 'TOKEN' room = campfire.find_room_by_name('General') room.speak "BluePill: #{@rails_env} worker(#{process.name}) died. Restarting it." end rescue => x @logger.info "Totally failed with: #{x.message}" end end end

In the initialize method I set up some parameters that I can pass in, as well as provide myself with some defaults to fall back on.

1 2	PARAMS = [:times, :within, :retry_in, :rails_env] attr_accessor *PARAMS

I create instance variables for each option for access later on, and call super to allow the Bluepill::Trigger class to do its own initialisation.

1 2 3

options.each_pair do |name, val| instance_variable_set("@#{name}", val) if PARAMS.include?(name) end

Next up I create one of the required methods that Bluepill will call periodically and sprinkle in a little logging (@logger is provided by Bluepill::Trigger).

1	@logger.info "I so just got triggered with a transition!"

Next up we wrap our Campfire notification in a rescue block, because we don’t want ay silly exceptions like a lack of net access from bombing Bluepill out. There are a few transitions we can use, the important one for me is when a process dies (even though Bluepill will restart it), I want to know when that happens so I can find out why and stop it.

1	if transition.to == "down" && transition.from == "up"

Tinder makes campfire access so simple. Connect to campfire, select a room, and then ‘speak’ into it.

1 2 3

campfire = Tinder::Campfire.new 'pharmmd', :token => 'TOKEN' room = campfire.find_room_by_name('General') room.speak "BluePill: #{@rails_env} worker(#{process.name}) died. Restarting it."

That’s it for the trigger. Now all we need to do is tell the process that it should fire that trigger. Add the following to the process block just beneath where we checked for memory usage.

1	process.checks :campfire, :rails_env => @rails_env

That’s it. Now when you load the config file in, Campfire will get notifications if the process needs restarting.

Simple Campfire notifications using Tinder and Bluepill? Done.