Want Some Campfire With That Bluepill

4 minute read

This is a seriously old post. I’d probably not bother reading it, but I keep it here for nostalgia.

Custom Triggers With the Bluepill Monitoring Gem

We use Chef for automating our server infrastructure, but I’ll show you the manual way of installing Bluepill (either locally or remotely) so you can play with it.

1
gem install bluepill

That wasn’t hard, was it? Dependencies will be resolved, and installed too. Now we’ve got it installed, let’s take a little look at the commands we can give it.

Check the status of all applications and processes

1
sudo bluepill status

Start all applications and stop all applications, or a single group.

1
2
3
sudo bluepill start
sudo bluepill stop
sudo bluepill start resque

Quit Bluepill.

1
$ sudo bluepill quit

You configure Bluepill with a configuration file. Here’s one we use to start multiple resque queues. It’s a work in progress but does what we want for now. I won’t explain the config file because there’s plenty of blogs (and of course the Bluepill README) that go into detail here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
queues = {}
queues["comms"] = 1
queues["rules"] = 2
queues["patients"] = 4
queues["import"] = 1

@app_name = "pharmmd"
@owner_name = "rails"
@rails_env = "production"
@rake_command = "bundle exec rake "

Bluepill.application("#{@app_name}_resque", :log_file => "/var/log/bluepill.log") do |app|
 app.uid = app.gid = @owner_name

 queues.each do |queue, count|
 count.times.each do |idx|
 app.process("resque_worker_#{queue}_#{idx}") do |process|
 process.group = 'resque'
 process.pid_file = "/var/run/resque/#{@app_name}/worker_#{queue}_#{idx}.pid"
 process.working_dir = "/data/#{@app_name}/current"
 process.start_command = @rake_command +
 "RACK_ENV=#{@rails_env} RAILS_ENV=#{@rails_env} " +
 "QUEUE=#{queue} " +
 "resque:work"
 process.stop_command = <<-EOF
 kill -QUIT 
 sleep_count=0
 while [-e /proc/]; do
 sleep 1
 let "sleep_count+=1"
 if [$sleep_count -eq 60]; then
 kill -TERM 
 fi
 if [$sleep_count -ge 70]; then
 kill -KILL 
 fi
 done
 EOF
 process.start_grace_time = 5.seconds
 process.stop_grace_time = 75.seconds
 process.restart_grace_time = 80.seconds
 process.daemonize = true

 process.checks :mem_usage, :below => 150.megabytes, :every => 1.minute, :times => 3
 end
 end
 end
end

What I wanted to add was a notification system that let me know when things had gone horribly wrong. Specifically I wanted Campfire notifications, as that’s where all the cool techies at PharmMD hang out.

For notifications, Bluepill allows us to write simple ‘Triggers’. These triggers can be added to a process in the config file, like zoh:

1
process.checks :some_trigger, :every => 1.minute

So let’s add a Campfire notifier. First things first, I’ll be using the tinder gem to talk to Campfire, have a look at their README for details on how it works (or the excellent tinder homepage). Install tinder.

1
gem install tinder

At the top of the Bluepill config file (anywhere above the application block) add the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
require 'rubygems'
require 'uri'
require 'tinder'

class Campfire < Bluepill::Trigger
 PARAMS = [:times, :within, :retry_in, :rails_env]

 attr_accessor *PARAMS

 def initialize(process, options = {})
 options.reverse_merge!(:times => 5, :within => 1, :retry_in => 5)

 options.each_pair do |name, val|
 instance_variable_set("@#{name}", val) if PARAMS.include?(name)
 end

 super
 end
 def notify(transition)
 @logger.info "I so just got triggered with a transition!"
 begin
 if transition.to == "down" && transition.from == "up"
 @logger.info "BluePill: #{@rails_env} worker(#{process.name}) went from #{transition.from_name} to #{transition.to_name}. Restarting them."
 campfire = Tinder::Campfire.new 'pharmmd', :token => 'TOKEN'
 room = campfire.find_room_by_name('General')
 room.speak "BluePill: #{@rails_env} worker(#{process.name}) died. Restarting it."
 end
 rescue => x
 @logger.info "Totally failed with: #{x.message}"
 end
 end
end

In the initialize method I set up some parameters that I can pass in, as well as provide myself with some defaults to fall back on.

1
2
PARAMS = [:times, :within, :retry_in, :rails_env]
attr_accessor *PARAMS

I create instance variables for each option for access later on, and call super to allow the Bluepill::Trigger class to do its own initialisation.

1
2
3
options.each_pair do |name, val|
 instance_variable_set("@#{name}", val) if PARAMS.include?(name)
end

Next up I create one of the required methods that Bluepill will call periodically and sprinkle in a little logging (@logger is provided by Bluepill::Trigger).

1
@logger.info "I so just got triggered with a transition!"

Next up we wrap our Campfire notification in a rescue block, because we don’t want ay silly exceptions like a lack of net access from bombing Bluepill out. There are a few transitions we can use, the important one for me is when a process dies (even though Bluepill will restart it), I want to know when that happens so I can find out why and stop it.

1
if transition.to == "down" && transition.from == "up"

Tinder makes campfire access so simple. Connect to campfire, select a room, and then ‘speak’ into it.

1
2
3
campfire = Tinder::Campfire.new 'pharmmd', :token => 'TOKEN'
room = campfire.find_room_by_name('General')
room.speak "BluePill: #{@rails_env} worker(#{process.name}) died. Restarting it."

That’s it for the trigger. Now all we need to do is tell the process that it should fire that trigger. Add the following to the process block just beneath where we checked for memory usage.

1
process.checks :campfire, :rails_env => @rails_env

That’s it. Now when you load the config file in, Campfire will get notifications if the process needs restarting.

Simple Campfire notifications using Tinder and Bluepill? Done.