Custom Triggers With the Bluepill Monitoring Gem

We use Chef for automating our server infrastructure, but I’ll show you the manual way of installing Bluepill (either locally or remotely) so you can play with it.

1
gem install bluepill

That wasn’t hard, was it? Dependencies will be resolved, and installed too. Now we’ve got it installed, let’s take a little look at the commands we can give it.

Check the status of all applications and processes

1
sudo bluepill status

Start all applications and stop all applications, or a single group.

1
2
3
sudo bluepill start
sudo bluepill stop
sudo bluepill start resque

Quit Bluepill.

1
$ sudo bluepill quit

You configure Bluepill with a configuration file. Here’s one we use to start multiple resque queues. It’s a work in progress but does what we want for now. I won’t explain the config file because there’s plenty of blogs (and of course the Bluepill README) that go into detail here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
queues = {}
queues["comms"] = 1
queues["rules"] = 2
queues["patients"] = 4
queues["import"] = 1

@app_name   = "pharmmd"
@owner_name = "rails"
@rails_env  = "production"
@rake_command = "bundle exec rake "

Bluepill.application("#{@app_name}_resque", :log_file => "/var/log/bluepill.log") do |app|
  app.uid = app.gid = @owner_name

  queues.each do |queue, count|
    count.times.each do |idx|
      app.process("resque_worker_#{queue}_#{idx}") do |process|
        process.group = 'resque'
        process.pid_file = "/var/run/resque/#{@app_name}/worker_#{queue}_#{idx}.pid"
        process.working_dir = "/data/#{@app_name}/current"
        process.start_command = @rake_command +
          "RACK_ENV=#{@rails_env} RAILS_ENV=#{@rails_env} " +
          "QUEUE=#{queue} " +
          "resque:work"
        process.stop_command = <<-EOF
        kill -QUIT 
        sleep_count=0
        while [ -e /proc/ ]; do
          sleep 1
          let "sleep_count+=1"
          if [ $sleep_count -eq 60 ]; then
            kill -TERM 
          fi
          if [ $sleep_count -ge 70 ]; then
            kill -KILL 
          fi
        done
        EOF
        process.start_grace_time = 5.seconds
        process.stop_grace_time = 75.seconds
        process.restart_grace_time = 80.seconds
        process.daemonize = true

        process.checks :mem_usage, :below => 150.megabytes, :every => 1.minute, :times => 3
      end
    end
  end
end

What I wanted to add was a notification system that let me know when things had gone horribly wrong. Specifically I wanted Campfire notifications, as that’s where all the cool techies at PharmMD hang out.

For notifications, Bluepill allows us to write simple ‘Triggers’. These triggers can be added to a process in the config file, like zoh:

1
process.checks :some_trigger, :every => 1.minute

So let’s add a Campfire notifier. First things first, I’ll be using the tinder gem to talk to Campfire, have a look at their README for details on how it works (or the excellent tinder homepage). Install tinder.

1
gem install tinder

At the top of the Bluepill config file (anywhere above the application block) add the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
require 'rubygems'
require 'uri'
require 'tinder'

class Campfire < Bluepill::Trigger
  PARAMS = [:times, :within, :retry_in, :rails_env]

  attr_accessor *PARAMS

  def initialize(process, options = {})
    options.reverse_merge!(:times => 5, :within => 1, :retry_in => 5)

    options.each_pair do |name, val|
      instance_variable_set("@#{name}", val) if PARAMS.include?(name)
    end

    super
  end
  def notify(transition)
    @logger.info "I so just got triggered with a transition!"
    begin
      if transition.to == "down" && transition.from == "up"
        @logger.info "BluePill: #{@rails_env} worker(#{process.name}) went from #{transition.from_name} to #{transition.to_name}. Restarting them."
        campfire = Tinder::Campfire.new 'pharmmd', :token => 'TOKEN'
        room     = campfire.find_room_by_name('General')
        room.speak "BluePill: #{@rails_env} worker(#{process.name}) died. Restarting it."
      end
    rescue => x
      @logger.info "Totally failed with: #{x.message}"
    end
  end
end

In the initialize method I set up some parameters that I can pass in, as well as provide myself with some defaults to fall back on.

1
2
PARAMS = [:times, :within, :retry_in, :rails_env]
attr_accessor *PARAMS

I create instance variables for each option for access later on, and call super to allow the Bluepill::Trigger class to do its own initialisation.

1
2
3
options.each_pair do |name, val|
  instance_variable_set("@#{name}", val) if PARAMS.include?(name)
end

Next up I create one of the required methods that Bluepill will call periodically and sprinkle in a little logging (@logger is provided by Bluepill::Trigger).

1
@logger.info "I so just got triggered with a transition!"

Next up we wrap our Campfire notification in a rescue block, because we don’t want ay silly exceptions like a lack of net access from bombing Bluepill out. There are a few transitions we can use, the important one for me is when a process dies (even though Bluepill will restart it), I want to know when that happens so I can find out why and stop it.

1
if transition.to == "down" && transition.from == "up"

Tinder makes campfire access so simple. Connect to campfire, select a room, and then ‘speak’ into it.

1
2
3
campfire = Tinder::Campfire.new 'pharmmd', :token => 'TOKEN'
room     = campfire.find_room_by_name('General')
room.speak "BluePill: #{@rails_env} worker(#{process.name}) died. Restarting it."

That’s it for the trigger. Now all we need to do is tell the process that it should fire that trigger. Add the following to the process block just beneath where we checked for memory usage.

1
process.checks :campfire, :rails_env => @rails_env

That’s it. Now when you load the config file in, Campfire will get notifications if the process needs restarting.

Simple Campfire notifications using Tinder and Bluepill? Done.

Comments
    

Comments