Dynamic Queue Assignment for Resque Jobs

Posted by Ariejan de Vroom on 16-3-2010

Resque is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Sounds great! Let’s dive in directly:

1
2
3
4
5
6
7
8
9
10
class Archive
  @queue = :file_serve

  def self.perform(repo_id, branch = 'master')
    repo = Repository.find(repo_id)
    repo.create_archive(branch)
  end
end

Resque.enqueue(Archive, @repo.id)

This example was taken from the Resque README. It works great, but there’s a problem: you hard-code the queue to use. For an app I’m currently working on this is unwanted behaviour. So, how do you make Resque pick the right queue based on the data you feed it? In the following example I’ll show you how to pick the right queue based on the user that requests the job.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class StreamWorker  
  def self.enqueue(resource)
    Resque::Job.create(select_queue(resource), self, resource.id)
  end    

  def self.select_queue(resource)
    resource.user.use_priority_queue? ? :stream_high : :stream_low
  end

  def self.perform(resource_id)
    resource = Resource.find_by_id(resource_id)
    return false if resource.nil?
    resource.process!
  end
end

Instead of using Resque’s convenience method Resque.enqueue we create a Resque::Job ourselves. The select_queue determines which queue is used. Creating a StreamWorker job in the right queue is as easy as running:

1
StreamWorker.enqueue(@resource)

In case you’re wondering, the User#use_priority_queue? method returns a boolean based on the role the user has. Happy queueing!

Ariejan de Vroom

Software Engineer • CodeRetreat Facilitator • Ruby, Go and C Programmer • Electronics Apprentice