Our previous article in this series established that Solid Queue is an excellent choice if you need a system for processing background jobs. It minimizes external dependencies — no need for Redis! — by storing all jobs in your database. Despite that, it is incredibly performant.

But just being performant is not enough for a production-ready background job system. Rails developers have come to expect a lot over the years. We don't just want to enqueue jobs to run in the background. We want to schedule jobs, run them on a recurring schedule, and we might even want to limit how many jobs can run concurrently. We want more features!

Amazingly, Solid Queue provides all of those features out of the box. Let's dive deeper into Solid Queue and learn how that's possible!

Scheduling Jobs with Solid Queue for Ruby on Rails

First, it's time for a small recap. Solid Queue uses your database — and only your database — to store job data. Everything it does is backed by one database table or another. Scheduling jobs — that is, designating jobs to run at some specific point in the future — is no different. Any scheduled job is stored in solid_queue_scheduled_executions.

This table is almost identical to the solid_queue_ready_executions table. The only difference is the addition of the scheduled_at column, which tells us when a scheduled job is supposed to be executed. Let's confirm that by looking at what happens when we schedule a job.

There are no surprises there. Solid Queue adds a new row to the solid_queue_scheduled_executions table, which contains the data that we'd expect. But how do we go from such a record existing to actually running a job at the right time?

We need a process that continuously polls the solid_queue_scheduled_executions table. That process is called the Dispatcher, and it is responsible for executing scheduled jobs on time. It begins when Solid Queue starts — no additional configuration is required. However, if needed you can start only the dispatcher process by running Solid Queue with a specific configuration.

In case you were wondering how the Dispatcher process is supervised, that is the responsibility of the aptly named Supervisor. It keeps track of any running processes within Solid Queue, including worker processes and Dispatchers.

So, how does the Dispatcher actually work? It defines a poll method called within a loop to continuously check for scheduled jobs. The polling code is spread over several classes and modules, but in a heavily simplified form, it looks like this:

The query to retrieve 'ready' scheduled executions is straightforward.

So, any scheduled job with scheduled_at in the past is ready to be dispatched. As we covered in part one of this series, when Solid Queue dispatches a job, it creates a ReadyExecution record and destroys the corresponding ScheduledExecution record. The ReadyExecution record is then picked up by regular worker processes, and the corresponding job runs.

So far, so good. Scheduled jobs are really not that complicated! Let's look at something more complex: recurring tasks.

Recurring Tasks

Recurring tasks are an oft-requested feature for background job processors. Simply put, they are background jobs that should run on a recurring schedule. They are similar to Cron jobs in that you define a schedule (such as every five minutes, every day at noon, and so on) for when work should occur.

In Solid Queue, you configure your recurring jobs using the config/recurring.yml file. For example, if we wanted to run a CleanupData job every day at noon, this is how we would do it.

Solid Queue uses Fugit to parse schedule expressions, which is why human-readable schedules such as 'every day at noon' are permitted. When using scheduled tasks, you define the class of the job to be run and any job arguments. The excellent SolidQueue recurring tasks ReadMe provides more details. We're here to learn how it works, so let's look under the hood.

Recurring tasks are represented by the RecurringTask model, which is backed by a corresponding solid_queue_recurring_tasks table. The columns therein correspond to the fields available in the configuration file.

When you start SolidQueue, recurring task records are created according to your recurring tasks configuration file. To create jobs at the right time, we once again need a new process — this time called the Scheduler. The Scheduler is a sibling to the Dispatcher, which we already know about. It works in almost the same way: A new process is spun up when Solid Queue starts, and this process runs an endless loop. The difference between the Scheduler and the Dispatcher is what happens within that loop. Where the Dispatcher queries the solid_queue_scheduled_executions table, the Scheduler queries solid_queue_recurring_tasks — and schedules jobs at the right time. So, how exactly does the Scheduler know what the right time is, and when to schedule the right jobs?

To answer that question, we have to examine the implementation closely. The scheduler class creates a new RecurringSchedule object that defines a schedule method. That method is repeatedly called for each scheduled task. Here's a simplified version:

Let's untangle this code. Solid Queue uses Concurrent::ScheduledTask (from the concurrent-ruby library) to spawn a new thread. That thread is scheduled to run at the time specified by the recurring task's schedule. When that thread runs, it first recursively spawns another thread to schedule the next recurring task. Then, it enqueues the 'current' scheduled job.

Let's look at an example of a simple recurring task to get a handle on things.

If we start Solid Queue at 8:30, the variables within the schedule method are assigned the following values. Not verbatim, mind you. We're greatly simplifying here.

Variable	Value
task	my_periodic_task
task.delay_from_now	30 minutes
task.next_time	9:00

So, our background thread is scheduled to run thirty minutes from now, which is 9:00. When that time rolls around, the background thread is executed. It runs thread_task.enqueue(at: 9:00) — so an instance of CleanupData is queued for execution. It also calls itself recursively via thread_schedule.schedule. Because it is now 9:00, the variables for this invocation have changed.

Variable	Value
task	my_periodic_task
task.delay_from_now	59 minutes
task.next_time	10:00

So, the background thread is scheduled to run again at 10:00, and the cycle continues. You may be wondering what happens if the scheduling thread is killed, for example during a re-deploy or system crash. Won't that upset your schedules? Luckily, the answer is no. Cron schedules are static. An expression like 'Every Hour' always resolves to 10:00, 11:00, 12:00, and so on, regardless of when Solid Queue starts. Any interruptions to the scheduling thread don't change that.

Here are some other implementation details to be aware of. First, this pattern of scheduling the next occurrence of a recurring task before executing it is inspired by GoodJob. Second, RecurringTask.enqueue does not create a new Job and ReadyExecution record as you might expect. Instead, it creates yet another record, namely RecurringExecution.

This record is solely to avoid executing recurring jobs multiple times. It has an index on task_key and run_at with unique constraints to serve that purpose. A RecurringTask is only queued if there is no prior RecurringExecution for the same time and the same job.

Attentive readers will notice that this code snippet points to a limitation in Solid Queue. That is, if you are not using Solid Queue as a backend to run your cron-style tasks — yes, you can do that — Solid Queue can't guarantee that recurring jobs are enqueued only once. If you find yourself in such a situation, you should be aware of that.

You may also be wondering what happens if the Scheduler process dies or is killed — for example, during a deployment. Since recurrences are managed by a thread, won't killing the thread break schedules? Luckily, the answer to that is no.

Concurrency Controls

Let's look at one final feature of Solid Queue, namely concurrency controls. Sometimes, you want to limit how many jobs of a certain kind can run simultaneously. You can do so using Solid Queue with limits_concurrency.

Here, we are telling SolidQueue to run a maximum of one instance of MyJob for each user. Let's examine the configuration in more detail.

• to: The maximum number of jobs you want running concurrently.
• key: A required argument to designate which jobs should be limited together. In our example, jobs with the same User ID are limited to a single concurrent execution. You may use any job arguments as key, but constants such as strings or symbols are also allowed.
• duration: The maximum time for which Solid Queue can guarantee concurrency after a job is enqueued. If your jobs run longer than that, concurrency controls will not apply and jobs may overlap. We'll learn why later!
• group: You can use this option to limit concurrency across different job classes.

If you want to learn more, I refer you to the concurrency control documentation. Concurrency controls are easily Solid Queue's most sophisticated feature. If scheduled tasks didn't already make your head spin, learning how this feature works surely will.

Let's start with the basics. Like other Solid Queue features, concurrency controls are backed by various models and their corresponding database tables. Two that you need to be particularly aware of are Semaphore and BlockedExecution.

Let's look at Semaphore first. As the name suggests, this is an implementation of the counting semaphore pattern. Whenever Solid Queue enqueues a job with limits_concurrency, it first tries to acquire a semaphore lock based on a concurrency key. This concurrency key is based on the arguments passed to limits_concurrency, namely the job class, the key, and the group name — if any was provided. If the semaphore is available, the job is enqueued. If it is not, a BlockedExecution record is created instead.

Semaphores have a value to support multiple concurrent jobs. You can think of it as the remaining capacity of the semaphore. Acquiring a semaphore means decrementing that value, and releasing it means incrementing it. A semaphore is considered unavailable once its value is zero. Let's look at an example of how the locking mechanism works for a simple job.

Let's see what happens if we try to enqueue this job multiple times in succession.

1. The first instance of MyJob is enqueued. There is no semaphore yet, so one is created. Its initial value is limit - 1. Because your limit is three, the initial value of the semaphore is two.
2. The second instance of MyJob is enqueued. Solid Queue attempts to acquire a lock for that job. The job can be enqueued because the value is two, which is greater than zero. The value of the semaphore is decremented to one.
3. A third instance of our job is enqueued. We repeat the same procedure as before. The value of the semaphore is now zero.
4. A fourth instance of MyJob is enqueued. Acquiring the semaphore fails because its value is now zero. A BlockedExecution record is created for the job.
5. The first instance of our job finishes. When it finishes, it releases the semaphore, so the semaphore value is once again one.
6. On finishing, the first job instance also calls a method to release any blocked jobs.
7. The fourth instance of MyJob is released and again tries to acquire a lock. The semaphore value is one, so the lock can be acquired and the blocked job queued. The semaphore value is now zero.

The code for releasing a semaphore when a job finishes is straightforward.

There is one more detail that we haven't touched on yet. Why do semaphores have an expiry date, and why do we need to set a duration when using limits_concurrency?

Let's consider what happens when a job crashes without releasing its semaphore — for example, when a worker processing that job dies. Unless we add some mechanism to clean up semaphores, the lock held by that job will be retained forever. In the worst case, this would forever block other jobs from being processed.

Semaphores have an expiry that corresponds to the duration given in the job definition to avoid that situation. If a semaphore expires — which happens if no jobs are enqueued — the semaphore will be destroyed. We already know the process responsible for that — it's our friend, the Dispatcher. It instantiates the ConcurrencyMaintenance class, which does two things:

• First, it removes any expired semaphores.
• Second, it will check if there are any blocked jobs and release them.

Jobs are released one by one, so concurrency limits will still hold. Consider, however, what happens if your job runs longer than the given duration. In that case, the semaphore will be cleaned up, although the job will still run. If another job is then enqueued, those jobs will overlap.

Monitoring Solid Queue for Rails with AppSignal

As we've established, Solid Queue can do a lot. However, with all these moving parts, monitoring becomes crucial. Luckily, AppSignal provides built-in support for Solid Queue, with ready-made dashboards for job execution times, throughput, and failure rates. Simply install AppSignal in your Rails application, and you're good to go.

AppSignal will automatically detect your usage of Solid Queue and create an active job dashboard that contains graphs for important metrics, such as error rate and throughput.

If you ever see jobs that misbehave — either because they run slowly or have too many errors — assign them a status and assignee to effectively resolve the issue.

Obviously, you shouldn't have to look at dashboards all day to figure out if there is a problem. AppSignal Alerts has your back. Simply create a new alert for job metrics, such as failure rate and job duration, and you're all set.

Solid Queue is amazing for adding powerful job processing to your application without hassle. AppSignal does the same when it comes to monitoring!

Wrapping Up

We've covered a lot of ground in our exploration of Solid Queue's advanced features. From scheduled jobs to complex dependency chains, each feature builds on the solid foundation we discussed in part one. As we've seen, it's not easy to build a job processing backend. But by diving deep into the Solid Queue source code and its workings, we've gained an understanding and some appreciation for the challenges involved.

In any case, Solid Queue is a wonderful addition to the Rails ecosystem, due to its excellent database design and process coordination. It provides the tools you need while maintaining its core promise: simplicity and reliability, without external dependencies.

Happy coding!