Attractor is a pretty background-job intensive app. If you think about it, every commit’s changes have to be analyzed, and the results stored, to keep track of code quality and emerging tech debt.
Since it’s based on Bullet Train, it makes sense to handle most of these workloads in incoming webhooks. Broadly speaking, there are three groups of incoming webhooks at work:
Each incoming webhook in Bullet Train is processed by a controller that is created by super scaffolding.
It might look like this:
class Webhooks::Incoming::SandboxWebhooksController < Webhooks::Incoming::WebhooksController before_action :authenticate_token! def create Webhooks::Incoming::SandboxWebhook.create(data: JSON.parse(request.body.read)) .process_async render json: {status: "OK"}, status: :created end # ... end
The created model file contains the processing logic:
class Webhooks::Incoming::SandboxWebhook < ApplicationRecord include Webhooks::Incoming::Webhook def process # logic goes here end end
The actual processing is done by a generic Webhooks::Incoming::WebhookProcessingJob
that’s part of Bullet Train and calls back to this process
method. In other words, in a vanilla Bullet Train app, Sidekiq is responsible for working off your incoming webhooks.
So, let’s look at the respective sidekiq.yml
file:
:concurrency: 5 staging: :concurrency: 5 production: :concurrency: 5 :queues: - critical - default - mailers - low - action_mailbox_routing
There are 5 Sidekiq queues configured. A pretty nasty issue is hidden here in plain sight, though: All incoming webhooks are processed in the default
queue.
Why is this a problem? Let’s walk through a scenario where a new customer signs up.
What’s happening here? It took me a while to find out.
As pointed out above, all incoming webhooks are processed in the default
queue, which in this case means:
In other words, the payment would be processed only after all analysis jobs (and not only hers, maybe a couple of other customers did sign up simultaneously) have concluded. An unbearable situation, to have a customer hanging in a state of uncertainty whether her payment did successfully activate her subscription of Attractor.
To remedy this, I could have changed the sign up flow to first require potential customers to check out, then allow them to connect the GitHub app. That would have been a pretty large effort, though, so I opted to try something else.
We can reopen the Webhooks::Incoming::WebhookProcessingJob
and have the queue_as
method decide dynamically where to queue the webhook it’s meant to process:
Webhooks::Incoming::WebhookProcessingJob.queue_as do webhook = arguments.first case webhook when Webhooks::Incoming::PaddleWebhook :critical when Webhooks::Incoming::SandboxWebhook :low else :default end end
The only remaining question was where to put this small monkey patch. Placing it into an initializer didn’t work, because your app’s constants (such as the Webhooks::Incoming::WebhookProcessingJob
class) haven’t loaded at this time.
So the only option is to run it after the app has finished initializing, i.e. in an after_initialize
block in your config/application.rb
:
require_relative "boot" require "rails/all" # ... module MyApp class Application < Rails::Application # more config config.after_initialize do Webhooks::Incoming::WebhookProcessingJob.queue_as do webhook = arguments.first case webhook when Webhooks::Incoming::PaddleWebhook :critical when Webhooks::Incoming::SandboxWebhook :low else :default end end end end end
With this tweak, all webhooks coming in from the code analysis sandbox are placed in the low job queue, keeping others in the default one. Subsequently, when a webhook comes in from GitHub indicating a new pull request, it is processed before any potentially waiting SandboxWebhook
s.
Even more importantly, incoming webhooks from Paddle are placed in the critical queue, making sure subscription changes are always processed first.
Thanks to Kasper Timm Hansen for pointing me in the right direction! 🙏
Keep shipping your Ruby and JavaScript apps fast and tackle tech debt before it hurts.