I work on a Rails (4.1) application, sitting behind Unicorn and backed by a couple of Sidekiq processes.

A quick reminder : Unicorn is an application server (for Rack-compatible Ruby applications) based on the multi-process master-workers model, and Sidekiq is a background processing based on the multi-threaded model.

The jobs we put in the Sidekiq queue are also multi-threaded and need to access the database behind ActiveRecord in parallel. It forces us to have a bigger connection pool than usual : 20 instead of the default 5.

But the main part of the application, which run solely inside Unicorn, doesn’t need to use that much connections. In fact we have 16 Unicorn workers, so the bigger the pool is the more connections are opened.

Even if they are mainly idle, it’s still a waste of resource and it make the supervision of resources consumption more difficult.

Here how I’ve managed to have a separate pool size for Sidekiq and Unicorn.

Unicorn

A typical Unicorn configuration contains something like this

before_fork do |server, worker|
  # the following is recommended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end
end

after_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end
end

Before forking, the master process close all its connections, and after forking, every worker re-open their connections.

ActiveRecord::Base.establish_connection can be called with a connection “name” if there is something by that name in the config/database.yml file.

Let’s create such a configuration, by overriding only what’s necessary :

production: &production
  adapter: mysql2
  host: localhost
  port: 3306
  database: database_name
  username: my_app
  password: password
  encoding: utf8
  reconnect: true
  pool: 5

production_unicorn:
  <<: *production
  username: my_app_unicorn
  pool: 5

You can see that I’ve kept the same pool setting’s default value. That way It’s easier to have a Unicorn setting different from the default setting, used when I run Rake tasks…

I’ve also changed the username because I want to differentiate them when looking at the database opened connections. It’s completely optional.

Then we nee to configure Unicorn to use that new configuration :

after_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    spec = "#{Rails.env}_unicorn"
    if Rails.application.config.database_configuration.key?(spec)
      ActiveRecord::Base.establish_connection(spec.to_sym)
    else
      ActiveRecord::Base.establish_connection
    end
  end
end

Sidekiq

For Sidekiq it’s quite similar. We begin by adding another override in the config/database.yml file :

production_sidekiq:
  <<: *production
  username: my_app_sidekiq
  pool: 20

NB : I set Sidekiq up in the config/initializers/sidekiq.rb file, but you can also put this in many other places. Consult Sidekiq’s documentation for more details.

Let’s use our custom connection inside Sidekiq :

Sidekiq.configure_server do |config|
  if defined?(ActiveRecord::Base)
    spec = "#{Rails.env}_sidekiq"
    if Rails.application.config.database_configuration.key?(spec)
      ActiveRecord::Base.establish_connection(spec.to_sym)
    else
      ActiveRecord::Base.establish_connection
    end
  end
end