I’ve spent an entire afternoon trying to debug an issue we’ve been having with one of our Rails apps.

It is deployed on a Linux server, where Monit is in charge of supervising the Unicorn processes.

A quick reminder :

• Unicorn is a Rack compatible application server (comparable to Passenger, Puma, Thin…) ;
• Monit is a daemon configured to look after resources, able to execute commands if the resource is not in the correct state. For example you can start/stop another daemon with Monit ;
• Rails is a framework written in Ruby.

With one (and only one) of our many Rails apps, but on every front end server, we couldn’t start or restart Unicorn with Monit.

Monit

Usually when something doesn’t start (or crashes) you look at the log files, or you try to pipe the standard error output to something you can read. But with Monit it’s another world of debugging pain that you’re about to enter.

With probably many good reasons, Monit is not executing your commands as is. It is able to execute them as another user, but the environment is almost completely blank (except for a few MONIT_XXX variables). The PATH is unset, as most of what you’re used to rely on.

A common trick is to start the command by a shell invocation, and you command. For example:

$ /bin/bash -c 'my_start_command'

or even

$ /bin/bash -c 'PATH=./bin:$PATH KEY=val my_start_command'

Our issue

So our app wasn’t starting when launched by Monit, but was starting OK when executing the exact start command from the Monit configuration. A typo wasn’t the issue.

The complete start command is like this :

$/bin/sh -c 'PATH=/home/user/.rbenv/bin:/home/user/.rbenv/shims:$PATH /home/user/app/current/bin/unicorn -E staging -c /home/user/app/current/config/unicorn/staging.rb -D'

I needed to find a way to print the output of this command. I’ve found a good solution. So I’ve put this wrapper call between my ENV setup and Unicorn’s binary execution.

My command was then :

$ /bin/sh -c 'PATH=/home/user/.rbenv/bin:/home/user/.rbenv/shims:$PATH /usr/local/bin/monit_wrapper.sh /home/user/app/current/bin/unicorn -E staging -c /home/user/app/current/config/unicorn/staging.rb -D'

In the log file I’ve found this :

/home/user/app/shared/bundle/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/configurator.rb:664:in `parse_rackup_file':
invalid byte sequence in US-ASCII (ArgumentError)

A rackup_file is usually a config.ru file (written in Ruby) to configure the Rack part of the application.

But our rackup file was begining by # encoding UTF-8, which is the Ruby way to say that the content of the file is UTF-8 encoded.
For a reason beyond my knowledge, Ruby was not using this information when Unicorn was (simply) reading its content.

The solution

Looking closely at the Monit wrapper’s output, I’ve noticed that the LANG environment variable was not set (removed by Monit), so Ruby was defaulting back to ASCII for external encoding.

We’ve been using Monit + Rails + Unicorn for years within a dozen of projects, without any issue. Why now?

On this app, it was blowing up because of UTF-8 characters in the rackup file. If we removed them, it would start normally, but we needed them.
It turns out that Ruby or Rails is taking care of setting the correct encoding during the normal execution of the process. But when Unicorn was parsing this rackup file very early on, it only used the information he add from its environment.

Adding an explicit LANG=en_US.UTF-8 in the start command solved the issue.

$ /bin/sh -c 'PATH=/home/user/.rbenv/bin:/home/user/.rbenv/shims:$PATH LANG=en_US.UTF-8 /home/user/app/current/bin/unicorn -E staging -c /home/user/app/current/config/unicorn/staging.rb -D'

The good thing is that it’s modifying the environment only when Monit starts the app, without changing anything in the app itself.

I hope this explaination will help someone else and prevent them from losing a handful of hours.