A few weeks ago, after deploying some new code to production I received a notification about a strange new-to-me error:
ActiveRecord::ConcurrentMigrationError: Cannot run migrations because another migration process is currently running
The new deployed code consisted only of Solidus and Rails minor version upgrades, so at first I suspected some unanticipated bug/incompatibility. But the error had not happened either on CI or on the staging server, so I concluded that it was probably something more pertinent to the application environment.
Let’s get back to the error message, which happens to be quite self-explanatory.
I started to reason about what could have gone wrong: while staging and CI run on
single Amazon instances, the production environment is based on multiple AWS
instances, and the Rails deploy procedure is basically concurrent on each of them… that’s just
how Opsworks does it.
So, during the deploy process one instance was still running migrations when also
tried to run them as well, eventually raising the error.
The error is generated by a Rails feature named Advisory locking added in 2015:
Attempting to run a migration while another one is in process will raise a ConcurrentMigrationError instead of attempting to run in parallel with undefined behavior. This could be rescued and the migration could exit cleanly instead. Perhaps as a configuration option?
This error can be quite insidious, as the deploy will fail on that instance, which will retain the previous application code. This time we were quite lucky as the new code was still functionally the same as before, but in other cases this could have easily caused a few headaches.
Imagine having one instance of the application serving outdated pages, not showing new
Solidus' features, or showing features that don’t exist anymore. And this is not the worst case
scenario… the instance could be throwing
500 server error to each visitor if something
more serious was wrong.
So, in order to fix the error (or, better said, ignore it and live happily) we customized the migration rake as follows:
# lib/tasks/migrate_ignore_concurrent.rake namespace :db do namespace :migrate do desc 'Run db:migrate but ignore ActiveRecord::ConcurrentMigrationError errors' task ignore_concurrent: :environment do begin Rake::Task['db:migrate'].invoke rescue ActiveRecord::ConcurrentMigrationError # Do nothing end end end end
and run this instead of the usual one:
bundle exec rake db:migrate:ignore_concurrent
This solution is, as often happens in life, not without caveat.
Consider the rare case when the first instance that got the lock is running a long DB migration (I’m not judging here, but you should not have long running migrations) which eventually fails. All other instances that tried to acquire the lock would now be already running the new code which also probably requires the new DB… what a day! So, all in all, it’s just a matter of tradeoffs, I chose mine :)