Elixir/Phoenix deployments using Distillery

Of late, we have been porting one of our internal apps from Rails to Phoenix. We are using Capistrano for deploying Rails apps. We have Jenkins CI which listens to features merged into master, and uses Capistrano to deploy to production.

For Elixir/Phoenix, we are looking for something similar. Merge features into master, let Jenkins CI run, and package Phoenix app which can be run on production. In Elixir world, there are bunch of package managers

Exrm: Legacy one, works mostly. Does not support latest Elixir features like umbrella apps
Edeliver: built on top of Exrm, has it's own learning curve.
Distillery: Rewrite of Exrm aimed to be part of Mix itself with goodies.

You can read more about Exrm and Distillery here

Since Distillery is the latest one and also fits our use case nicely, let's dig through that tool more. Documentation of Distillery is quite nice. In this blog post, we are going to explore:

How to initialize a simple Phoenix app.
Configure Jenkins CI script to deploy Phoenix app
Doing hot upgrades to deployed Phoenix app without bringing app down
Managing versions of app, and Distillery plugins.

Before getting started, you need these:

Dev machine: Mostly Linux or OSX, your choice. We have tested this on Linux.
Build server: System which has the same platform as that of production, mostly Linux. You can also use CI server as your build system.
Production server: Well, you know this :)

Since Elixir has a compilation step where it compiles Elixir code to BEAM, we need to set up a work flow for compiling and deploying our Elixir application. A typical work flow would look like:

Write code on your development machine, and push the same to any source control of your choice, say Github.
Make CI server listen to commits, or PR merges, pull code from Github.
Install dependencies like Erlang, Elixir, development tools like GCC on CI server.
Validate changes (means running specs, and all). Compile code to BEAM.
Package compiled code, along with Erlang runtime.
Ship the package to production server, unpack, and start the app.

Interesting thing to note here is, CI server needs source code of all dependencies of the Elixir app, but production server uses only the compiled BEAM code just to make it clear. So there is no bloat on production servers, it's easy to spin up new servers, unpack the package, and start the app.

Let's explore Distillery, and how it helps in deploying Phoenix apps. The steps to create and run the distillery based app are taken from Distillery documentation.

Create Phonenix app and initialize Distillery

Let's create a simple Phoenix app which we will use throughout our discussion.

 
> mix phoenix.new --no-ecto --no-brunch phoenix_app
* creating phoenix_app/config/config.exs
* creating phoenix_app/config/prod.secret.exs
* creating phoenix_app/config/test.exs
* ...
 
Fetch and install dependencies? [Yn] Y
* running mix deps.get

We are all set! Run your Phoenix application:

 
> cd phoenix_app
> mix phoenix.server

If it's all good, you should be able to see the Phoenix app at localhost:4000 Now, open mix.exs and append distillery

  defp deps do
    [{:phoenix, "~> 1.2.1"},
     {:phoenix_pubsub, "~> 1.0"},
     {:phoenix_html, "~> 2.6"},
     {:phoenix_live_reload, "~> 1.0", only: :dev},
     {:gettext, "~> 0.11"},
     {:cowboy, "~> 1.0"},
     {:distillery, "~> 0.10.1"}]
  end

Now, fetch dependencies using mix, and initialize repository with Distillery

 
> mix deps.get
> mix release.init

This should create a rel folder with config.exs file in it. Please take a moment to go through this file which is well commented to understand what it does.

Configuring CI and generating release pack

CI does all the heavy lifting in order to cut a release. You can run these commands locally and replicate the same on CI server. Make sure CI server has these packages installed:

Erlang (Erlang solutions)
Elixir

Cutting a release is very easy, just run this command:

 
> MIX_ENV=prod mix release --env=prod
==> gettext
Compiling 1 file (.erl)
...
==> Assembling release..
==> Building release phoenix_app:0.0.1 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Packaging release..
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/phoenix_app/bin/phoenix_app console
      Foreground: rel/phoenix_app/bin/phoenix_app foreground
      Daemon: rel/phoenix_app/bin/phoenix_app start

If you look at the output of this command, you'll notice that mix is packaging everything along with the app. Interpreting mix release output:

phoenix_app: Compiled files of all source code in deps folder along with source code of the Phoenix app.
ERTS: Erlang runtime which is required to run the server. Note that since mix is packaging runtime also, there is no need to install Erlang on production machine.
Helpful commands to run Phoenix app on server.

Optionally you can set up Travis/Jenkins to observe features merged into git, automatically pulling latest source code, and packaging app

Now, take a look at rel/phoenix_app folder:

 
rel
└── phoenix_app
    ├── bin
    │   ├── nodetool
    │   ├── phoenix_app
    │   ├── release_utils.escript
    │   └── start_clean.boot
    ├── erts-8.1
    │   ├── bin
    │   │   ├── beam
    │   │   ├── beam.smp
    ....
    │   ├── include
    │   │   ├── driver_int.h
    │   │   ├── erl_nif.h
    │   ├── lib
    │   │   ├── internal
    │   │   ├── liberts.a
    │   │   └── liberts_r.a
    ├── lib
    │   ├── compiler-7.0.2
    │   ├── cowboy-1.0.4
    │   ├── crypto-3.7.1
    ...
    │   ├── phoenix-1.2.1
    │   ├── phoenix_app-0.0.1
    │   ├── ranch-1.2.1
    └── releases
        ├── 0.0.1
        │   ├── commands
        │   ├── hooks
        │   ├── phoenix_app.boot
        │   ├── phoenix_app.rel
        │   ├── phoenix_app.script
        │   ├── phoenix_app.sh
        │   ├── phoenix_app.tar.gz
        │   ├── start_clean.boot
        │   ├── sys.config
        │   └── vm.args
        ├── RELEASES
        └── start_erl.data

Going through the top level folders:

erts: Erlang runtime, contains beam files, library files to run Erlang
lib: all compiled dependencies of phoenix_app, this folder resembles very similar to deps folder at root level
releases: Folder of our interest. By default, app version will be 0.0.1. All the files required to run phoenix_app. Inside a release version folder, all files are packaged into a single .tar.gz file. This file can be copied to production server for running the app.

Making successive deployments

Erlang has a rich heritage, and Erlang programs are designed to run for years without bringing servers down, which guarantees nearly 100% up time. Rolling out bug fixes, new features, improvements are done using hot updates to servers. Erlang provides ways to patch existing running code on production servers so that there is no need to stop and start the app. Let's look at ways to deploy phoenix_app

Hot upgrades

distillery provides support to create releases which can be applied as hot upgrades. The process of generating the tar file is same, but the command is different. For the sake of brevity, change version number in mix.exs from 0.0.1 to 0.0.2 before proceeding

 
> MIX_ENV=prod mix release --env=prod --upgrade
==> Assembling release..
==> Building release phoenix_app:0.0.2 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Generated .appup for phoenix_app 0.0.1 -> 0.0.2
==> Relup successfully created
==> Packaging release..
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/phoenix_app/bin/phoenix_app console
      Foreground: rel/phoenix_app/bin/phoenix_app foreground
      Daemon: rel/phoenix_app/bin/phoenix_app start

The command to create a new release is same as that of first run, but there is a new argument, i.e --upgrade. Also, output of command is also slightly different. Generated .appup for phoenix_app 0.0.1 -> 0.0.2, where .appup means hot upgrade. There will be new folder 0.0.2 under releases. There will be another file called relup which contains instructions about how to upgrade. It looks like this:

{"0.0.2",
 [{"0.0.1",[],
   [{load_object_code,{phoenix_app,"0.0.2",
                                   ['Elixir.PhoenixApp.Endpoint',
                                    'Elixir.PhoenixApp.Gettext']}},
    point_of_no_return,
    {load,{'Elixir.PhoenixApp.Endpoint',brutal_purge,brutal_purge}},
    {load,{'Elixir.PhoenixApp.Gettext',brutal_purge,brutal_purge}}]}],
 [{"0.0.1",[],
   [{load_object_code,{phoenix_app,"0.0.1",
                                   ['Elixir.PhoenixApp.Endpoint',
                                    'Elixir.PhoenixApp.Gettext']}},
    point_of_no_return,
    {load,{'Elixir.PhoenixApp.Endpoint',brutal_purge,brutal_purge}},
    {load,{'Elixir.PhoenixApp.Gettext',brutal_purge,brutal_purge}}]}]}.

This file contains instructions about how to switch between 0.0.1 and 0.0.2. If the upgrade needs to be rolled back, this file helps in downgrading from 0.0.2 to 0.0.1.

Note about versions

Mix knows only semantic versioning. If one has to use SHA-IDs as the version, mix will throw errors. Say, you change version from 0.0.2 to 48dcbccd, and try to generate new package, mix throws this error:

 
> MIX_ENV=prod mix release --env=prod --upgrade
Compiling 11 files (.ex)
** (Mix) Expected :version to be a SemVer version, got: "48dcbccd"

If you are coming from Rails world, and used to continuous deployments using Capistrano, editing version every time for deployments is a pain. Let's see how we can get Capistrano kind of continuous deployments.

Generating versions on the fly

One way to generate versions on the fly is to read version from environment variable. Also, since mix enforces semantic versioning, generating incremental versions makes sense. Capistrano generates folders with YYYYMMDDHHMMSS format, i.e. year, month, date. Change mix.exs file to have version like this:


def project do
  [app: :phoenix_app,
   version: (if Mix.env == :prod, do: System.get_env("APP_VERSION"), else: "0.0.1"),
   elixir: "~> 1.2",
   elixirc_paths: elixirc_paths(Mix.env),
   compilers: [:phoenix, :gettext] ++ Mix.compilers,
   build_embedded: Mix.env == :prod,
   start_permanent: Mix.env == :prod,
   deps: deps()]
end

Now when mix is building package for production, the app read the version from the environment variable, otherwise it's hard coded to 0.0.1. Since version can be dynamic now, we can let CI specify what version needs to be generated. Following how Capistrano generates versions, we can do

export APP_VERSION=`date +%Y.%m.%d%H%M`
 
npm install                     # for brunch
mix deps.get
MIX_ENV=prod mix release --env=prod --upgrade

So, this code generates the app version with year as major version, month as minor version, and <date>``<hours>``<minutes> as patch version. Note that in order for --upgrade to work on CI, you should have previous versions in releases folder, otherwise upgrade will fail because there is no reference release. If you are using Travis/Circle-CI, make sure that releases folder is cached.

Recovering from failures, doing proper patch updates

Say code which is pushed in is buggy, and CI has made a release. It can happen that hot upgrade itself fails. In such cases, CI will have a stale releases like this:

 
rel/
  phoenix_app/
    releases/
      0.0.1             # deploy succeeded
      2016.09.250826    # deploy succeeded
      2016.09.250829    # deploy failed
      2016.09.251007    # new package based on failed deploy package

When CI runs again, it will generate a new package assuming previous package is a good one. Hot upgrades will start failing from now on!

It's very important to keep a note of which deploy succeeded, or which one failed. There are two ways this can be done:

Maintain revisions.log just like how Capistrano keeps track of deploys.
Read which version is currently deployed directly from the production server itself.

If you follow 2nd approach, you can read current running version from the file start_erl.data. It contains runtime version and app version. CI can read that version from production server, and create a hot upgrade from that version. In addition to --upgrade option, Distillery supports --upfrom option also. This takes in app version from which upgrade package has to be generated. Typically CI code looks like this:

 
export PREV_APP_VERSION=`ssh mix@10.0.0.5 cat /srv/app/releases/start_erl.data | cut -d' ' -f2`
export APP_VERSION=`date +%Y.%m.%d%H%M`
 
touch mix.exs                   # so that new app_version is picked
npm install                     # for brunch
mix deps.get
MIX_ENV=prod mix release --env=prod --upgrade --upfrom=$PREV_APP_VERSION

Now hot upgrades always work!

Immutable infrastructure

Immutable infra means - services in production have to be treated as immutable entities and upgrades should not change anything in production, but instead should be redeployed each time. This means no hot-upgrades are allowed in production. If you are running stateless services like web servers written in Phoenix, sticking with immutable infrastructure is recommended. Just create a package, spin up a new node behind load balancer, run the app, and nuke old nodes.

Compiling assets using plugins

NOTE: We haven't initialized our app to use brunch, so the following section won't work. Try creating a new app without --no-brunch option, and follow this section.

Mostly people use Phoenix for APIs, and use Reactjs or other JS framework for front-end. If you are reading through Distillery docs, it suggests to push assets compilation to shell script itself. Let's take a small detour and see how we can do that using plugins. A Distillery plugin can be used to hook into the package generation process. It has 5 hooks:

before_assembly
after_assembly
before_package
after_package
after_cleanup

Names are pretty self-explanatory. If you want to know more about these, you can take a look at them here. You can hook into before_assembly block, and compile assets.


defmodule PhoenixApp.PhoenixDigestTask do
  use Mix.Releases.Plugin

  def before_assembly(%Release{} = _release) do
    info "before assembly!"
    case System.cmd("npm", ["run", "deploy"]) do
      {output, 0} ->
        info output
        Mix.Task.run("phoenix.digest")
        nil
      {output, error_code} ->
        {:error, output, error_code}
    end
  end

  def after_assembly(%Release{} = _release) do
    info "after assembly!"
    nil
  end

  def before_package(%Release{} = _release) do
    info "before package!"
    nil
  end

  def after_package(%Release{} = _release) do
    info "after package!"
    nil
  end

  def after_cleanup(%Release{} = _release) do
    info "after cleanup!"
    nil
  end
end


environment :prod do
  set plugins: [PhoenixApp.PhoenixDigestTask]
  set include_erts: true
  set include_src: false
end

So, more Elixir code and less shell scripting. When you run release command again, output looks like this:

 
==> Assembling release..
==> Building release phoenix_app:2016.10.251007 using environment prod
==> before assembly!                 ## <- Plugin print here
==>
> brunch build --production
 
25 Oct 10:07:47 - info: compiled 3 files into 2 files, copied 3 in 3.1 sec
 
Check your digested files at "priv/static"
==> Including ERTS 8.0.2 from /usr/lib/erlang/erts-8.0.2
==> Generated .appup for phoenix_app 2016.09.261241 -> 2016.10.251007
==> Relup successfully created
==> after assembly!                  ## <- Plugin print here
==> Packaging release..
==> before package!                  ## <- Plugin print here
==> after package!                   ## <- Plugin print here
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: rel/phoenix_app/bin/phoenix_app console
      Foreground: rel/phoenix_app/bin/phoenix_app foreground
      Daemon: rel/phoenix_app/bin/phoenix_app start

You can see logs from the plugin throughout the deploy process. Plugins is an interesting concept, please take a look at that.

You can find whole source code here

Thanks for reading! If you would like to get updates about subsequent blog posts from Codemancers, do follows us on twitter: @codemancershq. Feel free to get in touch if you'd like Codemancers to help you build your Elixir/Phoenix app.