Of late, we have been porting one of our internal apps from Rails to Phoenix. We are using Capistrano for deploying Rails apps. We have Jenkins CI which listens to features merged into master, and uses Capistrano to deploy to production.
For Elixir/Phoenix, we are looking for something similar. Merge features into master, let Jenkins CI run, and package Phoenix app which can be run on production. In Elixir world, there are bunch of package managers
- Exrm: Legacy one, works mostly. Does not support latest Elixir features like umbrella apps
- Edeliver: built on top of Exrm, has it's own learning curve.
- Distillery: Rewrite of Exrm aimed to be part of Mix itself with goodies.
You can read more about Exrm and Distillery here
Since Distillery is the latest one and also fits our use case nicely, let's dig through that tool more. Documentation of Distillery is quite nice. In this blog post, we are going to explore:
- How to initialize a simple Phoenix app.
- Configure Jenkins CI script to deploy Phoenix app
- Doing hot upgrades to deployed Phoenix app without bringing app down
- Managing versions of app, and Distillery plugins.
Before getting started, you need these:
- Dev machine: Mostly Linux or OSX, your choice. We have tested this on Linux.
- Build server: System which has the same platform as that of production, mostly Linux. You can also use CI server as your build system.
- Production server: Well, you know this :)
Since Elixir has a compilation step where it compiles Elixir code to BEAM, we need to set up a work flow for compiling and deploying our Elixir application. A typical work flow would look like:
- Write code on your development machine, and push the same to any source control of your choice, say Github.
- Make CI server listen to commits, or PR merges, pull code from Github.
- Install dependencies like Erlang, Elixir, development tools like GCC on CI server.
- Validate changes (means running specs, and all). Compile code to BEAM.
- Package compiled code, along with Erlang runtime.
- Ship the package to production server, unpack, and start the app.
Interesting thing to note here is, CI server needs source code of all dependencies of the Elixir app, but production server uses only the compiled BEAM code just to make it clear. So there is no bloat on production servers, it's easy to spin up new servers, unpack the package, and start the app.
Let's explore Distillery, and how it helps in deploying Phoenix apps. The steps to create and run the distillery based app are taken from Distillery documentation.
Create Phonenix app and initialize Distillery
Let's create a simple Phoenix app which we will use throughout our discussion.
> mix phoenix.new --no-ecto --no-brunch phoenix_app
* creating phoenix_app/config/config.exs
* creating phoenix_app/config/prod.secret.exs
* creating phoenix_app/config/test.exs
* ...
Fetch and install dependencies? [Yn] Y
* running mix deps.getWe are all set! Run your Phoenix application:
> cd phoenix_app
> mix phoenix.serverIf it's all good, you should be able to see the Phoenix app at localhost:4000
Now, open mix.exs and append distillery
defp deps do
[{:phoenix, "~> 1.2.1"},
{:phoenix_pubsub, "~> 1.0"},
{:phoenix_html, "~> 2.6"},
{:phoenix_live_reload, "~> 1.0", only: :dev},
{:gettext, "~> 0.11"},
{:cowboy, "~> 1.0"},
{:distillery, "~> 0.10.1"}]
end
Now, fetch dependencies using mix, and initialize repository with Distillery
> mix deps.get
> mix release.initThis should create a rel folder with config.exs file in it. Please take a
moment to go through this file which is well commented to understand what it
does.
Configuring CI and generating release pack
CI does all the heavy lifting in order to cut a release. You can run these commands locally and replicate the same on CI server. Make sure CI server has these packages installed:
- Erlang (Erlang solutions)
- Elixir
Cutting a release is very easy, just run this command:
> MIX_ENV=prod mix release --env=prod
==> gettext
Compiling 1 file (.erl)
...
==> Assembling release..
==> Building release phoenix_app:0.0.1 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Packaging release..
==> Release successfully built!
You can run it in one of the following ways:
Interactive: rel/phoenix_app/bin/phoenix_app console
Foreground: rel/phoenix_app/bin/phoenix_app foreground
Daemon: rel/phoenix_app/bin/phoenix_app startIf you look at the output of this command, you'll notice that mix is packaging
everything along with the app. Interpreting mix release output:
phoenix_app: Compiled files of all source code in deps folder along with source code of the Phoenix app.- ERTS: Erlang runtime which is required to run the server. Note that since mix is packaging runtime also, there is no need to install Erlang on production machine.
- Helpful commands to run Phoenix app on server.
Optionally you can set up Travis/Jenkins to observe features merged into git, automatically pulling latest source code, and packaging app
Now, take a look at rel/phoenix_app folder:
rel
└── phoenix_app
├── bin
│ ├── nodetool
│ ├── phoenix_app
│ ├── release_utils.escript
│ └── start_clean.boot
├── erts-8.1
│ ├── bin
│ │ ├── beam
│ │ ├── beam.smp
....
│ ├── include
│ │ ├── driver_int.h
│ │ ├── erl_nif.h
│ ├── lib
│ │ ├── internal
│ │ ├── liberts.a
│ │ └── liberts_r.a
├── lib
│ ├── compiler-7.0.2
│ ├── cowboy-1.0.4
│ ├── crypto-3.7.1
...
│ ├── phoenix-1.2.1
│ ├── phoenix_app-0.0.1
│ ├── ranch-1.2.1
└── releases
├── 0.0.1
│ ├── commands
│ ├── hooks
│ ├── phoenix_app.boot
│ ├── phoenix_app.rel
│ ├── phoenix_app.script
│ ├── phoenix_app.sh
│ ├── phoenix_app.tar.gz
│ ├── start_clean.boot
│ ├── sys.config
│ └── vm.args
├── RELEASES
└── start_erl.dataGoing through the top level folders:
erts: Erlang runtime, contains beam files, library files to run Erlanglib: all compiled dependencies ofphoenix_app, this folder resembles very similar todepsfolder at root levelreleases: Folder of our interest. By default, app version will be0.0.1. All the files required to runphoenix_app. Inside a release version folder, all files are packaged into a single.tar.gzfile. This file can be copied to production server for running the app.
Making successive deployments
Erlang has a rich heritage, and Erlang programs are designed to run for
years without bringing servers down, which guarantees nearly 100% up time.
Rolling out bug fixes, new features, improvements are done using hot
updates to servers. Erlang provides ways to patch existing running code
on production servers so that there is no need to stop and start the
app. Let's look at ways to deploy phoenix_app
Hot upgrades
distillery provides support to create releases which can be applied
as hot upgrades. The process of generating the tar file is same, but the
command is different. For the sake of brevity, change version number in
mix.exs from 0.0.1 to 0.0.2 before proceeding
> MIX_ENV=prod mix release --env=prod --upgrade
==> Assembling release..
==> Building release phoenix_app:0.0.2 using environment prod
==> Including ERTS 8.1 from /usr/local/Cellar/erlang/19.1/lib/erlang/erts-8.1
==> Generated .appup for phoenix_app 0.0.1 -> 0.0.2
==> Relup successfully created
==> Packaging release..
==> Release successfully built!
You can run it in one of the following ways:
Interactive: rel/phoenix_app/bin/phoenix_app console
Foreground: rel/phoenix_app/bin/phoenix_app foreground
Daemon: rel/phoenix_app/bin/phoenix_app startThe command to create a new release is same as that of first run, but
there is a new argument, i.e --upgrade. Also, output of command is also
slightly different. Generated .appup for phoenix_app 0.0.1 -> 0.0.2,
where .appup means hot upgrade. There will be new folder 0.0.2 under
releases. There will be another file called relup which contains
instructions about how to upgrade. It looks like this:
{"0.0.2",
[{"0.0.1",[],
[{load_object_code,{phoenix_app,"0.0.2",
['Elixir.PhoenixApp.Endpoint',
'Elixir.PhoenixApp.Gettext']}},
point_of_no_return,
{load,{'Elixir.PhoenixApp.Endpoint',brutal_purge,brutal_purge}},
{load,{'Elixir.PhoenixApp.Gettext',brutal_purge,brutal_purge}}]}],
[{"0.0.1",[],
[{load_object_code,{phoenix_app,"0.0.1",
['Elixir.PhoenixApp.Endpoint',
'Elixir.PhoenixApp.Gettext']}},
point_of_no_return,
{load,{'Elixir.PhoenixApp.Endpoint',brutal_purge,brutal_purge}},
{load,{'Elixir.PhoenixApp.Gettext',brutal_purge,brutal_purge}}]}]}.
This file contains instructions about how to switch between 0.0.1
and 0.0.2. If the upgrade needs to be rolled back, this file helps in
downgrading from 0.0.2 to 0.0.1.
Note about versions
Mix knows only semantic versioning. If one has to use SHA-IDs as the version,
mix will throw errors. Say, you change version from 0.0.2 to 48dcbccd,
and try to generate new package, mix throws this error:
> MIX_ENV=prod mix release --env=prod --upgrade
Compiling 11 files (.ex)
** (Mix) Expected :version to be a SemVer version, got: "48dcbccd"If you are coming from Rails world, and used to continuous deployments using Capistrano, editing version every time for deployments is a pain. Let's see how we can get Capistrano kind of continuous deployments.
Generating versions on the fly
One way to generate versions on the fly is to read version from environment variable.
Also, since mix enforces semantic versioning, generating incremental versions
makes sense. Capistrano generates folders with YYYYMMDDHHMMSS format, i.e.
year, month, date. Change mix.exs file to have version like this:
def project do
[app: :phoenix_app,
version: (if Mix.env == :prod, do: System.get_env("APP_VERSION"), else: "0.0.1"),
elixir: "~> 1.2",
elixirc_paths: elixirc_paths(Mix.env),
compilers: [:phoenix, :gettext] ++ Mix.compilers,
build_embedded: Mix.env == :prod,
start_permanent: Mix.env == :prod,
deps: deps()]
end
Now when mix is building package for production, the app read the version from
the environment variable, otherwise it's hard coded to 0.0.1. Since version
can be dynamic now, we can let CI specify what version needs to be generated.
Following how Capistrano generates versions, we can do
export APP_VERSION=`date +%Y.%m.%d%H%M`
npm install # for brunch
mix deps.get
MIX_ENV=prod mix release --env=prod --upgradeSo, this code generates the app version with year as major version, month as
minor version, and <date>``<hours>``<minutes> as patch version. Note that in
order for --upgrade to work on CI, you should have previous versions in
releases folder, otherwise upgrade will fail because there is no reference
release. If you are using Travis/Circle-CI, make sure that releases folder
is cached.
Recovering from failures, doing proper patch updates
Say code which is pushed in is buggy, and CI has made a release. It can happen that hot upgrade itself fails. In such cases, CI will have a stale releases like this:
rel/
phoenix_app/
releases/
0.0.1 # deploy succeeded
2016.09.250826 # deploy succeeded
2016.09.250829 # deploy failed
2016.09.251007 # new package based on failed deploy packageWhen CI runs again, it will generate a new package assuming previous package is a good one. Hot upgrades will start failing from now on!
It's very important to keep a note of which deploy succeeded, or which one failed. There are two ways this can be done:
- Maintain
revisions.logjust like how Capistrano keeps track of deploys. - Read which version is currently deployed directly from the production server itself.
If you follow 2nd approach, you can read current running version from the
file start_erl.data. It contains runtime version and app version. CI
can read that version from production server, and create a hot upgrade
from that version. In addition to --upgrade option, Distillery supports
--upfrom option also. This takes in app version from which upgrade
package has to be generated. Typically CI code looks like this:
export PREV_APP_VERSION=`ssh mix@10.0.0.5 cat /srv/app/releases/start_erl.data | cut -d' ' -f2`
export APP_VERSION=`date +%Y.%m.%d%H%M`
touch mix.exs # so that new app_version is picked
npm install # for brunch
mix deps.get
MIX_ENV=prod mix release --env=prod --upgrade --upfrom=$PREV_APP_VERSIONNow hot upgrades always work!
Immutable infrastructure
Immutable infra means - services in production have to be treated as immutable entities and upgrades should not change anything in production, but instead should be redeployed each time. This means no hot-upgrades are allowed in production. If you are running stateless services like web servers written in Phoenix, sticking with immutable infrastructure is recommended. Just create a package, spin up a new node behind load balancer, run the app, and nuke old nodes.
Compiling assets using plugins
NOTE: We haven't initialized our app to use brunch, so the following section
won't work. Try creating a new app without --no-brunch option, and
follow this section.
Mostly people use Phoenix for APIs, and use Reactjs or other JS framework for front-end. If you are reading through Distillery docs, it suggests to push assets compilation to shell script itself. Let's take a small detour and see how we can do that using plugins. A Distillery plugin can be used to hook into the package generation process. It has 5 hooks:
- before_assembly
- after_assembly
- before_package
- after_package
- after_cleanup
Names are pretty self-explanatory. If you want to know more about these, you can
take a look at them here. You can hook into before_assembly
block, and compile assets.
defmodule PhoenixApp.PhoenixDigestTask do
use Mix.Releases.Plugin
def before_assembly(%Release{} = _release) do
info "before assembly!"
case System.cmd("npm", ["run", "deploy"]) do
{output, 0} ->
info output
Mix.Task.run("phoenix.digest")
nil
{output, error_code} ->
{:error, output, error_code}
end
end
def after_assembly(%Release{} = _release) do
info "after assembly!"
nil
end
def before_package(%Release{} = _release) do
info "before package!"
nil
end
def after_package(%Release{} = _release) do
info "after package!"
nil
end
def after_cleanup(%Release{} = _release) do
info "after cleanup!"
nil
end
end
environment :prod do
set plugins: [PhoenixApp.PhoenixDigestTask]
set include_erts: true
set include_src: false
end
So, more Elixir code and less shell scripting. When you run release command again, output looks like this:
==> Assembling release..
==> Building release phoenix_app:2016.10.251007 using environment prod
==> before assembly! ## <- Plugin print here
==>
> brunch build --production
25 Oct 10:07:47 - info: compiled 3 files into 2 files, copied 3 in 3.1 sec
Check your digested files at "priv/static"
==> Including ERTS 8.0.2 from /usr/lib/erlang/erts-8.0.2
==> Generated .appup for phoenix_app 2016.09.261241 -> 2016.10.251007
==> Relup successfully created
==> after assembly! ## <- Plugin print here
==> Packaging release..
==> before package! ## <- Plugin print here
==> after package! ## <- Plugin print here
==> Release successfully built!
You can run it in one of the following ways:
Interactive: rel/phoenix_app/bin/phoenix_app console
Foreground: rel/phoenix_app/bin/phoenix_app foreground
Daemon: rel/phoenix_app/bin/phoenix_app startYou can see logs from the plugin throughout the deploy process. Plugins is an interesting concept, please take a look at that.
You can find whole source code here
Thanks for reading! If you would like to get updates about subsequent blog posts from Codemancers, do follows us on twitter: @codemancershq. Feel free to get in touch if you'd like Codemancers to help you build your Elixir/Phoenix app.