Imagine that we have the following Postgresql database dump:

--
-- PostgreSQL database dump
--

-- Dumped from database version 11.5
-- Dumped by pg_dump version 11.3

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

--
-- Name: my_db; Type: DATABASE; Schema: -; Owner: postgres
--

CREATE DATABASE my_db WITH TEMPLATE = template0 ENCODING = 'UTF8' LC_COLLATE = 'en_US.utf8' LC_CTYPE = 'en_US.utf8';

ALTER DATABASE my_db OWNER TO postgres;

\connect my_db

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

SET default_tablespace = '';

SET default_with_oids = false;

--
-- Name: clients; Type: TABLE; Schema: public; Owner: postgres
--

CREATE TABLE public.clients (
    id integer NOT NULL,
    name character varying(150) NOT NULL
);

ALTER TABLE public.clients OWNER TO postgres;

--
-- Name: clients_id_seq; Type: SEQUENCE; Schema: public; Owner: postgres
--

CREATE SEQUENCE public.clients_id_seq
    AS integer
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;

ALTER TABLE public.clients_id_seq OWNER TO postgres;

--
-- Name: clients_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: postgres
--

ALTER SEQUENCE public.clients_id_seq OWNED BY public.clients.id;

--
-- Name: clients id; Type: DEFAULT; Schema: public; Owner: postgres
--

ALTER TABLE ONLY public.clients ALTER COLUMN id SET DEFAULT nextval('public.clients_id_seq'::regclass);

--
-- Data for Name: clients; Type: TABLE DATA; Schema: public; Owner: postgres
--

COPY public.clients (id, name) FROM stdin;
1   Client 1
2   Client 2
\.

--
-- Name: clients_id_seq; Type: SEQUENCE SET; Schema: public; Owner: postgres
--

SELECT pg_catalog.setval('public.clients_id_seq', 2, true);

--
-- Name: clients clients_pkey; Type: CONSTRAINT; Schema: public; Owner: postgres
--

ALTER TABLE ONLY public.clients
    ADD CONSTRAINT clients_pkey PRIMARY KEY (id);

--
-- PostgreSQL database dump complete
--

It is a simple database with a Clients table and 2 records.

If we want to start a Postgresql Docker container with this dump loaded to share with our team, we can add this SQL file into the /docker-entrypoint-initdb.d/ folder inside the container, like explained into the Postgresql Image docs from DockerHub.

Initialization scripts If you would like to do additional initialization in an image derived from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under /docker-entrypoint-initdb.d (creating the directory if necessary). After the entrypoint calls initdb to create the default postgres user and database, it will run any *.sql files, run any executable *.sh scripts, and source any non-executable *.sh scripts found in that directory to do further initialization before starting the service.

The following Dockerfile uses postgres:11-alpine as base image and copies test_dump.sql file to the entrypoint folder.

FROM postgres:11-alpine

COPY test_dump.sql /docker-entrypoint-initdb.d/

If we build this image

$ docker image build . -t preloaded_db:latest

and start a container with the generated image

$ docker container run -d --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --name test_preloaded_db preloaded_db:latest

we can see in our database that the database was created. (password is postgres)

$ psql -h localhost -U postgres
postgres=# \c my_db
psql (11.3, server 11.5)
You are now connected to database “my_db” as user “postgres”.
my_db=# SELECT * FROM clients;
 id | name
 — — + — — — — —
 1 | Client 1
 2 | Client 2
(2 rows)

Awesome. Now we have a docker image that has our database loaded. But if we check the log of this container

$ docker container logs test_preloaded_db

we can see CREATE DATABASE and CREATE TABLE commands.

/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/test_dump.sql
SET
SET
SET
SET
SET
 set_config
-----------------

(1 row)

SET
SET
SET
SET
CREATE DATABASE
ALTER DATABASE
...

This tell us that the dump is being processed every time we create the container. If we destroy this container and create a new one, the dump will be processed again. This works fine but if we have a big database with a big dump file, the startup process of the container will be slow because it can take some time to process the whole dump. We can fix it by keeping the database preloaded in the image.

Before we moving on, let’s destroy the container we created

$ docker container rm -f test_preloaded_db

Preloading the database in the image

To preload the database in the image, we need to tell our Dockerfile to execute the same entrypoint of the original PostgreSQL image so it can execute the dump in the build step. Let's use Multi-Stage build to divide our build in two steps. The first one will execute the entrypoint with the dump file and the second one will copy the data folder to the resulting image.

# dump build stage
FROM postgres:11-alpine as dumper

COPY test_dump.sql /docker-entrypoint-initdb.d/

RUN ["sed", "-i", "s/exec \"$@\"/echo \"skipping...\"/", "/usr/local/bin/docker-entrypoint.sh"]

ENV POSTGRES_USER=postgres
ENV POSTGRES_PASSWORD=postgres
ENV PGDATA=/data

RUN ["/usr/local/bin/docker-entrypoint.sh", "postgres"]

# final build stage
FROM postgres:11-alpine

COPY --from=dumper /data $PGDATA

In the first step, we have the following instructions:

• FROM postgres:11-alpine as dumper We define the base image our step will use. postgres with the 11-alpine tag in this case.
• COPY test_dump.sql /docker-entrypoint-initdb.d/ Copy the test_dump.sql file to the /docker-entrypoint-initdb.d/ folder.
• RUN [“sed”, “-i”, “s/exec “$@”/echo “skipping…”/”, “/usr/local/bin/docker-entrypoint.sh”] We need to execute this sed command in order to remove the exec "$@" content that exists in the docker-entrypoint.sh file so it will not start the PostgreSQL daemon (we don't need it on this step).
• ENV POSTGRES_USER=postgres; ENV POSTGRES_PASSWORD=postgres; ENV PGDATA=/data Sets environment variables to define user and password and tell PostgreSQL to use /data as data folder, so we can copy it in the next step
• RUN [“/usr/local/bin/docker-entrypoint.sh”, “postgres”] Execute the entrypoint itself. It will execute the dump and load the data into /data folder. Since we executed the sed command to remove the $@ content it will not run the PostgreSQL daemon

The second step contains only this instruction:

If we build this Dockerfile

$ docker image build . -t preloaded_db:new

We can see in the output the dump being processed and after everything is finished, the image is built.

and we can start the container with this new image

$ docker container run -d --rm -p 5432:5432 -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres --name test_preloaded_db preloaded_db:latest

and our database is loaded

$ psql -h localhost -U postgres
psql (11.3, server 11.5)
Type “help” for help.
postgres=# \c my_db
psql (11.3, server 11.5)
You are now connected to database “my_db” as user “postgres”.
my_db=# SELECT * FROM clients;
 id | name
 — — + — — — — —
 1 | Client 1
 2 | Client 2
(2 rows)

But if we check the logs now, the dump is not being processed every time we create the container

$ docker container logs test_preloaded_db
2019–09–16 01:42:22.458 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2019–09–16 01:42:22.458 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2019–09–16 01:42:22.460 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2019–09–16 01:42:22.470 UTC [18] LOG: database system was shut down at 2019–09–16 01:41:02 UTC
2019–09–16 01:42:22.473 UTC [1] LOG: database system is ready to accept connections

We can see that only the PostgreSQL startup is being done. No dump is being executed because it was executed in the build image step.

Creating a Makefile to make the process easier

I like to create a Makefile to make easier the process of making a database dump and creating an image. This Makefile will contain commands to create the dump the database, create an image and tag it by date allowing me to have daily dumps on my registry to download.

default: all

.PHONY: default all fetch_dump

date := `date '+%Y-%m-%d'`
TARGET_IMAGE ?= my_app

all: check_vars fetch_dump generate_image push_to_registry clean finished

check_vars:
    @test -n "$(DB_ENDPOINT)" || (echo "You need to set DB_ENDPOINT environment variable" >&2 && exit 1)
    @test -n "$(DB_NAME)" || (echo "You need to set DB_NAME environment variable" >&2 && exit 1)
    @test -n "$(DESTINATION_REPOSITORY)" || (echo "You need to set DESTINATION_REPOSITORY environment variable" >&2 && exit 1)

fetch_dump: DB_USER ?= postgres
fetch_dump:
    @echo ""
    @echo "====== Fetching remote dump ======"
    @PGPASSWORD="$(DB_PASSWORD)" pg_dump -h $(DB_ENDPOINT) -d $(DB_NAME) -U $(DB_USER) > dump.sql

generate_image:
generate_image:
    @docker build . -t $(TARGET_IMAGE):latest -t $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE):latest -t $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE):$(date)

push_to_registry:
    @echo ""
    @echo "====== Pushing image to repository ======"
    @docker push $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE)

clean:
    @echo ""
    @echo "====== Cleaning used files ======"
    @rm -f dump.sql

finished:
    @echo ""
    @echo "Finished with success. Pushed image to $(DESTINATION_REPOSITORY)/$(TARGET_IMAGE)"

And I can execute the following command to generate my image with a new dump

$ make DB_ENDPOINT=127.0.0.1 DB_USER=postgres DB_PASSWORD=postgres DB_NAME=my_db TARGET_IMAGE=myapp-data DESTINATION_REPOSITORY=gcr.io/my_project

This command usually is integrated in a Cron job in some server to be executed daily. With this I can have on my image registry dumps from each day.

Another interesting thing to do is to add some SQL script to obfuscate users data. This article can be helpful if you want to achive this

Thanks ☕️

Originally published at https://cadu.dev on January 29, 2021.

Imagine que você possua o seguinte dump de uma base de dados Postgres.

Uma base simples com uma tabela Clients que contém 2 registros.

Se desejarmos iniciar um container Docker Postgres com essa base já pré-carregada para poder distribuir para as pessoas do time, podemos colocar esse arquivo sql na pasta /docker-entrypoint-initdb.d/ dentro do container, como informado na documentação da página da imagem no DockerHub:

Initialization scripts
If you would like to do additional initialization in an image derived from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under /docker-entrypoint-initdb.d (creating the directory if necessary). After the entrypoint calls initdb to create the default postgres user and database, it will run any *.sql files, run any executable *.sh scripts, and source any non-executable *.sh scripts found in that directory to do further initialization before starting the service.

O seguinte Dockerfile usa a imagem postgres:11-alpine como base e realiza a cópia do arquivo test_dump.sql para a pasta indicada:

Se realizarmos o build desse arquivo:

$ docker image build . -t preloaded_db:latest

E inicializarmos um container da imagem gerada:

$ docker container run -d --rm -p 5432:5432 -e POSTGRES_USER=postgres --name test_preloaded_db preloaded_db:latest

Podemos verificar no client do Postgres que o banco foi criado e inicializado:

$ psql -h localhost -U postgres

postgres=# \c my_db
psql (11.3, server 11.5)
You are now connected to database “my_db” as user “postgres”.
my_db=# SELECT * FROM clients;
 id | name
 — — + — — — — — 
 1 | Client 1
 2 | Client 2
(2 rows)

Se analisarmos o log desse container:

$ docker container logs test_preloaded_db

Vemos vários comandos do Postgres sendo executados. O que nos indica que o dump está sendo processado ao criar o container.

Ok. Nosso banco está sendo carregado, porém o dump está sendo executado toda vez que o container é criado. Se destruirmos o container e criarmos outro, o arquivo .sql vai ser executado novamente. Essa abordagem é ok, porém se tivermos um dump muito grande, a inicialização do container é mais lenta, pois a execução do dump pode demorar para ser completada. Para resolvermos isso, podemos gerar uma imagem com a base de dados já pré carregada na imagem

Antes de continuarmos, vamos parar o container que criamos:

$ docker container rm -f test_preloaded_db

Pré-carregando o banco na imagem

Para pré-carregar a imagem, precisamos fazer nosso Dockerfile executar o mesmo entrypoint da imagem original do Postgres, para ele realizar a inicialização do banco no processo de build. Vamos utilizar Multi-Stage build para dividirmos o build em duas etapas. A primeira irá executar o entrypoint com o arquivo de dump, e a segunda irá copiar a pasta de dados do Postgres para a imagem final

No primeira etapa temos as seguintes instruções:
- FROM postgres:11-alpine as dumper: Definimos a imagem base que nossa etapa irá usar. Nesse caso, imagem `postgres` com a tag `11-alpine`.

- COPY test_dump.sql /docker-entrypoint-initdb.d/: Copiamos o arquivo `test_dump.sql`, que contém o dump do nosso banco de dados para pasta `/docker-entrypoint-initdb.d/`

- RUN ["sed", "-i", "s/exec \"$@\"/echo "skipping…\"/", "/usr/local/bin/docker-entrypoint.sh"]: Nessa linha, executamos o sed para trocar o conteúdo exec "$@" para skipping… do arquivo /usr/local/bin/docker-entrypoint.sh. Fazendo isso, quando executarmos o entrypoint, ele irá carregar o script sql da pasta porém não irá iniciar o daemon do Postgres, pois não precisamos. Só queremos que ele faça a pré-carga dos dados para podermos copiar para a etapa posterior.

- ENV PG_USER=postgres ; ENV PGDATA=/data: Nessas duas linhas configuramos duas variáveis de ambientes. Uma para definir que o usuário será o `postgres` e outra para informar que a pasta de dados do postgres será `/data`.

- RUN ["/usr/local/bin/docker-entrypoint.sh", "postgres"]: Aqui executamos o entrypoint em si. Ele irá executar o dump e carregar os dados na pasta `/data`. E como executamos o`sed` para remover o `$@`, ele não irá executar o daemon do postgres.

Todas essas alterações de arquivos e variáveis de ambientes só ficam disponíveis nessa etapa de build. Na próxima, como veremos, a etapa vai se basear novamente da imagem base do Postgres, sem nenhuma configuração modificada que fizemos nessa etapa.

Na segunda etapa temos apenas a seguinte instrução:

- COPY — from=dumper /data $PGDATA: Essa instrução copia todos os arquivos da pasta `/data` da etapa `dumper` para a pasta `$PGDATA` dessa etapa, fazendo nossos dados do dump serem já carregados na imagem.

Se realizarmos um build desse Dockerfile:

$ docker image build . -t preloaded_db:new

Vemos no output que ele executa o script sql e gera a imagem.

Se criarmos um container com essa imagem:

$ docker container run -d --rm -p 5432:5432 -e POSTGRES_USER=postgres --name test_preloaded_db preloaded_db:new

Vemos que nosso banco de dados está carregado:

$ psql -h localhost -U postgres

psql (11.3, server 11.5)
Type “help” for help.

postgres=# \c my_db
psql (11.3, server 11.5)
You are now connected to database “my_db” as user “postgres”.
my_db=# SELECT * FROM clients;
 id | name
 — — + — — — — — 
 1 | Client 1
 2 | Client 2
(2 rows)

E ao analisar os logs, vemos que o dump não é executado ao criar o container:

$ docker container logs test_preloaded_db

2019–09–16 01:42:22.458 UTC [1] LOG: listening on IPv4 address “0.0.0.0”, port 5432
2019–09–16 01:42:22.458 UTC [1] LOG: listening on IPv6 address “::”, port 5432
2019–09–16 01:42:22.460 UTC [1] LOG: listening on Unix socket “/var/run/postgresql/.s.PGSQL.5432”
2019–09–16 01:42:22.470 UTC [18] LOG: database system was shut down at 2019–09–16 01:41:02 UTC
2019–09–16 01:42:22.473 UTC [1] LOG: database system is ready to accept connections

Vemos que apenas a inicialização do Postgres é feita. Não há nenhum comando da execução do dump em si, isso foi realizado no build da imagem.

Criando um Makefile para facilitar o processo

Para facilitar o processo de dump e criação da imagem customizada, eu gosto de criar um Makefile que contém uma série de passos para criar a imagem e tagear ela pela data, me possibilitando ter dumps diários no meu registry para poder baixar.

E eu consigo executar o seguinte comando para gerar a imagem com o dump presente (apenas trocando as variáveis para a configuração certa)

make DB_ENDPOINT=127.0.0.1 DB_USER=postgres DB_PASSWORD=password DB_NAME=my_db TARGET_IMAGE=myapp DESTINATION_REPOSITORY=gcr.io/my_project

Esse comando é integrado em um Cron para ser executado diariamente, então eu tenho em meu repositório imagens com dump de Dia - 1.

Outro ponto interessante é você colocar mais um script SQL para ser executado que irá realizar a ofuscação dos dados dos usuários. Esse artigo pode ser interessante para saber mais sobre.

Thanks ☕️

Criando uma imagem Docker com o banco de dados pré carregado was originally published in Opensanca on Medium, where people are continuing the conversation by highlighting and responding to this story.

(Portuguese version here)

A big problem that we face when deploying Docker into production is the image size. Large images take longer to download, consume much of your cloud network traffic quota, cost more money to be stored on the repository and don’t bring any good value.

In most situations, when we create a Docker image, we add steps and dependencies that sometimes we don’t need in the final image that will run in production.

I will use the following application as an example:
https://github.com/opensanca/opensanca_jobs

This is the Dockerfile that generates our image

FROM ruby:2.5.0-alpine
LABEL maintainer="contato@opensanca.com.br"
ARG rails_env="development"
ARG build_without=""
ENV SECRET_KEY_BASE=dumb
RUN apk update \
&& apk add \
openssl \
tar \
build-base \
tzdata \
postgresql-dev \
postgresql-client \
nodejs \
&& wget https://yarnpkg.com/latest.tar.gz \
&& mkdir -p /opt/yarn \
&& tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
&& mkdir -p /var/app
ENV PATH="$PATH:/opt/yarn/bin" BUNDLE_PATH="/gems" BUNDLE_JOBS=2 RAILS_ENV=${rails_env} BUNDLE_WITHOUT=${bundle_without}
COPY . /var/app
WORKDIR /var/app
RUN bundle install && yarn && bundle exec rake assets:precompile
CMD rails s -b 0.0.0.0

And the command used to build it:

docker build -t openjobs:latest --build-arg build_without="development test" --build-arg rails_env="production" .

This build generated an image with almost 1GB!!! 😱.

This image has some unnecessary stuff, like node but yarn (we only need them to precompile the assets but not to execute the application itself).

Multi-Stage build

Docker introduced the concept of Multi-Stage build in version 17.05. This build technic allows us to split our Dockerfile into several statements FROM. Each statement can use a different base image and you can copy artifacts from one stage to another, without bringing stuff that you don’t want in the final image. Our final image will only contain the build wrote in the last stage.

Now we have a Dockerfile divided into two stages. Pre-build and Final-Build.

# pre-build stage
FROM ruby:2.5.0-alpine AS pre-builder
ARG rails_env="development"
ARG build_without=""
ENV SECRET_KEY_BASE=dumb
RUN apk add --update --no-cache \
openssl \
tar \
build-base \
tzdata \
postgresql-dev \
postgresql-client \
nodejs \
&& wget https://yarnpkg.com/latest.tar.gz \
&& mkdir -p /opt/yarn \
&& tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
&& mkdir -p /var/app
ENV PATH="$PATH:/opt/yarn/bin" BUNDLE_PATH="/gems" BUNDLE_JOBS=2 RAILS_ENV=${rails_env} BUNDLE_WITHOUT=${bundle_without}
COPY . /var/app
WORKDIR /var/app
RUN bundle install && yarn && bundle exec rake assets:precompile

# final build stage
FROM ruby:2.5.0-alpine
LABEL maintainer="contato@opensanca.com.br"
RUN apk add --update --no-cache \
openssl \
tzdata \
postgresql-dev \
postgresql-client
COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app
ENV RAILS_LOG_TO_STDOUT true
WORKDIR /var/app
EXPOSE 3000
CMD rails s -b 0.0.0.0

In the pre-build stage we install node and yarn, all dependencies and precompile the assets. In the final stage, we use an alpine image (which is very small) with ruby, we install only the necessary dependencies to run the application and we then copy the libraries and assets generated in the build-stage with the following command:

COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app

Doing the build with this Dockerfile, we have now a 562MB image.

We have already reduced almost half the image size, but can we reduce it further?? 🤔

Yes. We can do some actions to reduce more this image.

Removing unnecessary files

We can delete files that are not necessary from the image, like cache and temporary files used by the installed libraries. We can add a .dockerignore file, telling the build what not to send to the image.

# build stage
FROM ruby:2.5.0-alpine AS pre-builder
ARG rails_env="development"
ARG build_without=""
ENV SECRET_KEY_BASE=dumb
RUN apk add --update --no-cache \
openssl \
tar \
build-base \
tzdata \
postgresql-dev \
postgresql-client \
nodejs \
&& wget https://yarnpkg.com/latest.tar.gz \
&& mkdir -p /opt/yarn \
&& tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
&& mkdir -p /var/app
ENV PATH="$PATH:/opt/yarn/bin" BUNDLE_PATH="/gems" BUNDLE_JOBS=4 RAILS_ENV=${rails_env} BUNDLE_WITHOUT=${bundle_without}
COPY . /var/app
WORKDIR /var/app
RUN bundle install && yarn && bundle exec rake assets:precompile \
&& rm -rf /gems/cache/*.gem \
&& find /gems/gems/ -name "*.c" -delete \
&& find /gems/gems/ -name "*.o" -delete

# final stage
FROM ruby:2.5.0-alpine
LABEL maintainer="contato@opensanca.com.br"
RUN apk add --update --no-cache \
openssl \
tzdata \
postgresql-dev \
postgresql-client
COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app
ENV RAILS_LOG_TO_STDOUT true
WORKDIR /var/app
EXPOSE 3000
CMD rails s -b 0.0.0.0

In this new Dockerfile, we added this part that removes caches and temporary C files used to build the libraries:

&& rm -rf /gems/cache/*.gem \
&& find /gems/gems/ -name "*.c" -delete \
&& find /gems/gems/ -name "*.o" -delete

We also included our .dockerignore to tell the build process the files that we don’t want in the image:

.env*
.git
.gitignore
.codeclimate.yml
.dockerignore
.gitlab-ci.yml
.hound.yml
.travis.yml
LICENSE.md
README.md
docker-compose.*
Dockerfile
log/*
node_modules/*
public/assets/*
storage/*
public/packs/*
public/packs-test/*
tmp/*

With these two steps, now our image has 272MB.

We can reduce it even more. For production, we don’t need test folders, npm raw folder (they are already included on the asset pipeline), no precompiled assets and caches.

To remove this files, we can include a strategy of passing an argument to build (we will call it: to_remove)

...
ARG to_remove
...
RUN bundle install && yarn && bundle exec rake assets:precompile  \
&& rm -rf /usr/local/bundle/cache/*.gem \
 && find /usr/local/bundle/gems/ -name "*.c" -delete \
 && find /usr/local/bundle/gems/ -name "*.o" -delete \
 && rm -rf $to_remove   # Here we remove all files that we passed as an argument to the build.
...

In this argument, we will pass all the files that we don’t want in production:

docker build -t openjobs:reduced --build-arg build_without="development test" --build-arg rails_env="production" . --build-arg to_remove="spec node_modules app/assets vendor/assets lib/assets tmp/cache"

Notice the `— build-arg to_remove=”spec node_modules app/assets vendor/assets lib/assets tmp/cache”`. These are the folders that we want to remove from our build process. We don’t need them to run in production.

Removing these files, now we have an image with 164MB, almost 6 times smaller than the original one.

If you still don’t believe me and want to see it, this is the PR that generates this reduction: https://github.com/opensanca/opensanca_jobs/pull/164

Cheers 🍻

Reduce your Docker images (an example with Ruby) was originally published in Code Pub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Um grande problema de fazer deploy em produção utilizando docker é tamanho que as imagens geradas podem ocupar. Imagens muito grandes demoram para ser baixadas, consomem parte da sua cota de tráfego de rede no seu provedor de nuvem, custam para ser guardadas no seu repositório e não trazer nenhuma valor adicional.

Na maioria das situações, quando criamos uma imagem docker, colocamos alguns passos e dependências que as vezes não precisamos na imagem final que iremos rodar em produção

Vou usar como exemplo a seguinte aplicação: https://github.com/opensanca/opensanca_jobs

e esse é o Dockerfile que nos gera a imagem.

FROM ruby:2.5.0-alpine

LABEL maintainer="contato@opensanca.com.br"

ARG rails_env="development"
ARG build_without=""

ENV SECRET_KEY_BASE=dumb

RUN apk update \
  && apk add \
    openssl \
    tar \
    build-base \
    tzdata \
    postgresql-dev \
    postgresql-client \
    nodejs \
  && wget https://yarnpkg.com/latest.tar.gz \
  && mkdir -p /opt/yarn \
  && tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
  && mkdir -p /var/app

ENV PATH="$PATH:/opt/yarn/bin" BUNDLE_PATH="/gems" BUNDLE_JOBS=2 RAILS_ENV=${rails_env} BUNDLE_WITHOUT=${bundle_without}

COPY . /var/app
WORKDIR /var/app

RUN bundle install && yarn && bundle exec rake assets:precompile
CMD rails s -b 0.0.0.0

E o comando utilizado para realizar o build:

docker build -t openjobs:latest --build-arg build_without="development test" --build-arg rails_env="production" .

Esse build nos gerou uma imagem de quase 1GB!!! 😱.

Essa imagem tem algumas coisas desnecessárias, como o node e o yarn (só precisamos deles para precompilar os assets e não para a execução da aplicação).

Multi-Stage build

Na versão 17.05 o docker introduziu o conceito de Multi-stage Builds. Essa técnica de build nos permite dividir o nosso Dockerfile em várias declarações FROM. Cada declaração pode usar imagens base diferentes e você pode copiar artefatos de um estágio para outro, sem trazer as coisas que você não quer na imagem final. Nossa imagem final só contará com o build escrito no ultimo estágio.

Nosso Dockerfile separado em dois estágios: Pré Build e Final Build

# pre-build stage
FROM ruby:2.5.0-alpine AS pre-builder

ARG rails_env="development"
ARG build_without=""

ENV SECRET_KEY_BASE=dumb

RUN apk add --update --no-cache \
    openssl \
    tar \
    build-base \
    tzdata \
    postgresql-dev \
    postgresql-client \
    nodejs \
  && wget https://yarnpkg.com/latest.tar.gz \
  && mkdir -p /opt/yarn \
  && tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
  && mkdir -p /var/app

ENV PATH="$PATH:/opt/yarn/bin" BUNDLE_PATH="/gems" BUNDLE_JOBS=2 RAILS_ENV=${rails_env} BUNDLE_WITHOUT=${bundle_without}

COPY . /var/app
WORKDIR /var/app

RUN bundle install && yarn && bundle exec rake assets:precompile

# final build stage
FROM ruby:2.5.0-alpine
LABEL maintainer="contato@opensanca.com.br"

RUN apk add --update --no-cache \
    openssl \
    tzdata \
    postgresql-dev \
    postgresql-client

COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app

ENV RAILS_LOG_TO_STDOUT true

WORKDIR /var/app

EXPOSE 3000

CMD rails s -b 0.0.0.0

No estágio de build, instalamos o node e o yarn, instalamos as dependências e precompilamos os assets. No estágio final, utilizamos uma imagem alpine (que é bem leve) com ruby, instalamos apenas dependências necessárias para rodar a aplicação e copiamos as bibliotecas e assets gerados no estágio anterior com o seguinte comando:

COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app  

Fazendo o build desse Dockerfile, agora temos uma imagem de 562mb.

Já diminuímos quase pela metade, mas será que ainda podemos reduzir ainda mais o tamanho da imagem?? 🤔

Sim. Podemos fazer algumas outras ações para reduzir ainda mais essa imagem.

Removendo arquivos desnecessários

Podemos apagar da imagem arquivos que não são necessários, como cache e os arquivos utilizados temporariamente das bibliotecas instaladas. E também podemos adicionar um arquivo .dockerignore, dizendo para o build o que não enviar para a imagem.

# build stage
FROM ruby:2.5.0-alpine AS pre-builder

ARG rails_env="development"
ARG build_without=""

ENV SECRET_KEY_BASE=dumb

RUN apk add --update --no-cache \
    openssl \
    tar \
    build-base \
    tzdata \
    postgresql-dev \
    postgresql-client \
    nodejs \
  && wget https://yarnpkg.com/latest.tar.gz \
  && mkdir -p /opt/yarn \
  && tar -xf latest.tar.gz -C /opt/yarn --strip 1 \
  && mkdir -p /var/app

ENV PATH="$PATH:/opt/yarn/bin" BUNDLE_PATH="/gems" BUNDLE_JOBS=4 RAILS_ENV=${rails_env} BUNDLE_WITHOUT=${bundle_without}

COPY . /var/app
WORKDIR /var/app

RUN bundle install && yarn && bundle exec rake assets:precompile \
 && rm -rf /gems/cache/*.gem \
 && find /gems/gems/ -name "*.c" -delete \
 && find /gems/gems/ -name "*.o" -delete

# final stage
FROM ruby:2.5.0-alpine
LABEL maintainer="contato@opensanca.com.br"

RUN apk add --update --no-cache \
    openssl \
    tzdata \
    postgresql-dev \
    postgresql-client

COPY --from=pre-builder /gems/ /gems/
COPY --from=pre-builder /var/app /var/app

ENV RAILS_LOG_TO_STDOUT true

WORKDIR /var/app

EXPOSE 3000

CMD rails s -b 0.0.0.0

Nesse novo Dockerfile, adicionamos esse trecho que remove caches e arquivos C temporários utilizados para buildar as bibliotecas.:

 && rm -rf /gems/cache/*.gem \
 && find /gems/gems/ -name "*.c" -delete \
 && find /gems/gems/ -name "*.o" -delete

Além de incluir tambem o nosso .dockerignore para informar ao processo de build os arquivos que não precisam ser enviados para a imagem:

.env*
.git
.gitignore
.codeclimate.yml
.dockerignore
.gitlab-ci.yml
.hound.yml
.travis.yml
LICENSE.md
README.md
docker-compose.*
Dockerfile
log/*
node_modules/*
public/assets/*
storage/*
public/packs/*
public/packs-test/*
tmp/*

Com esses dois passos, agora nossa imagem tem 272MB.

Podemos dar um passo a mais. Para produção, não precisamos das pastas de teste, dependências do npm (pois elas já foram incluídas no asset pipeline), assets não precompilados, e caches.

Pra remover esses arquivos, podemos incluir uma estratégia de passar um argumento para o build, que iremos chamar de to_remove .

...

ARG to_remove

...

RUN bundle install && yarn && bundle exec rake assets:precompile  \
&& rm -rf /usr/local/bundle/cache/*.gem \
 && find /usr/local/bundle/gems/ -name "*.c" -delete \
 && find /usr/local/bundle/gems/ -name "*.o" -delete \
 && rm -rf $to_remove   # Aqui removemos todos arquivos que passamos como argumento no build

...

Nesse argumento, passaremos os arquivos que não desejamos para produção:

docker build -t openjobs:reduced --build-arg build_without="development test" --build-arg rails_env="production" . --build-arg to_remove="spec node_modules app/assets vendor/assets lib/assets tmp/cache"

Perceba o --build-arg to_remove="spec node_modules app/assets vendor/assets lib/assets tmp/cache" Essas são as pastas que iremos remover do nosso processo de build, pois não necessitamos delas para rodar em produção.

Removendo esses arquivos, ficamos com uma imagem de 164MB, quase 6 vezes menor que a original.

Se você ainda não acreditou em mim e quer ver com seus próprios olhos, deixo aqui o pull request pra alteração que gerou essa redução em um projeto que estou trabalhando: https://github.com/opensanca/opensanca_jobs/pull/164

Cheers 🍻

Reduzindo o tamanho de suas imagens Docker (exemplo c/ Ruby) was originally published in Opensanca on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’m a Vim user. I use it in my workflow all the time, but I enjoy trying other editors sometimes and I will share my experience playing with Spacemacs.

What is Spacemacs?

Spacemacs is a starter kit for Emacs that makes it easier for anyone to use it. It has the Evil plugin by default. Evil is a layer for Emacs that emulates Vim. I always try vim mode in all other editors and they all are not so good, Evil is one of the best Vim emulators that I’ve already seen. It makes it easier for me to use Emacs, since it emulates Vim commands that I already know.

Spacemacs has easy to remember keybindings. Everything starts with the SPC key (it is like the leader key from Vim) and all belongs to some group. For example: SPC b for all buffers commands and SPC p for all project commands. If you don’t remember the command, press SPC SPC and it brings an autocomplete buffer to search for commands.

How do I experiment another editor?

It’s hard for me to learn all the stuffs from a tool in a day or two. So when I want to learn more about an editor, I force myself to experiment it for at least one week, trying to do everything on the new editor without opening others. To use an editor I need to learn how to search files, how to navigate through files and how split the window, only this.

Installation

First, you should install Emacs.
You can use Homebrew to install emacs on mac:

brew install emacs --use-git-head --cocoa --srgb

To install on Linux, you can follow this documentation.
Now you can install the Spacemacs distribution:

git clone https://github.com/syl20bnr/spacemacs ~/.emacs.d

Now open Emacs..

The Basics

Spacemacs has the concept oflayers . The idea is to have a collection of packages with some configurations. The documentation says:

Layer: A collected unit of configuration that can be enabled (or disabled) in Spacemacs. A layer typically brings together one or more packages, as well as the glue configuration code required to make them play well with each other and Spacemacs in general.

Once you open Emacs, enter SPC f e d, this will take you to Spacemacs configuration file (~/.spacemacs). Here you can add layers inside dotspacemacs-configuration-layers. For example:

dotspacemacs-configuration-layers
   '(
     elixir
     (ruby :variables ruby-version-manager 'rvm)
     ruby-on-rails
     )

This adds layers for working with Elixir, Ruby and Ruby on Rails

Navigating through a file

To navigate on the file, I use the Vim commands.

gg  => goes to the top of the file
G   => goes to the bottom
h   => left
j   => down
k   => up
l   => right

(to learn more about navigation, read here)

Loading your project

After loaded, you can use SPC a d (or File -> Open Directory) to open your project.

It will load this buffer and you can find your project file. Hit Enter to load it

Spacemacs includes Projectile, an Emacs library for handling projects. So, after you added your project the first time with SPC a d , you can use SPC p p to Switch between projects

Terminology

I like to use this image to understand Emacs terminology

Usually in most applications we call the running instance as window, but on Emacs we call it the Frame . A frame can contain multiple Windows and inside each window we have a Buffer loaded (on the image above, the buffer loaded is indicated by the yellow circle). A buffer can be the content of a loaded file or the output of some command execution. Every time you open a file, it changes the active buffer to this file, but the buffer with the past file opened is still alive. You can use SPC b b to see all buffers and switch them

You can easily switch between the last used buffer with SPC TAB

Searching for a file

The first thing that I learn in a new editor is: How can I search for a file?

In Spacemacs we can do a fuzzy file search pressing SPC p f

Another way is to search for Classes and methods using Tags. SPC p G will generate a TAGS file for your project. After generated, you can search through tags with SPC p g

You can also open a tree explorer using SPC p t

Splitting window

Another feature that I use a lot is splitting the window. You can use the Emacs keybinding to do it:

SPC w -   => Split window into two windows, one above the other.
SPC w /   => Split window into two windows, side by side.

Or using the vim commands:

:sp    => Split window into two windows, one above the other.
:vsp   => Split window into two windows, side by side.

After splitting, you will notice that a number is defined in the window

To jump into a window, you can press SPC NUMBER to go to that window (for example, SPC 3 take me to the right window on the image)

You can kill a window with SPC w d or :q .

Sometimes when working on more than one window in the same frame (for example, having a class and a test file opened together), I like to maximize one of them. You can use SPC w m to maximize and use it again to return it.

My Workflow working with it

Usually, my workflow with Vim involves Tmux too. I typically have 3 Tmux panes open. One for the editor (Vim), another for the terminal (to run tests, ssh, etc), and one for the server (and the logs).

I also like to use vim-test to run my tests inside Vim

Emacs for another side, have the idea that you can do everything on the Emacs itself. You can start your server on it, open a terminal, etc, without having the needing to have multiple terminals for each thing (although I still have they open when working on Emacs 😄)

My next examples use the rails layer on Spacemacs.

I can run bundle install (to install my ruby dependencies) inside it with SPC m b i

I can run the rails server inside it and check the logs:

and also run the tests directly from Emacs:

Follow a simple mindmap that I created for these basic commands:

And this is all the basics that I need to use it and that I want to share.

Cheers 🍻

An Introduction into Spacemacs from a Vim user was originally published in Code Pub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Publicado originalmente em thecode.pub em 31 de janeiro de 2018.

Nesse post irei demonstrar como você pode fazer o Deploy de sua aplicação Docker na AWS usando ECS e Fargate.

Como exemplo, eu irei fazer o deploy da aplicação do Openjobs. O código fonte pode ser encontrado nesse link.

Eu irei usar o Terraform para criar a infraestrutura, assim consigo controlar todos os recursos criados como código. Se você deseja saber o básico sobre Terraform, leia meu post sobre (inglês).

ECS

O que é ECS?

O Elastic Container Service (ECS) é um serviço da AWS que cuida da orquestração de containers Docker no seu cluster de instâncias EC2. É uma alternativa para o Kubernetes, Docker Swarm, e outros.

ECS Terminology

Para entender o que é o ECS, nós precisamos entender seus termos e definições, que diferem um pouco do mundo Docker.

• Cluster: É um group de instâncias EC2 que irão hospedar os containers.
• Task definition: É uma especificação de como o ECS deve rodar sua aplicação. Aqui se define qual imagem usar, mapeamento de portas, configuração de memória, variáveis de ambiente, etc.
• Service: Services executam e mantêm as tasks rodando dentro do cluster. O Service irá auto recuperar qualquer task que estiver parada, tentando deixar o total delas em execução com o mesmo valor que especificado.

Fargate

Fargate é a tecnologia que nos permite rodar containers no ECS sem a necessidade de gerenciarmos o cluster de servidores EC2. Apenas fazemos o deploy da aplicação Docker e definimos as regras de escalabilidade. Fargate é um método de execução do ECS.

Show me the code

O exemplo completo está no Github.

A estrutura do projeto

Nosso projeto do Terraform é composto pela seguinte estrutura:

├── modules
│ └── code_pipeline
│ └── ecs
│ └── networking
│ └── rds
├── pipeline.tf
├── production.tf
├── production_key.pub
├── terraform.tfvars
└── variables.tf

• Modules é onde iremos armazenar o código que cuida da criação de um grupo de recurso. Ele pode ser reutilizado por todos os ambientes (Produção, Staging, QA, etc) sem a necessidade de duplicação de código.
• production.tf é o arquivo que definimos nosso ambiente de produção. Ele executa os módulos passando as variáveis para definir as configurações.
• pipeline.tf O pipeline pode ser um recurso global sem a necessidade de ser isolado por ambiente. Esse arquivo irá cuidar da criação do pipeline usando o módulo code_pipeline.

A primeira parte, a rede

A branch para essa parte pode ser encontrada aqui.

A primeira coisa que precisamos criar é a VPC com 2 subnets (1 pública e 1 privada) em cada Availability Zone. Cada Availability Zone fica em uma região isolada geograficamente. Mantendo nossos recursos em mais de uma Availability Zone é o primeiro item para termos High Availability. Se um desses locais físicos falham por algum motivo, sua aplicação pode responder dos outros locais.

Mantendo nosso cluster na subnet privada protege nossa infraestrutura de acesso externo. A subnet privada é configurada para permitir acesso apenas da subnet pública (Apenas o Load Balancer no nosso caso).

Esse é o código para criar essa parte (é praticamente o mesmo do meu post de introdução ao Terraform):

O código acima cria a VPC, 4 subnets (2 públicas e 2 privadas) em cada Availability Zone. Também cria o NAT para permitir a subnet privada acessar a internet.

O Banco de dados

A branch para essa parte pode ser encontrada aqui.

Iremos criar uma instância de banco de dados no RDS. Ele estará localizado na subnet privada, permitindo apenas a public subnet de accessar.

Com esse código criamos o recurso do RDS com os valores recebidos das variáveis. Também criamos o Security Group que deverá ser usado pelos recursos que se comunicam com o banco de dados (no nosso caso, o cluster ECS).

Ok. Agora temos o banco de dados. Vamos finalmente criar nosso cluster ECS para fazer o deploy da aplicação \o/.

O ECS

A branch para essa parte pode ser encontrada aqui.

Estamos nos aproximando dos passos finais. Agora é a parte que definimos os recursos do ECS necessários para nossa aplicação.

O repositório ECR

A primeira coisa que devemos fazer é criar o repositório para armazenar nossas imagens Docker.

O cluster ECS

Agora precisamos de nosso cluster ECS. Mesmo usando Fargate (que não precisa de nenhuma instância EC2), nós devemos definir o cluster ECS para a aplicação.

As Tasks Definitions

Vamos definir 2 Task Definitions

• Web: Contem a definição da aplicação Web.
• Db Migrate: Essa tarefa executará apenas o comando para migrar nosso banco de dados e irá morrer. Como é uma tarefa de execução única, não precisamos de um Service para ela.

As Tasks Definitions são configuradas em um arquivo JSON e renderizadas como um template no Terraform.
Essa é a Task Definition para a aplicação Web:

No arquivo acima estamos definindo a Task Definition para o ECS. Nós passamos o repositório ECR criado como uma variável. Também configuramos outras variáveis para que o ECS consiga iniciar nossa aplicação Rails.
A definição da migração de banco de dados é praticamente a mesma. Nós apenas mudamos o comando que será executado.

O Load Balancer

Antes de criarmos os Serviços, nós precisamos criar o Load Balancer. Ele será colocado na subnet pública e irá enviar as requests para o Service no ECS.

No arquivo acima nós definimos que nosso Target Group irá usar a porta 80 com o protocolo HTTP. Nós também criamos um Security Group para permitir acessos vindos da internet apenas na porta 80. Depois criamos o Application Load Balancer e o Listener. É necessário usar o Application Load Balancer no lugar do Elastic Load Balancer para usar o Fargate.

O ECS Service finalmente

Agora iremos criar o Service. Para usar o Fargate, precisamos especificar o launch_typecom o valorFargate.

O Fargate nos permite escalar nossa aplicação facilmente. Para isso, precisamos apenas criar as métricas no CloudWatch e ativar os gatilhos para escalar para cima e para baixo.

Nós criamos 2 politicas de Auto Scaling. Uma para escalar para cima e outra para escalar para baixo a quantidade desejada de Tasks rodando no nosso Service Web do ECS.

Depois criamos uma métrica no CloudWatch baseada no uso de CPU. Se o uso de CPU está maior que 85% por 2 períodos, a trigger de alarm_actionexecuta a política de Scale Up. Se retornar para o estado Ok, irá executar a política de Scale Down.

O Pipeline para o deploy da aplicação

Nossa infraestrutura para fazer o deploy da aplicação Docker está pronta. Mas ainda é meio tedioso fazer o deploy para o ECS. No nosso caso, precisamos fazer manualmente o envio da imagem para o repositório ECR e atualizar a Task Definition com a nova imagem. Podemos executar isso através do Terraform, mas pode ser melhor se conseguirmos fazer um push no nosso código no Github na master branch e o deploy for feito automaticamente para nós.

Apresentando, CodePipeline e CodeBuild.

CodePipeline é o serviço gerenciado de Continuous Integration e Continuous Delivery da AWS.

CodeBuild é um serviço gerenciado de build que pode executar testes e gerar pacotes para nós (no nosso caso, uma imagem Docker).

Com eles nós podemos criar pipelines para entregar nossa aplicação para o ECS. O fluxo será:

• Você faz o push do código para a master branch.
• CodePipeline irá pegar o código no passo de Source e executará o passo de Build (CodeBuild).
• O passo de Build irá processar nosso Dockerfile, construindo e enviando a imagem para o ECR and chamando o passo de Deploy.
• O passo de Deploy atualiza nosso cluster ECS com a nova imagem.

Vamos definir nosso Pipeline com o Terraform:

No código acima nós criamos um projeto CodeBuild, usando o seguinte buildspec (arquivo de especificação de build):

Nós definimos algumas fases no arquivo acima..

• pre_build: Atualiza o aws-cli, configura algumas variáveis de ambiente: REPOSITORY_URL com o repositório ECR e IMAGE_TAG com versão de source do CodeBuild. O repositório ECR é passado como variável pelo Terraform.
• build: Constrói usando o Dockerfile e gera uma imagem Docker e cria uma tag LATEST no repositório.
• post_build: Envia a imagem gerada no passo anterior para o repositório. Cria um arquvio chamado imagedefinitions.json com o seguinte conteúdo:
  ‘[{“name”:”web”,”imageUri”:REPOSITORY_URL”}]’
  Esse arquivo é usado pelo CodePipeline para atualizar nosso cluster ECS no passo de Deploy.
• artifacts: Pega o arquivo imagedefinitions.jsongerado no passo anterior e usa como o artefato.

Depois é criado o CodePipeline com 3 estagágios:

• Source: Pega o repositório do Github e executa o próximo passo.
• Build: Executa o CodeBuild que criamos.
• Production: Pega o artefato gerado pelo passo de Build (imagedefinitions.json) e faz o deploy para o ECS.

Vamos ver tudo trabalhando junto?

Executando tudo

O código com o exemplo completo pode ser encontrado aqui.

Clone ele. Como usamos o Github como provider de código no CodePipeline, você precisa gerar o token de acesso para o repositório. (https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/)

Depois de gerar o token, exporte como uma variável de ambiente.

$ export GITHUB_TOKEN=YOUR_TOKEN

Agora precisamos importar os módulos e bibliotecas de providers do Terraform.

$ terraform init

Agora deixe a magia começar!

$ terraform apply

Ele irá mostrar que o Terraform irá criar alguns recursos e se você deseja continuar.

Digite yes.

Terraform irá começar a criar nossa infraestrutura.

Sério, pegue um café até ele terminar.

AWESOME!. Nossa infraestrutura está pronta!!. Se você entrar no seu CodePipeline no Dashboard da AWS você ira ver que ele ativou nossa primeira build.

Espere até todos passos ficarem verdes.

Pegue o DNS do seu Load Balancer e verifique a aplicação.

$ terraform output alb_dns_name

Finalmente, a aplicação está rodando. Magica!

Se tiver alguma dúvida, entre em contato comigo. Esse post foi apenas uma introdução para o ECS usando Fargate e Terraform.

Cheers 🍻

Deploy de aplicações Docker na AWS usando ECS e Fargate was originally published in Opensanca on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, I will try to demonstrate how you can deploy your Docker application into AWS using ECS and Fargate.

As an example, I will deploy this app to ECS. The source can be found here.

I will use Terraform to spin the infrastructure so I can easily track everything that I create as a code. If you want to learn the basics of Terraform, please read my post about it.

ECS

What is ECS?

The Elastic Container Service (ECS) is an AWS Service that handles the Docker containers orchestration in your EC2 cluster. It is an alternative for Kubernetes, Docker Swarm, and others.

ECS Terminology

To start understanding what ECS is, we need to understand its terms and definitions that differs from the Docker world.

• Cluster: It is a group of EC2 instances hosting containers.
• Task definition: It is the specification of how ECS should run your app. Here you define which image to use, port mapping, memory, environments variables, etc.
• Service: Services launches and maintains tasks running inside the cluster. A Service will auto-recover any stopped tasks keeping the number of tasks running as you specified.

Fargate

Fargate is a technology that allows running containers in ECS without needing to manage the EC2 servers for cluster. You only deploy your Docker applications and set the scaling rules for it. Fargate is an execution method from ECS.

Show me the code

The full example is on Github.

The project structure

Our Terraform project is composed of the following structure:

├── modules
│ └── code_pipeline
│ └── ecs
│ └── networking
│ └── rds
├── pipeline.tf
├── production.tf
├── production_key.pub
├── terraform.tfvars
└── variables.tf

• Modules are where we will store the code that handles the creation of a group of resources. It can be reused by all environments (Production, Staging, QA, etc.) without needing to duplicate a lot of code.
• production.tf is the file that defines the environment itself. It calls the modules passing variables to it.
• pipeline.tf Since the pipeline can be a global resource without needing to isolate per environment. This file will handle the creation of this pipeline using the `code_pipeline` module.

First part, the networking

The branch with this part can be found here.

The first thing that we need to create is the VPC with 2 subnets (1 public and 1 private) in each Availability Zone. Each Availability Zone is a geographically isolated region. Keeping our resources in more than one zone is the first thing to achieve high availability. If one physical zone fails for some reason, your application can answer from the others.

Keeping the cluster on the private subnet protects your infrastructure from external access. The private subnet is allowed only to be accessed from resources inside the public network (In our case, will be the Load Balancer only).

This is the code to create this structure (it is practically the same from my introduction post of Terraform):

The above code creates the VPC, 4 subnets (2 public and 2 private) in each Availability zone. It also creates a NAT to allow the private network access the internet.

The Database

The branch with this part can be found here.

We will create a RDS database. It will be located on the private subnet. Allowing only the public subnet to access it.

With this code, we create the RDS resource with values received from the variables. We also create the security group that should be used by resources that want to connect to the database (in our case, the ECS cluster).

Ok. Now we have the database. Let’s finally create our ECS to deploy our app \o/.

Take Three: The ECS

The branch with this part can be found here.

We are approaching the final steps. Now, it is the part that we define the ECS resources needed for our app.

The ECR repository

The first thing is to create the repository to store our built images.

The ECS cluster

Next, we need our ECS cluster. Even using Fargate (that doesn’t need any EC2), we need to define a cluster for the application.

The tasks definitions

Now, we will define 2 task definitions.

• Web: Contains the definition of the web app itself.
• Db Migrate: This task will only run the command to migrate our database and will die. Since it is a single run task, we don’t need a service for it.

The tasks definitions are configured in a JSON file and rendered as a template in Terraform.
This is the task definition of the web app:

In the file above, we are defining the task to ECS. We pass the created ECR image repository as variable to it. We also configure other variables so ECS can start our Rails app.
The definition of the DB migration task is almost the same. We only change the command that will be executed.

The load balancers

Before creating the Services, we need to create the load balancers. They will be on the public subnet and will forward the requests to the ECS service.

In the file above we define that our target group will use HTTP on port 80. We also create a security group to allow access into the port 80 from the internet. After, we create the Application Load Balancer and the listener. To use Fargate, you should use an Application Load Balancer instead an Elastic Load Balancer.

Finally, the ECS service

Now we will create the service. To use Fargate, we need to specify the lauch_type as Fargate.

Auto-scaling

Fargate allows us to auto-scale our app easily. We only need to create the metrics in CloudWatch and trigger to scale it up or down.

We create 2 auto scaling policies. One to scale up and other to scale down the desired count of running tasks from our ECS service.

After, we create a CloudWatch metric based on the CPU. If the CPU usage is greater than 85% from 2 periods, we trigger the alarm_action that calls the scale-up policy. If it returns to the Ok state, it will trigger the scale-down policy.

The Pipeline to deploy our app

Our infrastructure to run our Docker app is ready. But it is still boring to deploy it to ECS. We need to manually push our image to the repository and update the task definition with the new image and update the new task definition. We can run it through Terraform, but it could be better if we have a way to push our code to Github in the master branch and it deploys automatically for us.

Entering, CodePipeline and CodeBuild.

CodePipeline is a Continuous Integration and Continuous Delivery service hosted by AWS.

CodeBuild is a managed build service that can execute tests and generate packages for us (in our case, a Docker image).

With it, we can create pipelines to delivery our code to ECS. The flow will be:

• You push the code to master’s branch
• CodePipeline gets the code in the Source stage and calls the Build stage (CodeBuild).
• Build stage process our Dockerfile building and pushing the Image to ECR and triggers the Deploy stage
• Deploy stage updates our ECS with the new image

Let’s define our Pipeline with Terraform:

In the above code, we create a CodeBuild project, using the following buildspec (build specifications file):

We defined some phases in the above file.

• pre_build: Upgrade aws-cli, set some environment variables: REPOSITORY_URL with the ECR repository and IMAGE_TAG with the CodeBuild source version. The ECR repository is passed as a variable by Terraform.
• build: Build the Dockerfile from the repository tagging it as LATEST in the repository URL.
• post_build: Push the image to the repository. Creates a file named imagedefinitions.json with the following content:
  ‘[{“name”:”web”,”imageUri”:REPOSITORY_URL”}]’
  This file is used by CodePipeline to upgrade your ECS cluster in the Deployment stage.
• artifacts: Get the file created in the last phase and uses as the artifact.

After, we create a CodePipeline resource with 3 stages:

• Source: Gets the repository from Github (change it by your repository information) and pass it to the next stage.
• Build: Calls the CodeBuild project that we created in the step before.
• Production: Gets the artifact from Build stage (imagedefinitions.json) and deploy to ECS.

Let’s see they working together?

Running all together

The code with the full example is here.

Clone it. Also, since we use Github as the CodePipeline source provider, you need to generate a token to access the repositories. [Read here](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) to generate yours.

After generating your token, export it as an environment variable.

$ export GITHUB_TOKEN=YOUR_TOKEN

Now, we need to import the modules and the provider library.

$ terraform init

Now, let the magic begin!

$ terraform apply

it will display that Terraform will create some resources, and if you want to continue

Type yes.

Terraform will start create our infraestructure.

Seriously, get a coffee until it finishes.

AWESOME!. Our infrastructure is ready!!. If you enter in your CodePipeline at AWS Dashboard, you can see that it also triggered the first build:

Wait until all the Stages are green.

Get your Load Balancer DNS and check the deployed application:

$ terraform output alb_dns_name

Finally, the app is running. Almost magic!

If you have any questions about it, contact me. This was just an introduction post about ECS with Fargate using Terraform.

Cheers 🍻

Easy deploy your Docker applications to AWS using ECS and Fargate was originally published in Code Pub on Medium, where people are continuing the conversation by highlighting and responding to this story.

When I am writing code, usually I want to evaluate some piece of code. I used to do the following actions:

• Copy and paste my code to IRB (or run my ruby script file directly from the terminal).
• When using tmux, send my code directly from vim to tmux with the vim-tmux-runner plugin.

The first option needs an extra work of copying and pasting content. I prefer the second option, but sometimes I forgot to attach the VTR pane and get errors.

Now I’m using https://github.com/JoshCheek/seeing_is_believing along with https://github.com/t9md/vim-ruby-xmpfilter plugin

I set my .vimrc with the following content (I use Plug to manage my dependencies):

Plug 't9md/vim-ruby-xmpfilter'

" Enable seeing-is-believing mappings only for Ruby
let g:xmpfilter_cmd = "seeing_is_believing"

autocmd FileType ruby nmap <buffer> <F4> <Plug>(seeing_is_believing-mark)
autocmd FileType ruby xmap <buffer> <F4> <Plug>(seeing_is_believing-mark)
autocmd FileType ruby imap <buffer> <F4> <Plug>(seeing_is_believing-mark)

autocmd FileType ruby nmap <buffer> <F6> <Plug>(seeing_is_believing-clean)
autocmd FileType ruby xmap <buffer> <F6> <Plug>(seeing_is_believing-clean)
autocmd FileType ruby imap <buffer> <F6> <Plug>(seeing_is_believing-clean)

autocmd FileType ruby nmap <buffer> <F5> <Plug>(seeing_is_believing-run)
autocmd FileType ruby xmap <buffer> <F5> <Plug>(seeing_is_believing-run)
autocmd FileType ruby imap <buffer> <F5> <Plug>(seeing_is_believing-run)

Now I can visual select my code, use F4 to mark and that line will be evaluated, press F5 and get the result of that code. After, I can clean all marks with F6 .

Cheers,
🍻

Evaluate your ruby code directly from VIM was originally published in Code Pub on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post I will show a simple example about how to create apps for each pull request (or, creating your own Gitlab/Heroku Review App).

Let’s imagine the following scenario of a development team:

• Developer creates a new feature branch.
• Developer pushes the branch
• Developer opens a pull request so other developers can check his code and test the feature
• CI runs the tests and makes the branch green if it passes

Now the developer should send the feature to someone else to test it. The feature should not be merged into master so it can be tested. They should be tested isolated because one feature can interfere with another and master should only have deployable code. With this in mind, we will create an environment for each pull request using Docker.

I will use Jenkins as CI because I want to use an open source tool to demonstrate this, but you can use any CI tool that you prefer.

This post will assume that you already have Jenkins installed. I will use this opensource Rails application as an example. Fork this repo into your account and clone it in your computer.

git clone https://github.com/yourgithubusername/opensanca_jobs.git

You can get the full code repo that I used on https://github.com/duduribeiro/openjobs_jenkins_test

Configuring Docker

The first step that we need is to configuring our app with docker.

A Dockerfile is a file that has instructions to build an image that contains our application. You can learn more about Docker in their documentation.

Create a file named Dockerfile in the app folder:

This is our instructions to build an image. The first command FROM specifies to docker that we will use the image ruby:2.4.1 as our base image.

After, the first RUN command installs all dependencies the app needs: yarn, imagemagick and node (to precompile the assets. A fancy solution is to use a different container only with node and sprockets to precompile the assets). The second RUN command, creates a folder /var/app that will be responsible to store the application. The COPY command moves the current folder to the container in the /var/app folder. The next RUN command installs all dependencies from Rails and yarn. The CMD specify the command that the container should execute when it runs the image. You can learn more about Dockerfile here.

With this file we can build our image.

docker build -t myimage .

With this command, we are building our Dockerfile and generating an image with the tag myimage .

Running docker image ls we can check our image.

Now we can start our application:

docker run -d -p 3000:3000 myimage

The -d option tells to docker that this container will run in background and -p 3000:3000 will connect the local port 3000 with the exposed 3000 port from the container.

We can navigate now to http://localhost:3000.

And we receive this error. This is because our app requires a database connection. We need to run a new container with the database and link both containers so they can communicate with each other. Each container should have only one purpose, so, you should not run more than 1 service in a single container (ie: a container with the application and the database).

Instead of manually run each container, we will use docker compose , a tool to help us to run multi containers applications.

Use docker ps and docker kill to destroy your application container/

Create a docker-compose.yml file:

In this file, we are creating 2 services. One for the database using the postgres image, and another one with our web application using our Dockerfile image.

In the environment item on web service, I’m setting the DATABASE_URL environment variable. If this environment variable is set, rails will use it replacing the loaded configurations from config/database.yml. Read more about this here. DATABASE_URL follows this pattern for postgres:
postgres://user:password@host/database_name.Since our database user does not have password, we leave empty after the : . We don’t fill the database_name because we want the configured one in config/database.yml (one database name per environment).

Create the databases and run the migrations:

docker-compose run --rm web rake db:create db:migrate

In this command, we will run a container with the web service and execute the command rake db:create db:migrate . The --rm option is to remove the container after the execution.

Run the tests:

docker-compose run --rm -e RAILS_ENV=test web rake db:drop db:create db:migrate

docker-compose run --rm web rspec

Now we can start the application:

docker-compose up

Access http://localhost:3000 again and now our app is working.

Configuring our pipeline

We will use Jenkins pipeline as code to configure our pipeline. Read more about it here. We need to have docker, jq and Jenkins installed on CI server.

We will use the following Jenkins plugins: Blue Ocean, Blue Ocean Pipeline Editor, GitHub Pipeline for Blue Ocean

Create a file named Jenkinsfile in the project root with the following content:

We have 4 stages on this pipeline:

• Build: It will build our Dockerfile and generate an image from this build tagged with openjobs with version named latest . It will use docker-compose to build, install dependencies, create and migrate the database created with compose .
• Tests: It will run 2 parallel steps, one to run unit tests, and another to run feature tests.
• Deploy to staging: If it is the master branch, it will deploy the app to staging.domain . We will cover this in the next steps.
• Create feature environment: If it isn’t the master branch, it will deploy the app to branchname.domain . We will cover this in the next steps.

Let’s create our pipeline on Jenkins. Push your code, go to your Jenkins, and access the blue ocean interface and click in the New Pipeline button:

In the next screen, select Github, choose you account and find the repository. Click in Create Pipeline

And our build is passing 🎉

The red icon on the build is because this step will not run, since the branch built was master .

Now we need to create our environments after the build and dynamic route a domain to a specific container.

Entering, Traefik

Traefik is a tool that will help us make the dynamic routing and act as a load balancer. Example: If I access http://mybranch.mydomain.com , I want to access the container containing the app with mybranch that should be started by Jenkins.

Creating dynamic environments

The following steps should be executed on CI server.

We will use Docker Swarm. It is very helpful to create a docker cluster. I will use only one server to demonstrate, but you are able also to create a cluster.

Initialize swarm cluster:

docker swarm init --advertise-addr=10.10.0.5

This will initialize our server as the swarm master. It will generate a command that you can use to join the cluster in other servers.

Create a network that we can use with traefik and our containers:

docker network create --driver=overlay --attachable traefik-net

Initialize traefik:

docker service create \
--name traefik \
--constraint 'node.role==manager' \
--publish 80:80 \
--publish 8081:8080 \
--mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock \
--network traefik-net \
traefik \
--docker \
--docker.swarmmode \
--docker.domain=apps.carlosribeiro.me \
--docker.watch \
--logLevel=DEBUG \
--web

This will initialize traefik. You should treat this initialization on some startup script on your server.

If we access our server in 8081 port, we can access traefik dashboard.

The docker.domain specify that we will access the apps through appname.apps.carlosribeiro.me . To allow this, I created two alias on my DNS server pointing to the CI server’s IP:

• apps.carlosribeiro.me
• *.apps.carlosribeiro.me

Editing Jenkinsfile to create the environments

Lets add a method on Jenkinsfile that will create a environment when the build is triggered.

This method will accept an argument to inform the name of the environment. This name will be used to prefix the services name. It will stop all docker-compose containers that are still running. After, it will remove if exist, services with the same name (This is to recreate the environment when we push again to the branch). After, we create 3 services. One for Postgres, another for Redis, and the app itself using the imaged that we built on build step. The last command, runs a temporary docker container to create and migrate the database and precompile the assets (you can fetch your previously dump from production too).

Look that in the app service, we specify some environment variables.

• REDIS_URL, DATABASE_URL: Endpoint to connect in both services using the previously created services hosts.
• RAILS_ENV: We set it to production . It will enforce that our app should behave like a production environment.
• RAILS_SERVE_STATIC_FILES: Since we will not have a nginx in front of the app server, we need to set this to tell rails to service static files for us.

— label ‘traefik.port’ will inform traefik what is the container’s exposed port. In our case, is the 3000.

— name inform the service name that traefik will identify as the prefix on domain.

Now, lets call this method on our pipeline’s stages.

If it is master branch, we will deploy into an app called staging . If not, it will call the app with the same name of branch.

Look the final Jenkinsfile:

Push master and let’s wait Jenkins create our staging for us.

Our build passes…

… and our app is deployed on staging:

Now let’s open a pull request to change this button color.

git checkout -b 01-change-button-color

Edit app/assets/stylesheets/application.scss

.btn-register-vacancy {
  background-color: #2769ff;
  width: 300px;
  height: 35px;
  border-radius: 5px;
}

Push this branch, and open a pull request.

Github will tell us that the build is pending.

…

Job is passing

and Github is informed:

And now we have our dynamic environment up and running

I can send this URL to a QA or a PO review it before merging. If something is wrong, it can be validated before it is on master.

That’s all

Cheers 🍻

Creating review apps per pull requests was originally published in Code Pub on Medium, where people are continuing the conversation by highlighting and responding to this story.

Sometimes when you handle a lot of servers in the cloud, it is pretty easy to get lost on your infrastructure. “Where is that freaking server that I can’t find?”, or even “Why is this instance for?”.

In this post, I will introduce one tool that I found myself liking a lot: To have a bootstrap of an Infrastructure as Code flow in my applications. Terraform. Infrastructure as Code (IaC) allows us to have a repository with code that describes our infrastructure. This way, we can avoid reminding how rebuild the entire infrastructure for an application. It will be on the code and it can be versioned and tested. And if something goes wrong, we can revert it. It is very useful having a continuous integration of the infrastructure. The whole team knows how it was built and all the pieces. You can apply the same flow that you use in you app code to your infrastructure, i.e: Someone makes a change, open a Pull Request, someone reviews it and after it is approved, you merge and your CI tools apply the changes in your environment.

Why Terraform? What is the difference between Chef, Puppet or Ansible?

Chef, Puppet and Ansible are IaC tools too, but they focus on configuring operating system and applications. They are called Configuration Management Tools and they also can build infrastructure on the cloud with a help of plugins, but usually it is hard to configure and sometimes it is limited. With Terraform you can build from the services to the networking part. You can use Terraform to create the infrastructure and a configuration management tool to configure the applications. Terraform can’t replace your configuration management tool, but it’s made to work together with it.

A practical example

In order to follow this article, you’ll need an AWS account.

Let’s create the servers for our web application. Take a look at this diagram of the resources that we’ll build.

We will create 2 subnets (one for public access and another private). We have Elastic Load Balancer in the public subnet to handle the traffic to our web servers. Our web servers will be on the private subnet and it will only be accessible through the Load Balancer. This mean that we won’t have direct access to make connections (for example, SSH) on the server. In order to access via SSH an instance on a private subnet, you’ll need a bastion host and connect to the web server through it. Thus, we will create the bastion host on the public subnet.

Before getting your hands dirty, you need to install Terraform. Follow the instructions of this link to install it in your machine.

Directory structure

Let’s create a folder to handle our infrastructure code.

mkdir ~/terraform

I like to follow this pattern when working with Terraform:

├── modules
│   ├── networking
│   └── web
├── production
└── staging

Let’s create this folder structure

cd ~/terraform
mkdir -p modules/{networking,web} production staging

Our modules folder contains all the shared code to create the pieces of the infrastructure (web servers, app servers, databases, vpc, etc). Each folder inside the modules folder is related to a specific module.
Next, I have folders for my environments (staging, production, qa, development, etc). Each of this folder contains code to use our shared modules and create a different architecture for each environment (This is my personal approach using Terraform, but feel free to work on a different way).

Our first module: Networking

Let’s create our networking module. This will be responsible for creating the networking pieces of our infrastructure, like VPC, subnets, routing table, NAT server and the bastion instance.

Before we get deep in the code, I wanna explain how terraform works:

Terraform will provide us with some commands. Some of them are:

plan: Displays all the changes that Terraform makes on our infrastructure
apply: Executes all the changes to the infrastructure
destroy: Destroys everything that was created with Terraform

When you run Terraform inside a directory, it loads ALL .tf files from the directory and execute them (will not load on subfolders). Terraform will first create a graph of the resources to apply only in the final phase, so you don’t need to specify the resources in any specific order. The graph will determine the relations between the resources and ensure that Terraform creates they in the right order.

Continuing on our networking module

Enter in the networking module

cd modules/networking

Let’s create our first tf file. The one that specifies all variables needed for our module.

touch variables.tf

Insert the following content on the variables.tf file:

These are all variables that our networking module needs in order to create all resources. We need the CIDR for the VPC and the subnets, the AWS region that we will use, the key name and the environment that we are building.
This is the way that you specify a variable in Terraform

variable "variable_name" {
  description = "The description of the variable"
  default = "A default value if this isn't set
}

Ok. Now we have our variables.tf file to specify the interface of our module. Let’s create the file that will create networking stuffs for our module. Create the main.tf file in the networking folder. (you can specify any name that you want. Remember, all tf files will be loaded).

touch main.tf

Insert the following content on the main.tf file:

Here we are creating all the networking part of our infrastructure based on the diagram that we saw. A VPC, both subnets (public and private), the Internet Gateway to the public subnet, the NAT server for the private subnet, the bastion host and all security group for the VPC, allowing inbound and outbound inside the VPC, and the security group for the bastion host, allowing the SSH on the Port 22. You can check this link from AWS for more details.

This is how we create a resource on Terraform.

resource "resource" "name" {
  attribute = "value"
}

The resource is the name of the resource that we want to build. Each cloud provider has different resources. For example, these resources from AWS. We can also concatenate values. Remember that we created the variables file? We use them here with the var.variable_name . Like in this part of the code, which we use the key_name variable that we specified in the variables file:

resource "aws_instance" "bastion" {
  ...
  key_name = "${var.key_name}"
  ...
}

This is how we created the instance and called its bastion. You can also get property of this resource in other parts of the code. Example:

resource "someresource" "somename" {
  attribute = "${aws_instance.bastion.id}"
}

We use the same idea of var concatenation. But we specify ${resource_type.resource_name.property} .

Our networking module is almost ready. We need to output some variables after the module build the resources, so we can use it in other parts of the code. Terraform has the output command, allowing us to expose variables.

Create the output.tf file inside the networking folder.

touch output.tf

Insert the following content on the output.tf file:

This is how we output a variable from our module:

output "variable_name" {
  value = "variable value"
}

This will allow us to get these variables outside the module.

Using our module to build the networking from our environment

Now that our networking module is ready, we can use it to build our networking from our environment (i.e, staging).
This is how our terraform folder looks now:

├── modules
│   ├── networking
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   └── web
├── production
└── staging

Go into staging folder.

cd ~/terraform/staging

First, create the public key for the staging servers

ssh-keygen -t rsa -C "staging_key" -f ./staging_key

Let’s create our main file. In this file, we will specify information of the AWS provider.

touch _main.tf

(We use _ at the beginning of the name because since terraform loads all files alphabetically, we need this to be loaded first, since it will create the keypair)

You can specify things like Access and secret key in some ways:

• Specify it directly in the provider (not recommended)

provider "aws" {
  region     = "us-west-1"
  access_key = "myaccesskey"
  secret_key = "mysecretkey"
}

• Using the AWS_ACCESS_KEY and AWS_SECRET_KEY environment variables

$ export AWS_ACCESS_KEY_ID="myaccesskey"
$ export AWS_SECRET_ACCESS_KEY="mysecretkey"

terraform plan

The second option is recommended because you don’t need to expose your secrets on the file. Bonus point: If you have the AWS cli you don’t need to export these variables. Only run the aws configure command and terraform will use the variables that you set on it.

Insert the following content on the _main.tf file:

We will only specify the region to the provider. Both access and secret key, we will rely on the AWS cli .

Now create the networking.tf file. It will use our module to create the resources.

touch networking.tf

Insert the following content on the networking.tf file:

This is how we use modules on Terraform.

module "name" {
  source    = "location_path"
  attribute = "value"
}

The module attributes are all variables that we specified before in the variables.tf file from our networking module. Look that we are passing more variables to our module attributes. We need our environment to require these variables too. Create the variables.tf file for our staging environment.

touch variables.tf

Add the following content to the variables.tf file:

This file follows the same pattern of the module’s variables file. These are all variables that we need to build our staging networking piece.

Ok… I think that we are ready to go. Your terraform’s folder structure should be like this:

├── modules
│   ├── networking
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   └── web
├── production
└── staging
    ├── _main.tf
    ├── networking.tf
    └── variables.tf

Run this command on staging folder: (Terraform’s commands should be run on the environments folder).

cd ~/terraform/staging
terraform get
terraform plan

*the terraform get command only syncs all modules.

After executing the terraform plan command, it will ask you a lot of informations (our variables). Answer they:

It will output a lot of things. Resources that will be created. You can analyze it to check if everything is ok. terraform plan will output only the planned change of the infrastructure.

Now, let’s apply these modifications.

terraform apply

But it is asking for all that information again. To avoid this, you can follow 2 ways to automatically inject variable’s value to Terraform

• Using a terraform.tfvars file and remember to ignore this file in your VCS.
• Specifying variables in Environment variables. TF_VAR_environment=staging terraform apply (this can be useful when you run through some CI tool)

We will follow the first way. So, create a terraform.tfvars file

touch terraform.tfvars

Insert the following content on terraform.tfvars :

Now you can run the apply command without being asked for variable’s value.

terraform apply

You should receive the following message:

Congratulations..

This will generate 2 files on the staging folder: terraform.tfstate and terraform.tfstate.backup

Terraform controls all their resources on this terraform.tfstate file. You should NEVER delete this file. If you do, terraform will think that it needs to create new resources and will lose tracking with the others that it has been already created.

And how can I make my team in sync with the state?

You have 2 ways to keep the team with the remote in sync. You can commit this .tfstate file to your VCS repository, or use Terraform Remote State. If you rely on terraform’s remote state, I really recommend you to use some wrapper tool for Terraform like Terragrunt. It can handle the remote state locking and initializing it for you.

Creating the web servers

This is our folder structure until now.

├── modules
│   ├── networking
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   └── web
├── production
└── staging
    ├── _main.tf
    ├── networking.tf
    ├── staging_key
    ├── staging_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    └── variables.tf

Go into our web module

cd ~/terrform/modules/web

Following the same flow we used to create the networking module, let’s create our variables file to specify the interface to our module:

touch variables.tf

Insert the following content on variables.tf :

Create the main.tf file to handle the creation of the resources.

touch main.tf

Insert the following content on main.tf :

This file is practically the same from our networking’s main.tf, we are only creating different resources.
Some differences that you can note:

• In this example we are using the count attribute. It specifies to Terraform, to create N times this resource. If we pass 5 on the value, it will create 5 instances.

resource "aws_instance" "web" {
  count = "${var.web_instance_count}"
  ...
}

and you can create dynamic names with the counting number using the count.index property, for example:

tags {
  Name = "web-server-${count.index + 1}"
}

• We are passing a file to the user_data property. In order to execute some code on the instance initialization, we need to pass the user_data attribute to our instance and we are specifying a file.

user_data = "${file("${path.module}/files/user_data.sh")}"

This will load the content of the user_data.sh file and pass to the attribute.

But we haven’t created this file yet, let’s do it. On the web module folder, create the files folder and the user_data.sh file.

mkdir files
touch files/user_data.sh

Insert the following content on the files/user_data.sh :

This will install the nginx when the instance is created.

Now, let’s create the output.tf file to get the load balancer’s DNS after the execution.

touch output.tf

Insert the following content on the output.tf :

Our web module is done.

This is how our terraform structure is now:

├── modules
│   ├── networking
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   └── web
│       ├── files
│       │   └── user_data.sh
│       ├── main.tf
│       ├── output.tf
│       └── variables.tf
├── production
└── staging
    ├── _main.tf
    ├── networking.tf
    ├── staging_key
    ├── staging_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    └── variables.tf

Using our web module

Let’s back to our staging folder

cd ~/terraform
cd staging

Now, we will use our recent created web module. Create a web.tf file

touch web.tf

Insert the following content on the web.tf :

This is pretty much the same way we used in our networking module. We are using almost the same variables that we already specified (except for web_instance_count . And some variables, we pass the output from our networking module.

"${module.networking.public_subnet_id}"

This way we get the public_subnet_id output created on the networking module.

Let’s add the web_instance_count variable to our variables.tf file and terraform.tfvars . This variable represents the number of web instances that we will be created.

Your variables.tf from staging folder should be like this:

And your terraform.tfvars should be like this:

Let’s create an output.tf for our staging environment. With this, we can get the ELB hostname from our web module.

touch output.tf

Add the following content to output.tf :

This is the final directory structure that we have:

├── modules
│   ├── networking
│   │   ├── main.tf
│   │   ├── output.tf
│   │   └── variables.tf
│   └── web
│       ├── files
│       │   └── user_data.sh
│       ├── main.tf
│       ├── output.tf
│       └── variables.tf
├── production
└── staging
    ├── _main.tf
    ├── networking.tf
    ├── output.tf
    ├── staging_key
    ├── staging_key.pub
    ├── terraform.tfstate
    ├── terraform.tfstate.backup
    ├── terraform.tfvars
    ├── variables.tf
    └── web.tf

You can now run terraform plan to check which resources will be created. (before, run terraform get to update the modules)

terraform get
terraform plan

Done. Terraform will create 5 new resources from our web module.

Let’s apply it.

terraform apply

Yay.. our instances was created. Let’s get our Load Balancer DNS and try to open it on a Browser:

terraform output elb_hostname

This command will return the DNS from our Load Balancer. Open it on a browser:

It’s working!!! With this, you finished the creation of your infrastructure.

Final Notes:

Your web instances don’t have public ips:

In order to SSH then, you need to use the bastion host created on the networking module.

Get the bastion host public IP,

and SSH on it with the -A flag to enable agent forwarding:

chmod 400 staging_key.pub

ssh-add -K staging_key

ssh -A ubuntu@52.53.227.241

Now, inside the Bastion host, you can connect into your web server private IP:

ssh ubuntu@10.0.2.113

Now, destroy everything

terraform destroy

You can get the full code of the example here.

That’s all.

Cheers,

🍻

Creating your cloud servers with Terraform was originally published in Code Pub on Medium, where people are continuing the conversation by highlighting and responding to this story.