iridakos - tutorials

Mongoid - Inheritance: change embedded document's type via nested attributes

Fri, 22 Nov 2019 09:00:00 +0200

I recently faced some issues when trying to update a nested document’s type via nested attributes using the mongoid gem. Bare with me if this sentence doesn’t make any sense yet.

Use case

Suppose we have a document named Person that embeds a document that can be any of the types Book or BoardGame which inherit from the class Item.

Item

class Item
  include Mongoid::Document

  embedded_in :person, inverse_of: :item

  field :name
end

Book

class Book < Item
  field :authors, type: Array
end

BoardGame

class BoardGame < Item
  field :players, type: Integer
end

Person

class Person
  include Mongoid::Document

  embeds_one :item, inverse_of: :person
  accepts_nested_attributes_for :item, allow_destroy: true
end

The problems

To simplify the post I won’t go with the scenario of building the forms and the controller actions to demonstrate the problems but instead I will simulate the behavior in the rails console.

Let’s create a Person with a Book as an Item.

>> person = Person.create!({
  item_attributes: {
    _type: "Book",
    name: "Jitterbug Perfume",
    authors: ["Tom Robbins"]
  }
})
=> #<Person _id: 5dd7b01ec2889856ab8a619f, >
>> person.item
=> #<Book _id: 5dd7ae76c2889856ab8a619e, name: "Jitterbug Perfume", _type: "Book", authors: ["Tom Robbins"]>

Failing attribute assignments

Let’s try to change the person’s item from Book to BoardGame with 5 players.

>> person.update_attributes({
  item_attributes: {
    _type: "BoardGame",
    name: "Ticket to Ride",
    players: 5
  }
})
Traceback (most recent call last):
        1: from (irb):9
Mongoid::Errors::UnknownAttribute ()
message:
  Attempted to set a value for 'players' which is not allowed on the model Book.
summary:
  Without including Mongoid::Attributes::Dynamic in your model and the attribute does not already exist in the attributes hash, attempting to call Book#players= for it is not allowed. This is also triggered by passing the attribute to any method that accepts an attributes hash, and is raised instead of getting a NoMethodError.
resolution:
  You can include Mongoid::Attributes::Dynamic if you expect to be writing values for undefined fields often.

This didn’t work, the assignment of the nested attributes item_attributes as you can see from the error message were applied in the model Book.

The fact that we changed the _type attribute of the document doesn’t mean that the object in memory also was magically transformed from a Book to a BoardGame.

Document attribute leftovers

Let’s try to change the person’s item from Book to BoardGame but this time without the players to see if this fixes the issue.

>> person.update_attributes({
  item_attributes: {
    _type: "BoardGame",
    name: "Ticket to Ride"
  }
})
=> true
>> person.item
=> #<Book _id: 5dd7b270c2889858a08a619e, name: "Ticket to Ride", _type: "BoardGame", authors: ["Tom Robbins"]>

This seems to have worked even though the result of the person.item accessor gave a Book document with _type of a BoardGame and with an attribute authors present.

Let’s reload to see if the document is going to be properly instantiated.

>> person.reload.item
=> #<BoardGame _id: 5dd7b270c2889858a08a619e, name: "Ticket to Ride", _type: "BoardGame", players: nil>

Seems to be fine but what is going on with the actual document attributes?

>> person.item.attributes
=> {"_id"=>BSON::ObjectId('5dd7b270c2889858a08a619e'), "_type"=>"BoardGame", "name"=>"Ticket to Ride", "authors"=>["Tom Robbins"]}

As you can see, the authors attribute (that we had initially set) hasn’t been removed when we changed the type of the Item from Book to BoardGame.

Wrong validators

Let’s change the BoardGame document so that it validates the presence of players

class BoardGame < Item
  field :players, type: Integer

  validates :players, presence: true
end

Our person now has an item of type BoardGame. We will now try to change it to a Book

>> person.attributes = {
  item_attributes: {
    _type: "Book",
    name: "Jitterbug Perfume"
  }  
}
=> {:item_attributes=>{:_type=>"Book", :name=>"Jitterbug Perfume"}}
>> person.valid?
=> false
>> person.errors.full_messages
=> ["Item is invalid"]
>> person.item.errors.full_messages
=> ["Players can't be blank"]

As described in the previous section, the fact that the _type attribute changed doesn’t mean that the actual object in memory changed accordingly. Upon save of the parent document, the item is still an instance of the previous _type which was BoadGame thus the validations that take place are not the ones defined in the new Book class but those defined in the BoardGame.

This also means that if we had validations in the Book class too, they wouldn’t execute in this scenario as well.

Investigation

I tried to google the issue (I couldn’t even construct a proper query) to see how others dealt with this but I didn’t find anything helpful and I decided to dive in the source of mongoid in order to locate the code that deals with the nested assignments and possibly create a monkey patch.

First, I searched for the file that defines the accepts_nested_attributes_for method that we used in the Person class.

Found in mongoid/attributes/nested.rb

def accepts_nested_attributes_for(*args)
  options = args.extract_options!.dup
  options[:autosave] = true if options[:autosave].nil?

  options[:reject_if] = REJECT_ALL_BLANK_PROC if options[:reject_if] == :all_blank
  args.each do |name|
    meth = "#{name}_attributes="
    self.nested_attributes["#{name}_attributes"] = meth
    association = relations[name.to_s]
    raise Errors::NestedAttributesMetadataNotFound.new(self, name) unless association
    autosave_nested_attributes(association) if options[:autosave]

    re_define_method(meth) do |attrs|
      _assigning do
        if association.polymorphic? and association.inverse_type
          options = options.merge!(:class_name => self.send(association.inverse_type))
        end
        association.nested_builder(attrs, options).build(self)
      end
    end
  end
end

The line that is mostly related to our issue from the above snippet is this:

association.nested_builder(attrs, options).build(self)

which is executed upon attribute assignment.

Since the item association defined in the Person document is of type embeds_one, I located its code to find the nested_builder method.

Found it at mongoid/association/embedded/embeds_one.rb

def nested_builder(attributes, options)
  Nested::One.new(self, attributes, options)
end

Ok, now we have to see what is going on inside the #build method of the mongoid/association/nested/one.rb file.

def build(parent)
  return if reject?(parent, attributes)
  @existing = parent.send(association.name)
  if update?
    attributes.delete_id
    existing.assign_attributes(attributes)
  elsif replace?
    parent.send(association.setter, Factory.build(@class_name, attributes))
  elsif delete?
    parent.send(association.setter, nil)
  end
end

def replace?
  !existing && !destroyable? && !attributes.blank?
end

def update?
  existing && !destroyable? && acceptable_id?
end

The part that is related to the behavior we want to alter is the one inside the if update? .. elsif replace? block:

...
if update?
  attributes.delete_id
  existing.assign_attributes(attributes)
elsif replace?
  parent.send(association.setter, Factory.build(@class_name, attributes))
...  

This is where the attribute assignment in the existing embedded document takes place.

Desired behavior

Since we are talking about embedded documents, when changing the type instead of updating the existing document’s attributes we can easily replace it at once with a new instance of the proper class.

From that point on, we won’t have to deal with

the invalid attribute assignments
removing the old attributes that are not present in the new document type
fixing the usage of wrong validators.

The replaced object will be properly instantiated and assigned to the parent.

Solution

To change to the desired aforementioned behavior, I patched the Mongoid::Association::Nested::One class.

I created a module (I suggest you read this great article about monkey patching without making a mess) under lib/core_extensions/mongoid/association/nested/one/embedded_one_change_type.rb with the following contents:

module CoreExtensions
  module Mongoid
    module Association
      module Nested
        module One
          module EmbeddedOneChangeType
            attr_accessor :changing_type

            def build(parent)
              return if reject?(parent, attributes)

              @existing = parent.send(association.name)
              # Retrieve the new class
              new_class = attributes.dig(:_type)&.constantize
              # Resolve whether we should be changing type or not
              @changing_type = new_class && @existing.class != new_class && association.embedded?

              if update?
                attributes.delete_id
                existing.assign_attributes(attributes)
              elsif replace?
                parent.send(association.setter, ::Mongoid::Factory.build(@class_name, attributes))
              elsif delete?
                parent.send(association.setter, nil)
              end
            end

            def replace?
              # Replace if the association hasn't been set or if we are changing type.
              (!existing || changing_type) && !destroyable? && !attributes.blank?
            end

            def update?
              # Don't update the document if we are changing type
              existing && !changing_type && !destroyable? && acceptable_id?
            end
          end
        end
      end
    end
  end
end

and inside an initializer I patched the proper mongoid class with:

Mongoid::Association::Nested::One.prepend CoreExtensions::Mongoid::Association::Nested::One::EmbeddedOneChangeType

Explanation

We override the build method in order to additionally resolve if we are changing type in an embedded association or not:

@changing_type = new_class && @existing.class != new_class && association.embedded?

We are changing type if:

there’s a _type attribute present and
the _type’s value is different than the existing object’s one and
the association is embedded

Then we alter the methods that decide whether a document should be updated or replaced, so that they take in consideration if we are @changing_type (as calculated above) or not:

def update?
  # Don't update the document if we are changing type
  existing && !changing_type && !destroyable? && acceptable_id?
end

and

def replace?
  # Replace if the association hasn't been set or if we are changing type.
  (!existing || changing_type) && !destroyable? && !attributes.blank?
end

Feedback and comments

I am not very fond of monkey patching but this was my quick workaround. If you believe that the desired behavior as described here (for the `embedded_one` associations) is invalid or if you deal with this problem in any other way, I'd be glad to here about it.

For suggestions, feedback, comments, typos etc. please use this issue on GitHub.

Thanks for visiting!

Almost forgot, cat photo.

How to alias and navigate to directories with autocomplete in Linux

Wed, 10 Apr 2019 16:00:00 +0300

I use the terminal a lot and in my day to day work I tend to navigate to the same bunch of directories.

There are some awesome tools out there (like autojump or z) but sometimes, especially when the directories are similarly named, there is a need to be explicit to navigate to the proper one.

I decided to write a script to overcome this issue and to avoid having to edit my .bash* files to manage aliases each time I wanted to add or remove a directory.

goto

goto is a shell utility to quickly navigate to aliased directories with autocomplete (tab completion).

goto repo at GitHub

User registers directory aliases, for example:

goto --register dev /home/iridakos/development

and then cds to that directory with:

goto dev

Find below documentation on the script but make sure you check the script’s documentation page for updates.

Sorry for the name, I know it brings back memories but it’s not what it seems :)

Installation

To install goto all you have to do is clone the repository locally:

git clone https://github.com/iridakos/goto.git

navigate to it

cd goto

and execute:

sudo ./install

You need to restart your shell after installation.

Usage

Change to an aliased directory
Register an alias
Unregister an alias
List aliases
Expand an alias
Cleanup
Help
Version
Extras
- Push before changing directories
- Revert to a pushed directory

Change to an aliased directory

To change to an aliased directory, type:

goto <alias>

Example

goto dev

Register an alias

To register a directory alias, type:

goto -r <alias> <directory>

goto --register <alias> <directory>

Example

goto -r blog /mnt/external/projects/html/blog

goto --register blog /mnt/external/projects/html/blog

Notes

goto expands the directories hence you can easily alias your current directory with:
```
goto -r last_release .
```
and it will automatically be aliased to the whole path.
Pressing the tab key after the alias name, you have the default directory suggestions by the shell.

Unregister an alias

To unregister an alias, use:

goto -u <alias>

goto --unregister <alias>

Example

goto -u last_release

goto --unregister last_release

Notes

Pressing the tab key after the command (-u or --unregister), the completion script will prompt you with the list of registered aliases for your convenience.

List aliases

To get the list of your currently registered aliases, use:

goto -l

goto --list

Expand an alias

To expand an alias to its value, use:

goto -x <alias>

goto --expand <alias>

Example

goto -x last_release

goto --expand last_release

Cleanup

To cleanup the aliases from directories that are no longer accessible in your filesystem, use:

goto -c

goto --cleanup

Help

To view the tool’s help information, use:

goto -h

goto --help

Version

To view the tool’s version, use:

goto -v

goto --version

Extras

Push before changing directories

To first push the current directory onto the directory stack before changing directories, type:

goto -p <alias>

goto --push <alias>

Revert to a pushed directory

To return to a pushed directory, type:

goto -o

goto --pop

Notes

This command is equivalent to popd, but within the goto command.

Behind the scenes

Upon installation, a line is appended to your .bashrc file which sources the script, registering a goto function responsible for changing the directory based on the defined aliases of the database.

The script creates a directory alias database file under ~/.goto. Every register or unregister action modifies this file.

The script also uses the Bash programmable completion feature to define tab completions resolved via extraction of the aliases defined in the goto database file.

That’s all! Cat photo.

Code and comments

You can find the code of goto at https://github.com/iridakos/goto.

For feedback, comments, typos etc. please open an issue in the repository.

Thanks for visiting!

Dockerizing a Rails application

Sun, 07 Apr 2019 04:00:00 +0300

Hey!

In this post we are going to:

create a docker image for the Rails chat application that we created in the previous post
configure the Docker environment and run the application by:
- creating a container for the PostgreSQL database
- creating a container for the Redis server
- creating a container with the required configuration from the image we built

Prerequisites

Install docker

The first thing you need in order to follow this tutorial is to install Docker on your machine.

We are going to use the Docker Community Edition. Follow the installation instructions matching your system.

I’m on Ubuntu 18.04 LTS so following the instructions, I had to:

Uninstall previous versions with:

sudo apt-get remove docker docker-engine docker.io containerd runc

I chose to install the application using the repository.

sudo apt-get update

sudo apt-get install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg-agent \
     software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

and I verified that I had the key with the proper fingerprint after executing:

sudo apt-key fingerprint 0EBFCD88

I set up the repository with:

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

The installation took place with these commands:

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

I checked that the installation was successful with:

$ sudo docker run hello-world

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:...
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Important note

Upon installation, a new user group was created with the name docker. If you want to allow users to do docker stuff without using sudo, you have to do some extra configuration (read more here) but keep in mind that:

The docker group grants privileges equivalent to the root user. For details on how this impacts security in your system, see Docker Daemon Attack Surface. – Post-installation steps for Linux - Official Docker installation instructions

We are going to use sudo in this tutorial.

Clone the rails chat tutorial from GitHub

Navigate to your development directory on your machine and clone the sample Rails chat application with:

git clone https://github.com/iridakos/rails-chat-tutorial

Done.

Configure the application to use PostgreSQL for the production environment

The application is configured to use the predefined sqlite database adapter. For the purpose of this tutorial we will change the adapter to postgresql and at the second part of the post we will create a container running the PostgreSQL database.

Open the file config/database.yml file and change the production configuration as described below:

production:
  <<: *default
  adapter: postgresql
  host: <%= ENV.fetch('https://iridakos.com/DATABASE_HOST') %>
  port: <%= ENV.fetch('https://iridakos.com/DATABASE_PORT') %>
  username: <%= ENV.fetch('https://iridakos.com/DATABASE_USERNAME') %>
  password: <%= ENV.fetch('https://iridakos.com/DATABASE_PASSWORD') %>

Open the application’s Gemfile and add the following lines:

group :production do
  gem 'pg'
end

to install the adapter. The pg gem requires to have the package libpq-dev installed on the machine. We will satisfy this requirement when building the image.

Building the docker image

We are going to build the image gradually in order to understand what’s going on with every command we use.

Create the Dockerfile

Whenever we want to create a new image in docker we use a file named Dockerfile. It is a text file with instructions to be followed sequentially to assemble an image.

Navigate to the Rails chat tutorial (from now I will call this application directory) directory and create the file.

touch Dockerfile

and before continuing let’s try to build the image with just that empty file:

sudo docker build . -t rails-chat-tutorial

Of course we get an error, but take a look at the first line of the log:

Sending build context to Docker daemon  46.39MB
Error response from daemon: the Dockerfile (Dockerfile) cannot be empty

When building an image, Docker creates a build context which is actually the files that will be available when the Dockerfile’s commands get executed.

The . (dot) part of the build command that we used tells Docker to try to build the image using the current directory for its build context.

Since we didn’t explicitly specified in the command which Dockerfile to use, Docker will use the one that is located in the root of the context.

Define the parent image

A parent image is the image that your image is based on. It refers to the contents of the FROM directive in the Dockerfile. Each subsequent declaration in the Dockerfile modifies this parent image. Most Dockerfiles start from a parent image, rather than a base image. However, the terms are sometimes used interchangeably. – Create a base image - Official Docker Documentation

Building the image from scratch is out of the context of this tutorial but if you want to familiarize your self with this aspect, read more here.

Docker provides official Ruby images and we are going to use the version that the Rails chat tutorial uses which is 2.6.2 as our parent image.

Open Dockerfile and add the following line:

FROM ruby:2.6.2-stretch

and run the build command again:

$ sudo docker build . -t rails-chat-tutorial
Sending build context to Docker daemon  46.39MB
Step 1/1 : FROM ruby:2.6.2-stretch
2.6.2-stretch: Pulling from library/ruby
e79bb959ec00: Pull complete
d4b7902036fe: Pull complete
1b2a72d4e030: Pull complete
d54db43011fd: Pull complete
69d473365bb3: Pull complete
84ed2a0dc034: Pull complete
75df5efa5606: Pull complete
f0d10aea813b: Pull complete
Digest: sha256:d5af6b19da8381014f59e79245ae242dd5ea8dfe1a8a6c0e2bc481366f1e92b9
Status: Downloaded newer image for ruby:2.6.2-stretch
 ---> 8d6721e9290e
Successfully built 8d6721e9290e
Successfully tagged rails-chat-tutorial:latest

Execute the following command to see which images Docker has.

$ sudo docker image list
REPOSITORY            TAG                 IMAGE ID            CREATED             SIZE
rails-chat-tutorial   latest              8d6721e9290e        10 days ago         870MB
ruby                  2.6.2-stretch       8d6721e9290e        10 days ago         870MB
hello-world           latest              fce289e99eb9        3 months ago        1.84kB

The hello-world is the Docker’s image that we used after installing Docker. The other image, the ruby one is the parent image of our image. Since the Dockerfile didn’t have any custom instructions other that just defining a parent image, the resulting image rails-chat-tutorial is actually the same as the parent image and has the same IMAGE ID, CREATED and SIZE properties. Time to change this.

Copying the application code

The purpose of the image that we are building is to serve the Rails chat tutorial application. Eventually, to do so it’s pretty obvious that the image must contain the code of the application.

We will use the COPY command to copy the code inside the image.

Open the Dockerfile and append the following line:

COPY . /application

This command will copy all files from inside the build context to the image.

the first argument is the location of the build context to be copied
the second argument is the target location inside the image

Build the image again and execute the following command to confirm that we are good.

docker run -i -t rails-chat-tutorial

If you take a look at the ruby’s docker image, you will see that the last line is:

CMD [ "irb" ]

Since we don’t define something different in our image, the same command is being executed and that’s why the execution of the previous command brought us to Ruby’s irb console.

Let’s see what the /application directory of the container has.

Dir['/application/*']
=> ["/application/config.ru", "/application/Rakefile", "/application/lib", "/application/storage", "/application/test", "/application/Gemfile", "/application/app", "/application/Gemfile.lock", "/application/LICENSE", "/application/log", "/application/public", "/application/tmp", "/application/vendor", "/application/bin", "/application/README.md", "/application/config", "/application/package.json", "/application/Dockerfile", "/application/db"]

Cool, the application directory has been copied. Moving on.

Install application dependencies

Before starting the server (puma in our case), we have to install the dependencies of the application. To do so, add the following line to the Dockerfile.

RUN bundle install --deployment --without development test

and rebuild the image.

build . -t rails-chat-tutorial
Sending build context to Docker daemon  46.39MB
Step 1/3 : FROM ruby:2.6.2-stretch
 ---> 8d6721e9290e
Step 2/3 : COPY . /application
 ---> f9b9d813d6a0
Step 3/3 : RUN bundle install --deployment --without development test
 ---> Running in a6c8c25da3f5
Could not locate Gemfile
The command '/bin/sh -c bundle install --deployment --without development test' returned a non-zero code: 10

We have an error because for the bundle command to succeed we must first change to the application’s root directory that does contain the Gemfile file.

Change the contents of the Dockerfile to the following:

FROM ruby:2.6.2-stretch

# Copy application code
COPY . /application
# Change to the application's directory
WORKDIR /application

# Install gems
RUN bundle install --deployment --without development test

and rebuild. Now the gems are being installed and we are ready to start the server.

$ docker build . -t rails-chat-tutorial
Sending build context to Docker daemon  46.39MB
Step 1/4 : FROM ruby:2.6.2-stretch
 ---> 8d6721e9290e
Step 2/4 : COPY . /application
 ---> b1aae569faf4
Step 3/4 : WORKDIR /application
 ---> Running in ab90edf73be5
Removing intermediate container ab90edf73be5
 ---> 6bbdaa9942e3
Step 4/4 : RUN bundle install --deployment --without development test
 ---> Running in 22724a3684fe
The dependency tzinfo-data (>= 0) will be unused by any of the platforms Bundler is installing for. Bundler is installing for ruby but the dependency is only for x86-mingw32, x86-mswin32, x64-mingw32, java. To add those platforms to the bundle, run bundle lock --add-platform x86-mingw32 x86-mswin32 x64-mingw32 java.
Fetching gem metadata from https://rubygems.org/............
Fetching rake 12.3.2
Installing rake 12.3.2
Fetching concurrent-ruby 1.1.5
...
...
...
Fetching sqlite3 1.4.0
Installing sqlite3 1.4.0 with native extensions
Fetching uglifier 4.1.20
Installing uglifier 4.1.20
Bundle complete! 21 Gemfile dependencies, 69 gems now installed.
Gems in the groups development and test were not installed.
Bundled gems are installed into ./vendor/bundle
...
...
...

Removing intermediate container 22724a3684fe
 ---> d0d3163a8cca
Successfully built d0d3163a8cca
Successfully tagged rails-chat-tutorial:latest

Asset compilation

In the production environment, assets have to be pre-compiled.

We will add this task in our ENTRYPOINT script (see below) because during asset compilation Rails initializes the application and if we executed the task upon building the image, the initialization would fail since some components (like database connection, configuration of services based on environment variables like cable.yml) are not available.

We must install a Javascript environment in the container though or else when the time comes and the task is executed, will get the following error:

ExecJS::RuntimeUnavailable: Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes.

We will install nodejs, add the following line:

RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \
    apt install -y nodejs

Rebuild and you should be fine.

Start the server

We want to start the application in the production environment. Rails can resolve this via the environment variable RAILS_ENV.

To set it in the image, add in Dockerfile:

ENV RAILS_ENV production

The last instruction in the Dockerfile will be our ENTRYPOINT.

Add the following line:

ENTRYPOINT ./entrypoint.sh

What’s left to do is to configure the entrypoint.sh script to do the following:

compile the assets
start the server

Create a file named entrypoint.sh in the root application directory and add:

# Compile the assets
bundle exec rake assets:precompile

# Start the server
bundle exec rails server

This file has to be executable, so in your terminal:

chmod +x ./entrypoint.sh

That’s all.

Since there are no more modifications to be done in our Dockerfile, make sure its contents are the following:

FROM ruby:2.6.2-stretch

# Copy application code
COPY . /application
# Change to the application's directory
WORKDIR /application

# Install gems
RUN bundle install --deployment --without development test

# Set Rails environment to production
ENV RAILS_ENV production

RUN curl -sL https://deb.nodesource.com/setup_10.x | bash - \
    && apt install -y nodejs

# Start the application server
ENTRYPOINT ./entrypoint.sh

Compacting the Dockerfile

We will optimize the Dockerfile since each RUN command creates a new image (read more here).

We will merge the RUN commands in one and move the ENV command just before it. The resulting Dockerfile is:

FROM ruby:2.6.2-stretch

# Copy application code
COPY . /application
# Change to the application's directory
WORKDIR /application

# Set Rails environment to production
ENV RAILS_ENV production

# Install gems, nodejs and precompile the assets
RUN bundle install --deployment --without development test \
    && curl -sL https://deb.nodesource.com/setup_10.x | bash - \
    && apt install -y nodejs

# Start the application server
ENTRYPOINT ['./entrypoint.sh']

Running the application in the Docker environment

Creating the PostgreSQL container

We are going to create a container for the PostgreSQL database.

We need to specify two environment variables to configure a user for our application: POSTGRES_USER and POSTGRES_PASSWORD. The values of these environment variables will be used later on when creating the application’s container.

sudo docker run --name rails-chat-tutorial-pg
            -e POSTGRES_USER=postgres
            -e POSTGRES_PASSWORD=postgres
            -p 5432:5432
            -d postgres

Since the image doesn’t exist locally, Docker will fetch it from the Official Docker images and then it will create a container, binding the PostgreSQL default port 5432 to the same port of the host.

Notes: I suggest you read this documentation if you want to familiarize yourself with the options you have for customizing the container (volumes/database configuration etc).

Creating the Redis container

To create the redis container, all we have to do is run the following command:

sudo docker run --name rails-chat-tutorial-redis \
            -p 6379:6379 \
            -d redis

Again, since the image doesn’t exist locally, Docker will fetch it from the Official Docker images and then it will create a container, binding its 6376 to the same port of the host.

Creating the application’s container

At this point, your docker running containers should look like this:

$ sudo docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
<a container id>    redis               "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        0.0.0.0:6379->6379/tcp   rails-chat-tutorial-redis
<a container id>    postgres            "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        0.0.0.0:5432->5432/tcp   rails-chat-tutorial-pg

The Rails chat tutorial in production mode needs the following environmental variables:

database.yml
- DATABASE_HOST: 172.17.0.1
- DATABASE_PORT: 5432
- DATABASE_USERNAME: postgres
- DATABASE_PASSWORD: postgres
cable.yml
- REDIS_URL: redis://172.17.0.1:6379/1

To create the container for the image that we created in this post passing the required environment variables, use:

sudo docker run --name rails-chat-tutorial-web \
            -e DATABASE_HOST=172.17.0.1 \
            -e DATABASE_PORT=5432 \
            -e DATABASE_USERNAME=postgres \
            -e DATABASE_PASSWORD=postgres \
            -e REDIS_URL=redis://172.17.0.1:6379/1 \
            -p 3000:3000 \
            rails-chat-tutorial

Note: we bound the container’s 3000 port on the same port of the host.

Navigate to http://localhost:3000 and see what happens.

We don’t get very helpful information on the error since the application is running in production mode (at least this worked :P). Let’s connect to the container and check the logs:

$ sudo docker exec -it rails-chat-tutorial-web bash

Now you are connected to the container. Check the /application/production.log file. Somewhere among all these lines you will see the following:

ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation users does not exist

We set up the database server but we didn’t create/migrate the database. Since we are already connected to the container we will execute the required rake tasks.

bundle exec rake db:create db:migrate

Reload the page and voilà

What’s next

In the next tutorial we are going to:

refactor this image for production (use volume for postgreSQL, asset precompilation)
use nginx to serve them
use docker compose to bind them all.

Thanks

I am very grateful for your feedback (like this one from DeusOtiosus @ Reddit).

That’s all! Cat photo.

Code and comments

You can find the code of this tutorial on https://github.com/iridakos/rails-chat-tutorial on branch docker.

For feedback, comments, typos etc. please open an issue in the repository.

Thanks for visiting!

Creating a chat application from scratch using Rails and WebSockets

Thu, 04 Apr 2019 23:00:00 +0300

Hey! It’s been a while since my last post.

I recently familiarized myself with the awesomeness of WebSockets and I finally found the time to write a tutorial about it. I hope you find it helpful.

Update: I also published another post for dockerizing the application of this tutorial, you can find it here.

Introduction

In this tutorial we are going to create a chat web application from scratch using Rails and WebSockets.

Code and comments

You can find the code of this tutorial on GitHub.

For feedback, comments, typos etc. please open an issue in the repository.

What are WebSockets

WebSocket is actually a protocol that enables bidirectional communication between the client and the server of a web application over a single long living TCP connection.

The WebSocket protocol enables interaction between a web browser (or other client application) and a web server with lower overheads, facilitating real-time data transfer from and to the server.

This is made possible by providing a standardized way for the server to send content to the client without being first requested by the client, and allowing messages to be passed back and forth while keeping the connection open. In this way, a two-way ongoing conversation can take place between the client and the server.

The communications are done over TCP port number 80 (or 443 in the case of TLS-encrypted connections), which is of benefit for those environments which block non-web Internet connections using a firewall. Similar two-way browser-server communications have been achieved in non-standardized ways using stopgap technologies such as Comet. – WebSocket @ Wikipedia

Why WebSockets

Suppose you have to create a web page that shows the statuses of running processes. Without WebSockets you would have to either:

Use AJAX with Javascript intervals to request and render the latest state of the processes or
Automatically reload the page every x seconds (<meta http-equiv="refresh" content="x">) or
Add a message on the page “The statuses are not updated automatically ¯\_(ツ)_/¯ Press here to reload the page.”

All of these methods would request the process statuses from the server even if nothing has changed.

WebSockets are here to allow this communication to take place on demand. The cost is having to keep alive TCP connections between the server and all its clients (each for every open browser tab).

Building the application

We are going to build the web application using:

Ruby: version 2.6.2
Rails: version 5.2.3

Setting up the environment

We are going to install the proper ruby and rails versions.

Install ruby

I use rvm to manage the Ruby versions installed on my system. To install the desired ruby version use:

rvm install ruby 2.6.2

Install rails

Create a directory in your system with the name rails-chat-tutorial.

Navigate to that directory and create the following two files:

.ruby-version

ruby-2.6.2

.ruby-gemset

rails-chat-tutorial

With these file we are letting rvm know that when working on this directory, we want to use the specific ruby version (.ruby-version) and the gems from the specific gemset (.ruby-gemset)

Now, re-entering in the directory you should see something like this:

$ cd .

ruby-2.6.2 - #gemset created /home/iridakos/.rvm/gems/ruby-2.6.2@rails-chat-tutorial
ruby-2.6.2 - #generating rails-chat-tutorial wrappers...........

Install the desired rails version with:

gem install rails -v 5.2.3

Create the rails application

We are ready to create our new rails application:

rails new .

Note: We didn’t define a name for the application and rails will resolve it using the directory name: rails-chat-tutorial.

Rails will create all the application’s files and install the required gems.

Let’s start the application to make sure that everything is fine.

rails server

You should see something like:

=> Booting Puma
=> Rails 5.2.3 application starting in development
=> Run `rails server -h` for more startup options
Puma starting in single mode...
* Version 3.12.1 (ruby 2.6.2-p47), codename: Llamas in Pajamas
* Min threads: 5, max threads: 5
* Environment: development
* Listening on tcp://localhost:3000
Use Ctrl-C to stop

Open a browser and visit http://localhost:3000, if you see this we are good to go.

Users and devise

We are going to use the awesome devise solution for authentication.

Append the following gem requirement at the bottom of the Gemfile file located at the root of the application’s directory.

gem 'devise'

On your terminal, install the new gem by executing:

bundle

Finish integration with devise using:

rails generate devise:install

We will create the model representing our users using the devise generators.

On your terminal, execute:

rails generate devise User username:string

Note: we have added an extra attribute username to our model (besides the defaults generated by devise) so that we have something more friendly to present when displaying users instead of their email.

Open the generated migration which you will find under db/migrate/<datetime>_devise_create_users.rb and append the username’s unique index definition with^[4]:

  add_index :users,  :username,             unique: true

Find the line in the file that defines the username column and change it to:

t.string :username, null: false

to make the attribute required.

Then in the User model which is located at app/models/user.rb add the validation rule for uniqueness and presence:

  validates :username, uniqueness: true, presence: true

Finally, apply the database migration using:

rails db:migrate

Rooms and messages

Each chat message is going to take place in the context of a room.

Let’s build them all.

Use the following command to create the Room:

rails generate resource Room name:string:uniq

and the following command to create the RoomMessage:

rails generate resource RoomMessage room:references user:references message:text

We are now going to define the appropriate relations^[7].

Open app/models/room.rb and add the relation inside the class:

has_many :room_messages, dependent: :destroy,
                         inverse_of: :room

Open app/models/room_message.rb and add the relations inside the class:

belongs_to :user
belongs_to :room, inverse_of: :room_messages

Migrate the database with:

rails db:migrate

We can now setup our routes so that the root request is served by the RoomsController#index action.

Open your config/routes.rb file and change its contents to:

Rails.application.routes.draw do
  devise_for :users

  root controller: :rooms, action: :index

  resources :room_messages
  resources :rooms
end

Restart the server and try to navigate to the application’s root url.

You should see an error message, no worries:

We have to create the index action in the RoomsController. Open the controller app/controllers/rooms_controller.rb and change its contents to the following:

class RoomsController < ApplicationController
  def index
  end
end

Then create the file app/views/rooms/index.html.erb and for now just add the following:

<h1>Rooms index</h1>

Reload and voilà.

Adding authentication

We want all users to be authenticated before start chatting, so we are going to add the following line in the ApplicationController located at app/controllers/application_controller.rb:

  before_action :authenticate_user!

If we navigate to http://localhost:3000 now we should be redirected to the sign in page ^[10].

Before continuing with the good stuff, let’s gear up the application with some good features.

Add bootstrap

We are going to use Bootstrap and we will integrate it in the application using the bootstrap-rubygem gem.

Following the instructions of the gem, append the dependencies in your Gemfile.

gem 'bootstrap', '~> 4.3.1'
gem 'jquery-rails'

and execute bundle to fetch and install it.

Change the app/assets/stylesheets/application.css file’s extension to scss and replace its contents with:

@import "bootstrap";

Add the following lines to the app/assets/javascript/application.js just before the //= require_tree . line^[9]:

//= require jquery3
//= require popper
//= require bootstrap-sprockets

Add simple_form

We are going to use this great gem to generate forms easily.

Append the gem dependency in your Gemfile and bundle to install it.

gem 'simple_form'

Then complete the integration using:

rails generate simple_form:install --bootstrap

Note: We used the –bootstrap directive since that’s the framework we are using.

Devise views with bootstrap and simple form

Devise uses its own views for sign in, register etc. But we do have a way to customize these views and now that we have ended up using bootstrap and simple forms, we can generate these views in a way that our choices are respected.

In your terminal:

rails generate devise:views

The view for signing in is under app/views/devise/sessions/new.html.erb and for signing up is under app/views/devise/registrations/new.html.erb. Open these two files and change the submit button’s class by replacing the following line^[6]:

<%= f.button :submit, "Sign up" %>

with

<%= f.button :submit, "Sign up", class: 'btn btn-success' %>

to render the buttons bootstrap style.

Before viewing our changes, let’s do one last thing in our default layout.

Open app/views/layouts/application.html.erb and replace its contents with:

<!DOCTYPE html>
<html>
  <head>
    <title>RailsChatTutorial</title>
    <%= csrf_meta_tags %>
    <%= csp_meta_tag %>

    <%= stylesheet_link_tag    'application', media: 'all', 'data-turbolinks-track': 'reload' %>
    <%= javascript_include_tag 'application', 'data-turbolinks-track': 'reload' %>
  </head>

  <body>
    <div class="container">
      <div class="row">
        <div class="col-12">
          <%= yield %>
        </div>
      </div>
    </div>
  </body>
</html>

This last one was to use Bootstrap’s grid in our views.

Navigate to http://localhost:3000 and view what we have created.

Let’s try to sign up following the Sign up link of the form:

As you can see, there is no field to fill in the username. For that to work we have to:

Add the field in the sign up form
Configure devise to accept the new attribute (username) or else the ApplicationController will ignore it once submitted from the form.

To add the field in the sign up form, open app/views/devise/registrations/new.html.erb and add these lines between the email and password fields.

  <%= f.input :username,
              required: true %>

Then, open the app/controllers/application_controller.rb file to configure the new attribute. Change the contents to:

class ApplicationController < ActionController::Base
  before_action :authenticate_user!

  before_action :configure_permitted_parameters, if: :devise_controller?

  protected

  def configure_permitted_parameters
    devise_parameter_sanitizer.permit(:sign_up, keys: [:email, :username])
  end
end

Done, reload and sign up^[5].

Cleanup unused components

We will not be using coffee script or turbolinks so let’s remove all the related stuff.

Open Gemfile and remove the following lines:

# Use CoffeeScript for .coffee assets and views
gem 'coffee-rails', '~> 4.2'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'

Open app/assets/javascripts/application.js and remove the following line:

//= require turbolinks

Open app/views/layouts/application.html.erb and change the following lines ^[3]:

    <%= stylesheet_link_tag    'application', media: 'all', 'data-turbolinks-track': 'reload' %>
    <%= javascript_include_tag 'application', 'data-turbolinks-track': 'reload' %>

    <%= stylesheet_link_tag    'application', media: 'all' %>
    <%= javascript_include_tag 'application' %>

Check that your app/assets/javascripts folder doesn’t have any files with extension .coffee and if you find any, remove them.²

In the terminal, also execute the following command:

rails tmp:cache:clear

to clear any cached compiled coffee scripts.

Done. Restart you server.

To improve the usability of the web pages will add a top navigation bar.

Create the directory app/views/shared and a file inside it named _navigation_bar.html.erb. This is going to be the partial responsible for rendering the navigation bar and which later on we will add to the application’s default layout in order to be rendered on all web pages. Add these contents:

<nav class="navbar navbar-expand-lg navbar-dark bg-dark justify-content-between">
  <a class="navbar-brand" href="#">Rails chat tutorial</a>
  <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#nav-bar-collapse" aria-controls="navbarColor01" aria-expanded="false" aria-label="Toggle navigation">
    <span class="navbar-toggler-icon"></span>
  </button>

  <% if current_user %>
    <div class="dropdown">
      <a class="nav-link dropdown-toggle" href="#" id="navbarDropdown" role="button" data-toggle="dropdown" aria-haspopup="true" aria-expanded="false">
        <img class="avatar" src="<%= gravatar_url(current_user) %>">
        <%= current_user.username %>
      </a>

      <div class="dropdown-menu dropdown-menu-right" aria-labelledby="navbarDropdown">
        <%= link_to 'Logout', destroy_user_session_path, method: :delete, class: 'dropdown-item' %>
      </div>
    </div>
  <% end %>
</nav>

Mind the gravatar_url(https://iridakos.com/current_user) line. This is a helper method that we are going to use in order to resolve the gravatar url of the signed in user. This is not a builtin method, we have to defined it but it’s pretty straightforward.

Edit app/helpers/application_helper.rb and add the following method:

def gravatar_url(user)
  gravatar_id = Digest::MD5::hexdigest(user.email).downcase
  url = "https://gravatar.com/avatar/#{gravatar_id}.png"
end

Note:

As you can see, the user’s username, avatar and sign out link will only be rendered if the user is signed in.

The avatar image has a css class avatar. We have to define this class in the application’s stylesheets. Create a css file in which we will gather all css class that we will use in the application under the name app/assets/stylesheets/rails-chat-tutorial.scss.

For now add the rule for the avatar:

.avatar {
  max-height:30px;
  border-radius: 15px;
  width:auto;
  vertical-align:middle;
}

and open application.scss to import the newly created stylesheet. Add the line:

@import "rails-chat-tutorial"

We have to add this partial in the application layout. Edit app/views/layouts/application.html.erb and change its contents to:

<!DOCTYPE html>
<html>
  <head>
    <title>RailsChatTutorial</title>
    <%= csrf_meta_tags %>
    <%= csp_meta_tag %>

    <%= stylesheet_link_tag    'application', media: 'all' %>
    <%= javascript_include_tag 'application' %>
  </head>

  <body>
    <div class="container">
      <div class="row">
        <div class="col-12">
          <%= render partial: 'shared/navigation_bar' %>
          <div class="my-3">
            <%= yield %>
          </div>
        </div>
      </div>
    </div>
  </body>
</html>

Reload to view the bar.

Awesome. Fill in with your desired credentials and submit the form.

Room management

We are going to create a simple layout for groups.

One narrow column displaying vertically all the available rooms
One wide column which is going to host the chat messages and form.

The rooms index page will have the second column empty since this column will be present only when user is inside a specific room.

In the index page we will provide the option to create a room.

Room index

First we have to load all rooms in the RoomsController. Open app/controllers/rooms_controller.rb and change the index action as:

def index
  @rooms = Room.all
end

Open app/views/rooms/index.html.erb and change its contents to^[8]:

<div class="row">
  <div class="col-12 col-md-3">
    <div class="mb-3">
      <%= link_to new_room_path, class: "btn btn-primary" do %>
        Create a room
      <% end %>
    </div>

    <% if @rooms.present? %>
      <nav class="nav flex-column">
        <% @rooms.each do |room| %>
          <%= link_to room.name, room_path(room), class: "nav-link room-nav-link" %>
        <% end %>
      </nav>
    <% else %>
      <div class="text-muted">
        The are no rooms
      </div>
    <% end %>
  </div>

  <div class="col">
    <div class="alert alert-primary">
      <h4 class="alert-heading">
        Welcome to the RailsChatTutorial!
      </h4>

      <p>
        We need to talk.
      </p>

      <hr />

      <p>
        You can create or join a room from the sidebar.
      </p>
    </div>
</div>

If there are rooms, the left column of the page will render a vertical navigation with links leading to each room’s page. The right column displays a simple welcome message.

Pressing the Create a room button we get the expected error for the non-existent action.

Room new/edit

We have to define the actions for creating and updating a room.

Open the app/controllers/rooms_controller.rb and change its contents to:

class RoomsController < ApplicationController
  # Loads:
  # @rooms = all rooms
  # @room = current room when applicable
  before_action :load_entities

  def index
    @rooms = Room.all
  end

  def new
    @room = Room.new
  end

  def create
    @room = Room.new permitted_parameters

    if @room.save
      flash[:success] = "Room #{@room.name} was created successfully"
      redirect_to rooms_path
    else
      render :new
    end
  end

  def edit
  end

  def update
    if @room.update_attributes(permitted_parameters)
      flash[:success] = "Room #{@room.name} was updated successfully"
      redirect_to rooms_path
    else
      render :new
    end
  end

  protected

  def load_entities
    @rooms = Room.all
    @room = Room.find(params[:id]) if params[:id]
  end

  def permitted_parameters
    params.require(:room).permit(:name)
  end
end

Note: we preload the @rooms and the @room variables making them available to all actions with the before_action :load_entities hook.

We will create a simple form for the Room object and we will use it both when creating and editing a room. Create the app/views/rooms/_form.html.erb and add:

<%= simple_form_for @room do |form| %>
  <%= form.input :name %>
  <%= form.submit "Save", class: 'btn btn-success' %>
<% end %>

Then, create the views for the new/edit action accordingly:

app/views/rooms/new.html.erb

<h1>
  Creating a room  
</h1>

<%= render partial: 'form' %>

app/views/rooms/edit.html.erb

<h1>
  Editing room <%= @room.name %>
</h1>

<%= render partial: 'form' %>

Time to create the first room. From the rooms’ index page, press the Create a room

Save and here it is.

Add this class in app/assets/stylesheets/rails-chat-tutorial.scss to improve the display of the rooms.

.room-nav-link {
  border: 1px solid lighten($primary, 40%);
  background: lighten($primary, 45%);

  & + .room-nav-link {
    border-top: 0 none;
  }
}

Note: We will add the edit link in the room’s page a.k.a. show action.

Before moving on to the Room page, we will refactor the index page so as to be able to use the left column’s content inside the room page as well.

Create the partial app/views/rooms/_rooms.html.erb with contents:

<div class="mb-3">
  <%= link_to new_room_path, class: 'btn btn-primary' do %>
    Create a room
  <% end %>
</div>

<% if @rooms.present? %>
  <nav class="nav flex-column">
    <% @rooms.each do |room| %>
      <%= link_to room.name, room_path(room), class: 'nav-link room-nav-link' %>
    <% end %>
  </nav>
<% else %>
  <div class="text-muted">
    The are no rooms
  </div>
<% end %>

and change the app/views/rooms/index.html.erb to use it:

<div class="row">
  <div class="col-12 col-md-3">
    <%= render partial: 'rooms' %>
  </div>

  <div class="col">
    <div class="alert alert-primary">
      <h4 class="alert-heading">
        Welcome to the RailsChatTutorial!
      </h4>

      <p>
        We need to talk.
      </p>

      <hr />

      <p>
        You can create or join a room from the sidebar.
      </p>
    </div>
  </div>
</div>

Room page

Add the show action in the app/controllers/rooms_controller.rb:

def show
  @room_message = RoomMessage.new room: @room
  @room_messages = @room.room_messages.includes(:user)
end

Notes:

We construct a new room message which we are going to use in the view to build a form for creating the chat messages.
When displaying the room message, we access its user’s email attribute to resolve the gravatar hash. We used .includes(:user) in the query for the @room_messages to fetch them along with their users avoiding N+1 queries^[1].

Create the view app/views/rooms/show.html.erb:

<h1>
  <%= @room.name %>
</h1>

<div class="row">
  <div class="col-12 col-md-3">
    <%= render partial: 'rooms' %>
  </div>

  <div class="col">
    <div class="chat">
      <% @room_messages.each do |room_message| %>
        <%= room_message %>
      <% end %>
    </div>

    <%= simple_form_for @room_message, remote: true do |form| %>
      <div class="input-group mb-3">
        <%= form.input :message, as: :string,
                                 wrapper: false,
                                 label: false,
                                 input_html: {
                                   class: 'chat-input'
                                 } %>
        <div class="input-group-append">
          <%= form.submit "Send", class: 'btn btn-primary chat-input' %>
        </div>
      </div>

      <%= form.input :room_id, as: :hidden %>
    <% end %>
  </div>
</div>

Notes:

We reused the app/views/rooms/_rooms.html.erb partial that we created in the previous step
We added a div with class .chat and this is where the room’s messages are rendered.
We added a form for the @room_message that we instantiated in the controller. We also used the directive remote: true when we instantiated the form thus the form is going to be submitted by Ajax.
We added a hidden field for the attribute :room_id so that the value reaches the RoomMessagesController once we submit the form.

Style the chat components by adding the following lines to the app/assets/stylesheets/rails-chat-tutorial.scss:

.chat {
  border: 1px solid lighten($secondary, 40%);
  background: lighten($secondary, 50%);
  height: 50vh;
  border-radius: 5px 5px 0 0;
  overflow-y: auto;
}

.chat-input {
  border-top: 0 none;
  border-radius: 0 0 5px 5px;
}

Navigate to a room to see what has been done.

Pressing the Send button nothing happens on the page but if you check the server’s console you will notice:

AbstractController::ActionNotFound (The action 'create' could not be found for RoomMessagesController):

Let’s fix that.

Creating room messages

This is going to be easy. All we have to do is implement the create action in the RoomMessagesController.

app/controllers/room_messages_controller.rb

class RoomMessagesController < ApplicationController
  before_action :load_entities

  def create
    @room_message = RoomMessage.create user: current_user,
                                       room: @room,
                                       message: params.dig(:room_message, :message)
  end

  protected

  def load_entities
    @room = Room.find params.dig(:room_message, :room_id)
  end
end

Notes:

we preload the room using the room_id parameter that we added as a hidden field in the form in the previous step
we create a new message for the room setting its user to the currently signed in user

If you try to submit a message now, again you will see nothing but in the server console you can see from the log that the room message has been created.

Started POST "/room_messages" for ::1 at 2019-04-04 19:24:33 +0300
Processing by RoomMessagesController#create as JS
  Parameters: {"utf8"=>"✓", "room_message"=>{"message"=>"My first message", "room_id"=>"8"}, "commit"=>"Send"}
  User Load (0.2ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = ? ORDER BY "users"."id" ASC LIMIT ?  [["id", 1], ["LIMIT", 1]]
  ↳ /home/iridakos/.rvm/gems/ruby-2.6.2@rails-chat-tutorial/gems/activerecord-5.2.3/lib/active_record/log_subscriber.rb:98
  Room Load (0.2ms)  SELECT  "rooms".* FROM "rooms" WHERE "rooms"."id" = ? LIMIT ?  [["id", 8], ["LIMIT", 1]]
  ↳ app/controllers/room_messages_controller.rb:13
   (0.1ms)  begin transaction
  ↳ app/controllers/room_messages_controller.rb:5
  RoomMessage Create (0.7ms)  INSERT INTO "room_messages" ("room_id", "user_id", "message", "created_at", "updated_at") VALUES (?, ?, ?, ?, ?)  [["room_id", 8], ["user_id", 1], ["message", "My first message"], ["created_at", "2019-04-04 16:24:33.456641"], ["updated_at", "2019-04-04 16:24:33.456641"]]
  ↳ app/controllers/room_messages_controller.rb:5
   (4.0ms)  commit transaction
  ↳ app/controllers/room_messages_controller.rb:5
No template found for RoomMessagesController#create, rendering head :no_content
Completed 204 No Content in 88ms (ActiveRecord: 5.1ms)

A user would expect to have the message field cleared after sending a new message. We don’t disappoint users.

Create a file app/assets/javascripts/room.js and add the following:

$(function() {
  $('#new_room_message').on('ajax:success', function(a, b,c ) {
    $(this).find('input[type="text"]').val('');
  });
});

We bind to the ajax:success event triggered by Rails on successful submission of the form and all we want to do is clear the text field’s value.

Reload the page and try submitting again and check it out. The field value should be emptied after sending the message.

Displaying room messages

If you reload the page you will see something like this:

Let’s beautify the messages.

Replace the contents of app/views/rooms/show.html.erb with:

<h1>
  <%= @room.name %>
</h1>

<div class="row">
  <div class="col-12 col-md-3">
    <%= render partial: 'rooms' %>
  </div>

  <div class="col">
    <div class="chat">
      <% @room_messages.each do |room_message| %>
        <div class="chat-message-container">
          <div class="row no-gutters">
            <div class="col-auto text-center">
              <img src="<%= gravatar_url(room_message.user) %>" class="avatar" alt="">
            </div>

            <div class="col">
              <div class="message-content">
                <p class="mb-1">
                  <%= room_message.message %>
                </p>

                <div class="text-right">
                  <small>
                    <%= room_message.created_at %>
                  </small>
                </div>
              </div>
            </div>
          </div>
        </div>
      <% end %>
    </div>

    <%= simple_form_for @room_message, remote: true do |form| %>
      <div class="input-group mb-3">
        <%= form.input :message, as: :string,
                                 wrapper: false,
                                 label: false,
                                 input_html: {
                                   class: 'chat-input'
                                 } %>
        <div class="input-group-append">
          <%= form.submit "Send", class: 'btn btn-primary chat-input' %>
        </div>
      </div>

      <%= form.input :room_id, as: :hidden %>
    <% end %>
  </div>
</div>

and add the following css classes inside the .chat class:

.chat-message-container {
  padding: 5px;

  .avatar {
    margin: 5px;
  }

  .message-content {
    padding: 5px;
    border: 1px solid $primary;
    border-radius: 5px;
    background: lighten($primary, 10%);
    color: $white;
  }

  & + .chat-message-container {
    margin-top: 10px;
  }
}

Reload the page. Magic.

Introducing WebSockets - ActionCable

Time to start using WebSockets with ActionCable.

Action Cable seamlessly integrates WebSockets with the rest of your Rails application. It allows for real-time features to be written in Ruby in the same style and form as the rest of your Rails application, while still being performant and scalable. It’s a full-stack offering that provides both a client-side JavaScript framework and a server-side Ruby framework. You have access to your full domain model written with Active Record or your ORM of choice. – Action Cable Overview @ Ruby on Rails Guides (v5.2.3)

Install redis

We are going to use the redis adapter which is a safe option for production environments unlike the async one.

You first must install redis on your system.

To install it on Ubuntu you just have to execute the following commands in your terminal:

sudo apt update
sudo apt install redis-server

To check that installation is successful, in your terminal make sure you get a PONG:

$ redis-cli
127.0.0.1:6379> ping
PONG

Configure ActionCable

Since we are working on the development environment, open your config/cable.yml and replace its contents with:

development:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %>
  channel_prefix: rails-chat-tutorial_development

test:
  adapter: async

production:
  adapter: redis
  url: <%= ENV.fetch("REDIS_URL") { "redis://localhost:6379/1" } %>
  channel_prefix: rails-chat-tutorial_production

Note: we have added an option channel_prefix because:

Additionally, a channel_prefix may be provided to avoid channel name collisions when using the same Redis server for multiple applications – Action Cable Overview # Redis Adapter @ Ruby on Rails Guides (v5.2.3)

Finally, we are going to add the dependency in the Gemfile:

gem 'redis'

Don’t forget to bundle after altering the Gemfile.

Configure Devise for authenticating websocket connections

When establishing a websocket connection, we don’t have access to the user session but we do have access to the cookies. So, in order to be able to authenticate the user we need to do some devise related stuff first (credits to Greg Molnar).

Create an initializer for warden hooks under the name config/initializers/warden_hooks.rb and add the following lines:

Warden::Manager.after_set_user do |user,auth,opts|
  scope = opts[:scope]
  auth.cookies.signed["#{scope}.id"] = user.id
end

Warden::Manager.before_logout do |user, auth, opts|
  scope = opts[:scope]
  auth.cookies.signed["#{scope}.id"] = nil
end

Explain: We add a cookie with the user’s id upon successful sign in and we remove it once the user logs out.

Configure the websocket connection

Open app/channels/application_cable/connection.rb and change its contents to the following:

module ApplicationCable
  class Connection < ActionCable::Connection::Base
    identified_by :current_user

    def connect
      self.current_user = find_verified_user
    end

    private

    def find_verified_user
      if verified_user = User.find_by(id: cookies.signed['user.id'])
        verified_user
      else
        reject_unauthorized_connection
      end
    end
  end
end

Explain:

Here identified_by is a connection identifier that can be used to find the specific connection later. Note that anything marked as an identifier will automatically create a delegate by the same name on any channel instances created off the connection. – Action Cable Overview # Connection setup @ Ruby on Rails Guides (v5.2.3)

In the find_verified_user method we access the cookie that we previously set in the warden hook.

Create the room channel

A channel encapsulates a logical unit of work, similar to what a controller does in a regular MVC setup. – Action Cable Overview # Channels @ Ruby on Rails Guides (v5.2.3)

We will create the RoomChannel in which all Room pages will subscribe to.

Create app/channels/room_channel.rb with the following contents:

class RoomChannel < ApplicationCable::Channel
  def subscribed
    room = Room.find params[:room]
    stream_for room

    # or
    # stream_from "room_#{params[:room]}"
  end
end

Explain:

The subscribed method gets called once a subscription to the channel is established and it is responsible to setup the stream from which data will be sent back and forth.

We are going to configure the room page code later on to request subscriptions to this channel passing the room parameter.

We have two options:

Use stream_for: this way Rails automatically generates a stream name for the given object (room in our case), for example: “room:asdfwer234”. When we want afterwards to broadcast data to the stream, all we have to do is call RoomChannel.broadcast_to(room_object, data) in which case Rails resolves the stream name from the room_object. In other words, we don’t have to manually resolve the stream name in which the data have to be send (see next item).
- This option is available when the channel handles subscriptions bound to models like in our case, a specific room
Use stream_from: we manually define the name of the stream and later on, when we want to broadcast to the stream, we have to use: ActionCable.server.broadcast("room_#{a_room_id_here}", data).

Broadcast room messages

Every time a room message is being created, we just need to broadcast to the message’s room stream. To do so, alter the create action of the app/controllers/room_messages_controller.rb to this:

def create
  @room_message = RoomMessage.create user: current_user,
                                     room: @room,
                                     message: params.dig(:room_message, :message)

  RoomChannel.broadcast_to @room, @room_message
end

Explain: we added the line RoomChannel.broadcast_to @room, @room_message which will broadcast to the room’s specific stream (as explained above) the @room_message transformed to json via the to_json method.

So, on the other side, the client side, we are going to be receiving the json representation of the RoomMessage model. Let’s see what that is:

{
  "id":29,
  "room_id":8,
  "user_id":1,
  "message":"My first message",
  "created_at":"2019-04-04T17:09:00.637Z",
  "updated_at":"2019-04-04T17:09:00.637Z"
}

We plan to add the new messages in the room’s page via Javascript and this information is not adequate. The only thing we are missing is the user avatar. Time to refactor.

Open app/models/user.rb and add the following method:

def gravatar_url
  gravatar_id = Digest::MD5::hexdigest(email).downcase
  "https://gravatar.com/avatar/#{gravatar_id}.png"
end

We had already implemented this in the app/helpers/application_helper.rb file and we won’t be using it anymore so remove it.

Update app/views/shared/_navigation_bar.html.erb and change the previous gravatar resolution to this:

<img class="avatar" src="<%= current_user.gravatar_url %>">

Update app/views/rooms/show.html.erb and change it there too, with:

<img src="<%= room_message.user.gravatar_url %>" class="avatar" alt="">

Finally, we will change the JSON representation of the RoomMessage to include the user’s url:

app/models/room_message.rb

def as_json(options)
  super(options).merge(user_avatar_url: user.gravatar_url)
end

Let’s confirm that the new JSON transformation is valid:

{
  "id":29,
  "room_id":8,
  "user_id":1,
  "message":"My first message",
  "created_at":"2019-04-04T17:09:00.637Z",
  "updated_at":"2019-04-04T17:09:00.637Z",
  "user_avatar_url":"https://gravatar.com/avatar/02a28db6886d578f75a820b50f2dd334.png"
}

Great, moving on.

We are going to add some data in the room page in order to use them via Javascript to subscribe to the appropriate streams each time we visit a room.

Open the file app/views/rooms/show.html.erb and alter the line in which we define the chat div, to this:

<div class="chat" data-channel-subscribe="room" data-room-id="<%= @room.id %>">

Explain: We added two data attributes, one defining to which channel we want to subscribe and one defining to which room we are.

At the end of the file, add the following snippet:

<div class="d-none" data-role="message-template">
  <div class="chat-message-container">
    <div class="row no-gutters">
      <div class="col-auto text-center">
        <img src="" class="avatar" alt="" data-role="user-avatar">
      </div>

      <div class="col">
        <div class="message-content">
          <p class="mb-1" data-role="message-text"></p>

          <div class="text-right">
            <small data-role="message-date"></small>
          </div>
        </div>
      </div>
    </div>
  </div>
</div>

This snippet is going to be used as a template for each incoming message. Every time a message arrives, we will

clone this html
alter the appropriate elements’ values and
append the resulting html to the end of the chat div.

Now we will create the Javascript that will do the work of subscribing and handling incoming channel data.

Create the file app/assets/javascripts/channels/room_channel.js and add the following code:

$(function() {
  $('[data-channel-subscribe="room"]').each(function(index, element) {
    var $element = $(element),
        room_id = $element.data('room-id')
        messageTemplate = $('[data-role="message-template"]');

    $element.animate({ scrollTop: $element.prop("scrollHeight")}, 1000)        

    App.cable.subscriptions.create(
      {
        channel: "RoomChannel",
        room: room_id
      },
      {
        received: function(data) {
          var content = messageTemplate.children().clone(true, true);
          content.find('[data-role="user-avatar"]').attr('src', data.user_avatar_url);
          content.find('[data-role="message-text"]').text(data.message);
          content.find('[data-role="message-date"]').text(data.updated_at);
          $element.append(content);
          $element.animate({ scrollTop: $element.prop("scrollHeight")}, 1000);
        }
      }
    );
  });
});

Explain:

For each element that has a data attribute channel-subscribe with value room
- Create a subscription to the “RoomChannel” passing the element’s room-id data attribute as a parameter with name room (remember this line of the RoomChannel: Room.find params[:room] ?)
  - When data is received, clone the template snippet and alter its contents based on the incoming object attributes.
  - Append the newly generated content to the chat div and finally
  - Do some animation to impress by scrolling smoothly to the bottom of the div.

Acknowledgments

Thank you very much for your feedback.

[1] Armando Andini - N+1 queries
[2] Rodolfo Ruiz - Coffeescript leftovers
[3] Felix Wolfsteller - Turbolinks leftovers
[4] Maria Kravtsova - Migration typo
[5][7][8] Tony Dehnke - Sign up step, Missing step for adding model relations, Missing line from html code block
[6] keytonw - Devise view missing button class
[9] Martin - Mention order of requirements in application.js
[10] Sumak - Remove duplicate authentication related code

That’s all! Long post, tired cat photo.

Code and comments

You can find the code of this tutorial on GitHub.

For feedback, comments, typos etc. please open an issue in the repository.

Thanks for visiting!

Elasticsearch bucket aggregations and faceted navigation - facets

Mon, 22 Oct 2018 20:00:00 +0300

Introduction

I decided to write a series of tutorials about Elasticsearch aggregations. In this first post of the series, we are going to deal with the bucket aggregations that allow us to implement faceted navigation.

What is Elasticsearch

Elasticsearch is an opensource JSON-based search engine that allows us to search indexed data quickly and with options that are not provided by classic data stores.

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected. – official Elasticsearch product page

What is Kibana

Kibana is a tool mainly allowing visualization of Elasticsearch data. We will use Kibana because it also provides a very convenient way for writing and executing queries (with autocomplete).

Kibana lets you visualize your Elasticsearch data and navigate the Elastic Stack, so you can do anything from learning why you’re getting paged at 2:00 a.m. to understanding the impact rain might have on your quarterly numbers. – official Kibana product page

What are Elasticsearch bucket aggregations

Before familiarizing myself with the term aggregations in the Elasticsearch world, what I was actually trying to learn was how to implement the widely known feature of facets for my indexed data.

Chances are that you know about facets, you have seen it in many sites. They are usually placed as a sidebar in search results landing pages and they are rendered as links or checkboxes that act as filters to help you narrow the results based on their properties.

A section of Elasticsearch’s aggregations framework named bucket aggregations provides the functionality we need to implement a faceted navigation.

Bucket aggregations don’t calculate metrics over fields like the metrics aggregations do, but instead, they create buckets of documents. Each bucket is associated with a criterion (depending on the aggregation type) which determines whether or not a document in the current context “falls” into it. In other words, the buckets effectively define document sets. In addition to the buckets themselves, the bucket aggregations also compute and return the number of documents that “fell into” each bucket. – official elasticsearch bucket aggregations reference

Prerequisites

In order to be able to execute the tutorial’s commands and queries, you must first:

Install Elasticsearch (duh)

Providing instructions for installing Elasticsearch is out of scope. I recommend visiting the product’s related documenation site. After the installation, just make sure to start the service if it’s not started. You can check that with

curl http://localhost:9200

It should give something like:

{
  "name" : "blabla",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "blabla",
  "version" : {
    "number" : "6.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "04711c2",
    "build_date" : "2018-09-26T13:34:09.098244Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

As you can see, we will use Elasticsearch version 6.4.2 in this tutorial.

Install Kibana

I recommend following the official instructions for installing kibana at your machine’s OS. I followed this guide for installing the product via a repository in Ubuntu.

Start the kibana service and navigate to its home page at http://localhost:5601. We will heavily use the Dev Tools which is a powerful console talking to the Elasticsearch engine.

What is it going to be covered

We are going to deal with:

Terms aggregations
Sub-aggregations
Nested and reverse nested aggregations
Global aggregations

Don’t be afraid, everything will become very clear once we start playing with the data.

Hands on

Let the fun begin.

The model

Suppose that we want to implement a web application for city pet registrations. We will define the following entities:

City office: a city pet registration office
- city: the city in which the office is located
- office type: the type of the office (central, district)
Citizen: a citizen with registered pets
- occupation: the occupation of the citizen
- age: the age of the citizen
Pet: a citizen’s pet
- kind: the kind of the pet (cat, dog etc)
- name: the name of the pet
- age: the age of the pet

Our entities relations are:

One city office has many citizens
One citizen has many registered pets

Prepare the data

Before starting explaining the aggregations, we first have to create the index to store our data and then we have to feed this index with sample data.

Create the index

Navigate to Kibana’s Dev Tools page and execute the following request:

PUT city_offices
{
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "properties": {
        "city": {
          "type": "keyword"
        },
        "office_type": {
          "type": "keyword"
        },
        "citizens": {
          "type": "nested",
          "properties": {
            "occupation": {
              "type": "keyword"
            },
            "age": {
              "type": "integer"
            },
            "pets": {
              "type": "nested",
              "properties": {
                "kind": {
                  "type": "keyword"
                  },
                "name": {
                  "type": "keyword"
                },
                "age": {
                  "type": "integer"
                }
              }
            }
          }
        }
      }
    }
  }
}

In the output section, you should see:

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "index": "city_offices"
}

We created an index with name city_offices with the aforementioned properties.

Important note: we chose to define the entity relations as nested objects. Why?

The nested type is a specialized version of the object datatype that allows arrays of objects to be indexed in a way that they can be queried independently of each other … Lucene has no concept of inner objects, so Elasticsearch flattens object hierarchies into a simple list of field names and values – official elasticsearch nested datatype reference

I will explain this with an example.

Suppose we had a city office with two citizens. One 35 year old Dentist and one 30 year old Developer. If we used the object datatype, Elasticsearch would merge all sub-properties of the entity relation resulting to something like this:

{
  "citizens": {
    "occupation": ["Dentist", "Developer"],
    "age": ["35", "30"]
  }
}

Thus, if we wanted to search the index for offices that have a Dentist citizen with age 30, this document would fulfill the criteria even though the Dentist is 35 years old.

Mapping the relation as nested overcomes this problem since

Internally, nested objects index each object in the array as a separate hidden document, meaning that each nested object can be queried independently of the others – official elasticsearch nested datatype reference

Feed sample data

I have created some sample data with random cities, occupations, pet names etc. You can find it at this tutorial’s GitHub repo.

Download the file sample-data.json and from within the download folder execute:

curl -s -H "Content-Type: application/x-ndjson" -XPOST http://localhost:9200/_bulk --data-binary "@sample-data.json"; echo

Note: The data were generated with this ruby script. You can alter it as you please and execute it to produce your desired sample data json file.

That’s it. Aggregations time.

Aggregation request format

All aggregation queries are embedded in search requests.

GET <index_name>/_search
{
  "query": { ... },
  "aggs": {
    "<aggregation name>": {
      "<aggregation type>": { <aggregation properties> }
    }
  }
}

where:

aggregation name: is the name we give to our aggregation. This is required in order to be able to parse the search response afterwards and locate the specific aggregation results (you will understand this better after we execute the first aggregation request).
aggregation type: is the type of the aggregation we want to execute. You can browse all available aggregations provided by Elasticsearch here.
aggregation properties: are the available properties specific to the aggregation type.

Terms aggregations

We will use the Terms aggregation in order to find out how many different values our documents have in a specific field.

Navigate to Kibana’s Dev Tools section and in the console (left section of the page) type the following:

GET city_offices/_search
{
  "aggs": {
    "cities": {
      "terms": { "field": "city" }
    }
  }
}

Following the syntax described in the Aggregation request format section above

we are searching in the city_offices index
we are requesting an aggregation of type terms
we specify that we are interested in the different values of the field city

Now execute the request by pressing the play link that you should be seeing given that the query you typed is focused.

Voila!

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 1,
    "hits": [
      {
        ...
      }
    ]
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 37,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8
        },
        {
          "key": "London",
          "doc_count": 8
        },
        {
          "key": "Oslo",
          "doc_count": 8
        },
        {
          "key": "Paris",
          "doc_count": 8
        },
        {
          "key": "San Francisco",
          "doc_count": 8
        },
        {
          "key": "Tokyo",
          "doc_count": 8
        },
        {
          "key": "Athens",
          "doc_count": 7
        },
        {
          "key": "Barcelona",
          "doc_count": 7
        },
        {
          "key": "Chicago",
          "doc_count": 7
        },
        {
          "key": "Madrid",
          "doc_count": 7
        }
      ]
    }
  }
}

Since we are executing a search request, the response contains the hits attribute with all the results matching our query (in this case all documents since we didn’t add a query). It also contains though another attribute named aggregations and that’s where our aggregation results fall under.

Inside the aggregations you can see our cities aggregation and that’s why it is required to name the aggregations you define.

Before explaining the weird sum_other_doc_count 37 value, let’s first examine the buckets.

There’s our different values for the city field of the city offices. “Human reading” the values, we have 8 offices in Amsterdam, London, Oslo, Paris, San Francisco and Tokyo, 7 offices in Athens, Barcelona, Chicago and Madrid.

The total of these offices is 76 but the search response says we have 113. Let’s subtract the bucket’s office total from the actual total offices: (113 - 76 = 37) == sum_other_doc_count. That’s right. Explanation: the aggregation we defined has another property named size which has a default value of 10. Since we didn’t define another value, Elasticsearch limited the buckets to the top 10 different city values based on their occurrences in the index. 37 are the documents whose cities are not listed in the buckets and NOT the number of the unlisted cities.

Let’s change the size to 50 (hoping that we don’t have offices in more than 50 cities).

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      }
    }
  }
}

Hint: In addition to defining the aggregation’s size, I also set in the top level query the size to 0 so that I can browse the response faster since the hits attribute will be empty.

Execute again.

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8
        },
        {
          "key": "London",
          "doc_count": 8
        },
        {
          "key": "Oslo",
          "doc_count": 8
        },
        {
          "key": "Paris",
          "doc_count": 8
        },
        {
          "key": "San Francisco",
          "doc_count": 8
        },
        {
          "key": "Tokyo",
          "doc_count": 8
        },
        {
          "key": "Athens",
          "doc_count": 7
        },
        {
          "key": "Barcelona",
          "doc_count": 7
        },
        {
          "key": "Chicago",
          "doc_count": 7
        },
        {
          "key": "Madrid",
          "doc_count": 7
        },
        {
          "key": "New York",
          "doc_count": 7
        },
        {
          "key": "Warsaw",
          "doc_count": 7
        },
        {
          "key": "Berlin",
          "doc_count": 6
        },
        {
          "key": "Budapest",
          "doc_count": 6
        },
        {
          "key": "Melbourne",
          "doc_count": 6
        },
        {
          "key": "Prague",
          "doc_count": 5
        }
      ]
    }
  }
}

Perfect. The previously missed cities are now fetched and the sum_other_doc_count’s 0 value confirms that we haven’t left any document outside the aggregation.

Let’s add another aggregation for the offices, this time for their type.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type"
      }
    }
  }
}

We defined another terms aggregation named office_types and kept the default bucket size since we know that we don’t have more than 10 distinct office types.

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        ...
      ]
    },
    "office_types": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "primary",
          "doc_count": 57
        },
        {
          "key": "secondary",
          "doc_count": 56
        }
      ]
    }
  }
}

So, we have 57 primary and 56 secondary offices and 0 offices left outside the aggregation’s buckets.

Important note

If we do define a search query along with the aggregations definition, then the aggregations will be applied to the subset of the documents that match the search query.

For example, if we want to see how many different office types are in Athens, we can specify the appropriate search query without altering the aggregations’ one.

GET city_offices/_search
{
  "size": 0,
  "query": {
    "term": {
      "city": {
        "value": "Athens"
      }
    }
  },
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type",
        "size": 10
      }
    }
  }
}

Execute and…

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 7,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Athens",
          "doc_count": 7
        }
      ]
    },
    "office_types": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "secondary",
          "doc_count": 5
        },
        {
          "key": "primary",
          "doc_count": 2
        }
      ]
    }
  }
}

We have 5 secondary and 2 primary offices in Athens.

What if we wanted to have this information for all cities without having to limit the search’s results? In other words, what if we wanted to present the facets like this:

Amsterdam (8)

Primary (4)

Secondary (4)

London (8)

Primary (6)

Secondary (2)

Athens (7)

Primary (2)

Secondary (5)

Read the next section.

Sub-bucket aggregations

The Terms aggregations (and other type of aggregations) allow the definition of sub-aggregations. The sub-aggregations are executed in the documents belonging to the bucket of the parent aggregation. It’s like asking: Give me the different cities. Ok, now give me for each one how many office types it has.

The syntax of sub-aggregations is pretty straight forward.

GET <index_name>/_search
{
  "query": { ... },
  "aggs": {
    "<aggregation name>": {
      "<aggregation type>": { <aggregation properties> },
      "aggs": {
        "<sub-aggregation name>": {
          "<sub-aggregation type>": { <sub-aggregation properties> }
        }
      }
    }
  }
}

Applying the above in our case, all we have to do is to define the search request as follows:

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type"
          }
        }
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type",
        "size": 10
      }
    }
  }
}

The response:

{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        {
          "key": "London",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 6
              },
              {
                "key": "secondary",
                "doc_count": 2
              }
            ]
          }
        },
        {
          "key": "Oslo",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        {
          "key": "Paris",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        {
          "key": "San Francisco",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 4
              },
              {
                "key": "secondary",
                "doc_count": 4
              }
            ]
          }
        },
        ...
      ]
    },
    "office_types": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "primary",
          "doc_count": 57
        },
        {
          "key": "secondary",
          "doc_count": 56
        }
      ]
    }
  }
}

If we wanted to retrieve even more sub-results for another property of the city offices (for example building type), we could add a sub-aggregation inside the already defined office_types sub-aggregation of the cities aggregation.

Awesome. Time to play with the pets.

Nested aggregations

As previously mentioned, we defined the city office’s relations as nested objects. In order to perform aggregations on these relations we have to follow another approach, the Nested aggregations.

A special single bucket aggregation that enables aggregating nested documents. – official Elasticsearch Nested Aggregation reference

Syntax:

GET <index_name>/_search
{

  "aggs" : {
    "<aggregation-name>" : {
      "nested" : {
        "path" : "<nested-object-path>"
      },
      "aggs" : {
        "<nested-aggregation-name>": {
          "<aggregation-type>" : { <aggregation-properties> }  
        }
      }
    }
  }
}

where nested-object-path is the navigation path of the desired object in the root document. For example, if we want aggregations for the citizens we will set this property to citizens. If we wanted aggregations for the pets, we will set this property to citizens.pets.

Let’s see how we can breakdown the number of citizens per occupation.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens"
      },
      "aggs": {
        "occupations": {
          "terms": {
            "field": "citizens.occupation",
            "size": 50
          }
        }
      }
    }
  }
}

The response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 3966,
      "occupations": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Hairdresser",
            "doc_count": 243
          },
          {
            "key": "Microbiologist",
            "doc_count": 241
          },
          {
            "key": "Farmer",
            "doc_count": 234
          },
          {
            "key": "Marketing Manager",
            "doc_count": 231
          },
          {
            "key": "Clinical Laboratory Technician",
            "doc_count": 230
          },
          {
            "key": "Librarian",
            "doc_count": 230
          },
          {
            "key": "Editor",
            "doc_count": 229
          },
          ...
        ]
      }
    }
  }
}

So, there are 3966 citizens that have registered their pets to our city offices of which 243 are Hairdressers, 241 are Microbiologists etc.

Important note

Mind the field definition of the nested Terms aggregation in the query. It has to be the full path of the nested object.

Ok good, but what if wanted to see in how many offices are these citizens registered. In other words, how many offices have Marketing Managers? How many offices have Librarians?

Reverse nested aggregations

We have to use another type of aggregation, named Reverse nested aggregation.

A special single bucket aggregation that enables aggregating on parent docs from nested documents. Effectively this aggregation can break out of the nested block structure and link to other nested structures or the root document, which allows nesting other aggregations that aren’t part of the nested object in a nested aggregation.

The reverse_nested aggregation must be defined inside a nested aggregation. –official Elasticsearch Reverse nested aggregation reference

Even though it might sound complicated, it’s not. And playing around with the data is the best way for understanding the provided functionality of any feature.

Syntax:

GET <index_name>/_search
{

  "aggs" : {
    "<aggregation-name>" : {
      "nested" : {
        "path" : "<nested-object-path>"
      },
      "aggs" : {
        "<nested-aggregation-name>": {
          "<aggregation-type>" : { <aggregation-properties> },
          "aggs": {
            "in_offices": {
              "reverse_nested": { <reverse-nested-options> }
            }
          }
        }
      }
    }
  }
}

As you can see, the reverse nested aggregation is always defined as a sub-aggregation inside a nested aggregation.

Time to see in how many offices each occupation is spread.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens"
      },
      "aggs": {
        "occupations": {
          "terms": {
            "field": "citizens.occupation",
            "size": 50
          },
          "aggs": {
            "in_offices": {
              "reverse_nested": {}
            }
          }
        }
      }
    }
  }
}

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 3966,
      "occupations": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Hairdresser",
            "doc_count": 243,
            "in_offices": {
              "doc_count": 98
            }
          },
          {
            "key": "Microbiologist",
            "doc_count": 241,
            "in_offices": {
              "doc_count": 98
            }
          },
          {
            "key": "Farmer",
            "doc_count": 234,
            "in_offices": {
              "doc_count": 99
            }
          },
          {
            "key": "Marketing Manager",
            "doc_count": 231,
            "in_offices": {
              "doc_count": 91
            }
          },
          {
            "key": "Clinical Laboratory Technician",
            "doc_count": 230,
            "in_offices": {
              "doc_count": 98
            }
          },
          {
            "key": "Librarian",
            "doc_count": 230,
            "in_offices": {
              "doc_count": 98
            }
          },
          ...
        ]
      }
    }
  }
}

Translating the response, we have 243 Hairdressers registered in 98 offices, 241 Microbiologists registered in 98 offices, 234 Farmers registered in 99 offices etc.

The reverse nested aggregation accepts a single option named path. This options defines how many steps backwards in the document hierarchy we want Elasticsearch to go to calculate the aggregations. In our case, since citizens is the immediate relation of the city office type we had to leave it undefined implying that we want the aggregations of the occupations to be calculated against the root object a.k.a. the offices. Confused? Don’t worry, it will become clear after playing around with the pets. Now.

How many pets per kind are registered per citizen?

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens.pets"
      },
      "aggs": {
        "kinds": {
          "terms": {
            "field": "citizens.pets.kind",
            "size": 10
          },
          "aggs": {
            "per_citizen": {
              "reverse_nested": {}
            }
          }
        }
      }
    }
  }
}

Note that we didn’t define the path option and

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 11845,
      "kinds": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Dog",
            "doc_count": 2421,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Hamster",
            "doc_count": 2403,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Cat",
            "doc_count": 2380,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Bird",
            "doc_count": 2330,
            "per_citizen": {
              "doc_count": 113
            }
          },
          {
            "key": "Rabbit",
            "doc_count": 2311,
            "per_citizen": {
              "doc_count": 113
            }
          }
        ]
      }
    }
  }
}

So, there are 2421 registered dogs, 2403 registered hamsters, 2380 registered cats etc. The per_citizen bucket info though doesn’t seem right. Does the 113 number ring a bell? Exactly, that’s how many offices we have. Since we didn’t define the path in the reverse nested aggregation, Elasticsearch calculated the count of root documents (a.k.a. the offices) that have each pet kind registered. Let’s fix that.

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "citizens": {
      "nested": {
        "path": "citizens.pets"
      },
      "aggs": {
        "kinds": {
          "terms": {
            "field": "citizens.pets.kind",
            "size": 10
          },
          "aggs": {
            "per_citizen": {
              "reverse_nested": {
                "path": "citizens"
              }
            }
          }
        }
      }
    }
  }
}

There it is:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "citizens": {
      "doc_count": 11845,
      "kinds": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Dog",
            "doc_count": 2421,
            "per_citizen": {
              "doc_count": 1864
            }
          },
          {
            "key": "Hamster",
            "doc_count": 2403,
            "per_citizen": {
              "doc_count": 1852
            }
          },
          {
            "key": "Cat",
            "doc_count": 2380,
            "per_citizen": {
              "doc_count": 1823
            }
          },
          {
            "key": "Bird",
            "doc_count": 2330,
            "per_citizen": {
              "doc_count": 1803
            }
          },
          {
            "key": "Rabbit",
            "doc_count": 2311,
            "per_citizen": {
              "doc_count": 1800
            }
          }
        ]
      }
    }
  }
}

There are 2421 dogs registered by 1864 citizens, 2403 hamsters registered by 1852 citizens, 2380 cats registered by 1823 citizens etc.

Note: Each citizen has more than one pets which can be of different kind that’s why the per_citizen sum of doc_count is way bigger than our citizens.

Before moving on to the last type of aggregations, let’s execute a more advanced nested query answering to the questions:

For each city, how many kind of pets are registered per citizen occupation and in how many offices?

Breaking down the question, we have to think like this:

Since we want results per city, we are going to add a Terms aggregation on the field city
Since we want to have results per citizen occupation, we are going to add a Terms sub-aggregation on the field occupation.
- Since the citizen is a nested object, the aggregation of the previous item has to be a sub-aggregation of a Nested aggregation on the path citizen
Since we want to have results per pet kind, we are going to add a Terms sub-aggregation on the field kind.
- Since the pet is a nested object, the aggregation of the previous item has to be a sub-aggregation of a Nested aggregation on the path citizen.pets

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "citizens": {
          "nested": {
            "path": "citizens"
          },
          "aggs": {
            "occupations": {
              "terms": {
                "field": "citizens.occupation",
                "size": 50
              },
              "aggs": {
                "pets": {
                  "nested": {
                    "path": "citizens.pets"
                  },
                  "aggs": {
                    "kinds": {
                      "terms": {
                        "field": "citizens.pets.kind",
                        "size": 10
                      },
                      "aggs": {
                        "per_occupation": {
                          "reverse_nested": {
                            "path": "citizens"
                          }
                        },
                        "per_office": {
                          "reverse_nested": {}
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

{
  "took": 17,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 113,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Amsterdam",
          "doc_count": 8,
          "citizens": {
            "doc_count": 230,
            "occupations": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "Dancer",
                  "doc_count": 19,
                  "pets": {
                    "doc_count": 49,
                    "kinds": {
                      "doc_count_error_upper_bound": 0,
                      "sum_other_doc_count": 0,
                      "buckets": [
                        {
                          "key": "Cat",
                          "doc_count": 13,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 9
                          }
                        },
                        {
                          "key": "Rabbit",
                          "doc_count": 11,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 10
                          }
                        },
                        {
                          "key": "Bird",
                          "doc_count": 9,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 7
                          }
                        },
                        {
                          "key": "Dog",
                          "doc_count": 8,
                          "per_office": {
                            "doc_count": 5
                          },
                          "per_occupation": {
                            "doc_count": 7
                          }
                        },
                        {
                          "key": "Hamster",
                          "doc_count": 8,
                          "per_office": {
                            "doc_count": 3
                          },
                          "per_occupation": {
                            "doc_count": 7
                          }
                        }
                      ]
                    }
                  }
                },
                ...
              ]
            }
          }
        },
        ...
        ]
      }
    }
  }
}

“Human” reading the response, in Amsterdam there are 8 offices with 230 citizens that have registered pets of which:

19 are Dancers that have registered a total of 49 pets of which
- 13 are cats registered by 9 dancers in 5 offices
- 11 are rabbits registered by 10 dancers in 5 offices
- …
- 8 are hamsters registered by 7 dancers in 3 offices
- I stop this now because I got dizzy

Global aggregations

I will end this tutorial with something easier. Global aggregations.

Defines a single bucket of all the documents within the search execution context. This context is defined by the indices and the document types you’re searching on, but is not influenced by the search query itself. – official Elasticsearch Global aggregation reference

To better explain this, let’s re-think of query:

GET city_offices/_search
{
  "size": 0,
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type"
          }
        }
      }
    },
    "office_types": {
      "terms": {
        "field": "office_type",
        "size": 10
      }
    }
  }
}

which would help us build this form:

Amsterdam (8)

Primary (4)

Secondary (4)

London (8)

Primary (6)

Secondary (2)

Athens (7)

Primary (2)

Secondary (5)

When we render such a form, what is the expected behavior once he/she clicks on a checkbox, for example London? Well, we expect the aggregations to be re-applied to the search results that were narrowed by the clicked facet. So the request will become:

GET city_offices/_search
{
  "size": 0,
  "query": {
    "term": {
      "city": {
        "value": "London"
      }
    }
  },
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type",
            "size": 10
          }
        }
      }
    }
  }
}

and the response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "London",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 6
              },
              {
                "key": "secondary",
                "doc_count": 2
              }
            ]
          }
        }
      ]
    }
  }
}

If we rendered the form based on the response, it would only have the London entry:

London (8)

Primary (6)

Secondary (2)

It would be better if we could render the previous form with disabled checkboxes for the terms that are no longer valid.

Amsterdam

Primary

Secondary

London

Primary (6)

Secondary (2)

Athens

Primary

Secondary

We could accomplish that if we had the aggregation results of a search request without a query and compared them with the search request that was triggered after user narrows the results by clicking on a checkbox. To avoid executing an additional search request, we can use the Global aggregation.

GET city_offices/_search
{
  "size": 0,
  "query": {
    "term": {
      "city": {
        "value": "London"
      }
    }
  },
  "aggs": {
    "cities": {
      "terms": {
        "field": "city",
        "size": 50
      },
      "aggs": {
        "office_types": {
          "terms": {
            "field": "office_type",
            "size": 10
          }
        }
      }
    },
    "unfiltered": {
      "global": {},
      "aggs": {
        "cities": {
          "terms": {
            "field": "city",
            "size": 50
          },
          "aggs": {
            "office_types": {
              "terms": {
                "field": "office_type",
                "size": 10
              }
            }
          }
        }
      }
    }
  }
}

Now, in the response we have an unfiltered section which we can use to render the form as we please.

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 8,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "cities": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "London",
          "doc_count": 8,
          "office_types": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "key": "primary",
                "doc_count": 6
              },
              {
                "key": "secondary",
                "doc_count": 2
              }
            ]
          }
        }
      ]
    },
    "unfiltered": {
      "doc_count": 113,
      "cities": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "key": "Amsterdam",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "London",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 6
                },
                {
                  "key": "secondary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Oslo",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "Paris",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "San Francisco",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "Tokyo",
            "doc_count": 8,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 4
                }
              ]
            }
          },
          {
            "key": "Athens",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 5
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Barcelona",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Chicago",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 5
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Madrid",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 4
                },
                {
                  "key": "primary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "New York",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Warsaw",
            "doc_count": 7,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 4
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Berlin",
            "doc_count": 6,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 4
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          },
          {
            "key": "Budapest",
            "doc_count": 6,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 3
                },
                {
                  "key": "secondary",
                  "doc_count": 3
                }
              ]
            }
          },
          {
            "key": "Melbourne",
            "doc_count": 6,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "primary",
                  "doc_count": 5
                },
                {
                  "key": "secondary",
                  "doc_count": 1
                }
              ]
            }
          },
          {
            "key": "Prague",
            "doc_count": 5,
            "office_types": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "key": "secondary",
                  "doc_count": 3
                },
                {
                  "key": "primary",
                  "doc_count": 2
                }
              ]
            }
          }
        ]
      }
    }
  }
}

Code and comments

You may find the script generating random data and the already generated sample data here.

You can submit your feedback or any comments you may have here.

Thanks for visiting!

What’s next

My next post will be a tutorial for implementing what we covered in this post within a Rails application using the elasticsearch gem
Next, I might come back with a tutorial covering the metrics aggregations

That’s all! Bucket cat photo.

Indexing and full text searching Linux man pages

Thu, 12 Apr 2018 19:00:00 +0300

In this post we are going to play around with full text searching Linux man pages using Elasticsearch.

The idea

We will:

setup an Elasticsearch instance locally
create an index for the data
feed the index with the man pages of the OS
create a search method for full text searching
full text search the man pages

The code

I have gathered the code snippets you will find here in a script called elman and it is available on GitHub so after reading this post to get the idea, you can experiment by modifying it as you please.

Let’s start.

Prerequisites

Ruby

You should have ruby installed on your system. I use RVM to manage multiple ruby versions installed on my system. If you want to go with that too, you can find instructions for installing the tool in its homepage and for installing ruby versions (a.k.a. rubies) the related documentation page.

This tutorial is using Ruby 2.4.2. You can check yours using: ruby --version or via RVM with rvm list.

Elasticsearch

Providing instructions for installing Elasticsearch is out of this post’s scope. I recommend visiting the tool’s related documenation site.

After the installation, just make sure to start the service if it’s not started.

You can check that with curl http://localhost:9200.

It should give something like:

{
  "name" : "asdfasdf",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "asasdfdf-as",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "asdf",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

As you can see, we will use elasticsearch version 6.0.0 in this tutorial.

Hands on

We will be working in the awesome Ruby interactive shell (irb) but before starting it there’s a gem we need to install.

The elasticsearch gem

The elasticsearch gem is the official library provided by Elastic to interact with Elasticsearch instances in Ruby.

Install the gem and its dependencies use the following command in your terminal:

gem install elasticsearch

Ruby interactive shell

Start the Ruby interactive shell with:

irb

Load the Elasticsearch library we previously installed with:

require 'elasticsearch'

You should see something like the following:

>> require 'elasticsearch'
=> true
>>

From now on, in our session we can use the gem’s provided classes and modules to interact with Elasticsearch.

Creating the client

We have to create an Elasticsearch::Client instance through which we will execute commands on the index. In irb type:

@client = Elasticsearch::Client.new host: 'localhost', port: '9200'

Check the cluster’s health to make sure everything is right:

client.cluster.health
=> {"cluster_name"=>"elasticsearch", "status"=>"yellow", "timed_out"=>false, "number_of_nodes"=>1, "number_of_data_nodes"=>1, "active_primary_shards"=>10, "active_shards"=>10, "relocating_shards"=>0, "initializing_shards"=>0, "unassigned_shards"=>10, "delayed_unassigned_shards"=>0, "number_of_pending_tasks"=>0, "number_of_in_flight_fetch"=>0, "task_max_waiting_in_queue_millis"=>0, "active_shards_percent_as_number"=>50.0}

Make sure that the status from the output hash is not red.

Creating the index

Let’s name the index elastic_manpages and set that it is going to have three properties:

command: the Linux command (even though there are man pages for other stuff like time.conf which is a configuration file)
description: the short description provided in the top of the man page, in the NAME section
manpage: the corresponding man page

Create the index with the following command:

@client.indices.create   index: 'elastic_manpages',
                         body: {
                           mappings: {
                             document: {
                               properties: {
                                 command: {
                                   type: :text
                                 },
                                 description: {
                                   type: :text,
                                   analyzer: :english
                                 },
                                 manpage: {
                                   type: :text,
                                   analyzer: :english
                                 }
                               }
                             }
                           }
                         }

As you can see, we assigned the english analyzer to the description and manpage properties.

Why? Quoting the documentation on Language analyzers:

They are able to take the peculiarities of the specified language into account. For instance, the english analyzer comes with a set of English stopwords (common words like and or the that don’t have much impact on relevance), which it removes. This analyzer also is able to stem English words because it understands the rules of English grammar.

After typing the command, you should get a response like this one:

=> {"acknowledged"=>true, "shards_acknowledged"=>true, "index"=>"elastic_manpages"}

Indexing the man pages

Time to index the man pages. We are going to use the apropos command to list all entries of the mandb.

all_pages = `apropos .`.split "\n"

The all_pages variable now contains a list of all man entries in the following format:

erb (1)              - Ruby Templating

Note: the (1) part above defines what section of the manual the page is from.

For each entry of this list, we need to:

export the command and the description
retrieve the man page for each command
index a new document mapping the values of the previous items to the appropriate properties.

To split the raw apropos entry to the desired field, we will use the following regex:

/(.*)\s\(\d*.*\)\s*-\s*(.*)/

You can check that is working on Rubular.

Back in irb:

apropos_regex = /(.*)\s\(\d*.*\)\s*-\s*(.*)/
all_pages.each do |line|
  matches = apropos_regex.match line
  command = matches[1]
  description = matches[2]
  manpage = `man #{command}`

  @client.index index: 'elastic_manpages',
                type: :document,
                body: {
                  command: command,
                  description: description,
                  manpage: manpage
                }
end

It will take some time depending on your hardware and the amount of man pages on your OS. Wait for it to end. During the process of indexing you might see some warnings for some pages but don’t worry, the process will complete after all and we don’t care that much about them not being indexed correctly for the purpose of this tutorial.

Creating the search method

Now will create a method for full text searching so that we can execute it more than once to search.

In irb:

def search(term)
  result = @client.search  index: 'elastic_manpages',
                           size: 10,
                           body: {
                             query: {
                               multi_match: {
                                 query: term,
                                 type: :cross_fields,
                                 fields: ['command', 'description^3', 'manpage^3'],
                                 operator: :or,
                                 tie_breaker: 1.0,
                                 cutoff_frequency: 0.1
                               }
                             }
                           }

  result['hits']['hits'].map{ |hit| { command: hit['_source']['command'],
                                      description: hit['_source']['description'],
                                      manpage: hit['_source']['manpage'] } }
end

Notes on the search definition

size: 10: set the size of the results to be 10
multi_match: because we want to search on multiple fields (command, description, manpage)
type: :cross_fields: The cross_fields type is particularly useful with structured documents where multiple fields should match (read more here)
tie_breaker: add the scores for each term of the query (read more here)
cutoff_frequency: it prevents scoring / iterating high frequency terms and only takes the terms into account if a more significant / lower frequency term matches a document (read more here)

Play time!

That’s it. Time to check what we have created.

In irb:

search('edit image').each{ |a| puts "#{a[:command].ljust(20)} - #{a[:description]}"}; nil
gimp-2.8             - an image manipulation and paint program.
gimp                 - an image manipulation and paint program.
Image::Magick::Q16   - objected-oriented Perl interface to ImageMagick (Q16). Use it to create, edit, compose, or convert ...
gimp-console-2.8     - an image manipulation and paint program.
gimp-console         - an image manipulation and paint program.
bitmap               - bitmap editor and converter utilities for the X Window System
bmtoa                - bitmap editor and converter utilities for the X Window System
atobm                - bitmap editor and converter utilities for the X Window System
mailcap.order        - the mailcap ordering specifications
xwd                  - dump an image of an X window
=> nil
search('change owner').each{ |a| puts "#{a[:command].ljust(20)} - #{a[:description]}"}; nil
chown                - change file owner and group
chown                - change ownership of a file
fchownat             - change ownership of a file
fchown               - change ownership of a file
lchown32             - change ownership of a file
chown32              - change ownership of a file
fchown32             - change ownership of a file
lchown               - change ownership of a file
fchmod               - change permissions of a file
XSetEventQueueOwner  - set event queue owner on a shared Xlib/XCB connection
=> nil

You can modify the search definition as you please. Check more on the multi_match query here.

As always, cat photo.

Creating a bash completion script

Thu, 01 Mar 2018 02:00:00 +0200

I recently worked on creating a bash completion script for registering directory aliases and navigating to them with autocomplete in Bash and I found the feature really interesting.

In this post I will familiarize you with the process of adding bash completion to your scripts.

What is bash completion?

Bash completion is a functionality through which bash helps users type their commands faster and easier. It accomplishes that by presenting possible options when users press the tab key while typing a command.

$ git<tab><tab>
git                 git-receive-pack    git-upload-archive  
gitk                git-shell           git-upload-pack     
$ git-s<tab>
$ git-shell

How it works

The completion script is code that uses the builtin bash command complete to define which completion suggestions can be displayed for a given executable. The nature of the completion options vary from simple static to highly sophisticated.

Why bother

It is good to provide such a functionality to users:

to save them from typing text when it can be auto-completed
to help them know what are the available continuations to their commands
to prevent errors and improve their experience by hiding or showing options based on what they have already typed

Hands on

Here’s what we are going to do in this tutorial.

We will first create a dummy executable script called dothis. All it does is execute the command which resides on the number, that was passed as an argument, in user’s history. For example, the following command will simply execute the ls -a command given that it exists in history with number 235:

dothis 235

Then we will create a bash completion script that will display commands along with their number from user’s history and we will “bind” it to the dothis executable.

$ dothis <tab><tab>
ls
ls -la
cd ~
man history
git status
history | cut -c 8-

You can see a gif demonstrating the functionality at this tutorial’s code repository on GitHub.

Let the show begin.

Creating the executable script

Create a file named dothis in your working directory and add the following code:

if [ -z "$1" ]; then
  echo "No command number passed"
  exit 2
fi

exists=$(fc -l -1000 | grep ^$1 -- 2>/dev/null)

if [ -n "$exists" ]; then
  fc -s -- "$1"
else
  echo "Command with number $1 was not found in recent history"
  exit 2
fi

Notes:

We first check if the script was called with an argument
We then check if the specific number is included in the last 1000 commands
- if it exists we execute the command using the fc functionality
- otherwise we display an error message

Make the script executable with:

chmod +x ./dothis

We are going to execute this script many times in this tutorial so I suggest you place it in a folder that is included in your path so that we can access it from anywhere with just typing dothis.

I installed it in my home bin folder using:

install ./dothis ~/bin/dothis

You can do the same given that you have a ~/bin folder and that it is included in your PATH variable.

Try to see if it’s working:

dothis

You should see this.

$ dothis
No command number passed

Done.

Creating the completion script

Create a file named dothis-completion.bash. From now on we will be referring to this file with the term completion script.

Once we add some code to it, we will source it to allow the completion to take effect. We must source this file every single time we change something in it.

We will discuss our options for registering this script whenever a bash shell opens later on.

Static completion

Suppose that the dothis program supported a list of commands, for example:

now
tomorrow
never

Let’s use the complete command to register this list for completion. To use the proper terminology, we say we use the complete command to define a completion specification (compspec) for our program.

Add this to the completion script.

#/usr/bin/env bash
complete -W "now tomorrow never" dothis

What did we specify with the complete command above:

we used the -W (wordlist) option to provide a list of words for completion.
we defined to which “program” these completion words will be used (the dothis parameter)

Source the file:

source ./dothis-completion.bash

Now try pressing tab twice in the command line as shown below:

$ dothis <tab><tab>
never     now       tomorrow

Try again after typing the n:

$ dothis n<tab><tab>
never now

Magic! The completion options are automatically filtered to match only those starting with n.

Note: The options are not displayed in the order that we defined them in the word list, they are automatically sorted.

There are a lot of other options to be used instead of the -W that we used in this section. Most of them produce completions in a fixed manner, meaning that we don’t intervene dynamically to filter their output.

For example, if we wanted to have directory names as completion words for the dothis program, we would change the complete command to the following:

complete -A directory dothis

Pressing tab after the dothis program, we would get a list of the directories in the current directory from which we execute the script:

$ dothis <tab><tab>
dir1/ dir2/ dir3/

Find the complete list of the available flags here.

Dynamic completion

We will be producing the completions of the dothis executable with the following logic:

If user presses the tab key right after the command, we will show the last 50 executed commands along with their numbers in history
If user presses the tab key after typing a number that matches more than one commands from history, we will show only those commands along with their numbers in history
If user presses the tab key after a number that matches exactly one command in history, we auto-complete the number without appending the command’s literal (if it’s confusing, you will better understand later on, no worries)

Let’s start by defining a function that will execute each time user requests completion on a dothis command. Change the completion script to this:

#/usr/bin/env bash
_dothis_completions()
{
  COMPREPLY+=("now")
  COMPREPLY+=("tomorrow")
  COMPREPLY+=("never")
}

complete -F _dothis_completions dothis

Note the following:

we used the -F flag in the complete command defining that the _dothis_completions is the function that will provide the completions of the dothis executable
COMPREPLY is an array variable used to store the completions - the completion mechanism uses this variable to display its contents as completions

Now source the script and go for completion:

$ dothis <tab><tab>
never now tomorrow

Perfect. We produce the same completions as in the previous section with the word list. Or not? Try this:

$ dothis nev<tab><tab>
never     now       tomorrow

As you can see, even though we type nev and then we request for completion, the available options are always the same and nothing gets completed automatically. Why is this happening?

The contents of the COMPREPLY variable are always displayed. The function is responsible to add/remove entries from there now.
If the COMPREPLY variable had only one element then that word would be automatically completed in the command. Since current implementation always return the same three words, this will not happen.

Enter compgen: a builtin command that generates completions supporting most of the options of the complete command (ex. -W for word list, -d for directories) and filtering them based on what already has been typed by the user.

Don’t worry if you feel confused, everything will become clear later on.

Type the following in the console to better understand what compgen does:

$ compgen -W "now tomorrow never"
now
tomorrow
never
$ compgen -W "now tomorrow never" n
now
never
$ compgen -W "now tomorrow never" t
tomorrow

So now we can use it but we need to find a way to know what has been typed after the dothis command. We already have the way. The bash completion facilities provide bash variables related to the completion taking place. Here are the more important ones:

COMP_WORDS: an array of all the words typed after the name of the program the compspec belongs to
COMP_CWORD: an index of the COMP_WORDS array pointing to the word the current cursor is at - in other words, the index of the word the cursor was when the tab key was pressed
COMP_LINE: the current command line

To access the word just after the dothis word, we can use the value of COMP_WORDS[1]

Change the completion script again:

#/usr/bin/env bash
_dothis_completions()
{
  COMPREPLY=($(compgen -W "now tomorrow never" "${COMP_WORDS[1]}"))
}

complete -F _dothis_completions dothis

Source and there you are:

 $ dothis
never     now       tomorrow  
$ dothis n
never  now

Now, instead of the now, tomorrow, never words, we would like to see actual numbers from the command history.

The fc -l command followed by a negative number -n displays the last n commands. So, we will use:

fc -l -50

which lists the last 50 executed commands along with their numbers. The only manipulation we need to do is replace tabs with spaces in order to be displayed properly from the completion mechanism. sed to the rescue.

Change the completion script as follows:

#/usr/bin/env bash
_dothis_completions()
{
  COMPREPLY=($(compgen -W "$(fc -l -50 | sed 's/\t//')" -- "${COMP_WORDS[1]}"))
}

complete -F _dothis_completions dothis

Source and test in the console:

$ dothis <tab><tab>
source dothis-completion.bash   649 source dothis-completion.bash   666 cat ~/.bash_profile
clear                           650 clear                           667 cat ~/.bashrc
source dothis-completion.bash   651 source dothis-completion.bash   668 clear
source dothis-completion.bash   652 source dothis-completion.bash   669 install ./dothis ~/bin/dothis
clear                           653 source dothis-completion.bash   670 dothis
source dothis-completion.bash   654 clear                           671 dothis 6546545646
clear                           655 dothis 654                      672 clear
source dothis-completion.bash   656 dothis 631                      673 dothis
source dothis-completion.bash   657 dothis 150                      674 dothis 651
source dothis-completion.bash   658 dothis                          675 source dothis-completion.bash
clear                           659 clear                           676 dothis 651
dothis 623  ls -la              660 dothis                          677 dothis 659
clear                           661 install ./dothis ~/bin/dothis   678 clear
source dothis-completion.bash   662 dothis                          679 dothis 665
clear                           663 install ./dothis ~/bin/dothis   680 clear
source dothis-completion.bash   664 dothis                          681 clear
clear                           665 cat ~/.bashrc

Not bad.

We do have a problem though. Try typing a number as you see it in your completion list and the press the key again.

$ dothis 623<tab>
$ dothis 623  ls 623  ls -la
...
$ dothis 623  ls 623  ls 623  ls 623  ls 623  ls -la

This is happening because in our completion script, we used the ${COMP_WORDS[1]} to always check the first typed word after the dothis command (the number 623 in the above snippet). Hence the completion continues suggesting the same completion again and again while pressing the tab key.

To fix this, we will not allow any kind of completion to take place if at least one argument has already been typed. We will add a condition in our function checking the size of the aforementioned COMP_WORDS array.

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  COMPREPLY=($(compgen -W "$(fc -l -50 | sed 's/\t//')" -- "${COMP_WORDS[1]}"))
}

complete -F _dothis_completions dothis

Source and retry.

$ dothis 623<tab>
$ dothis 623 ls -la<tab> # SUCCESS: nothing happens here

There is another thing we don’t like though. We do want to display the numbers along with the corresponding commands to help users decide which one is the desired but when there is only one completion suggestion and gets automatically picked by the completion mechanism, we shouldn’t append the command literal too.

In other words, our dothis executable accepts only a number and we haven’t added any functionality checking or expecting other arguments. When our completion function gives only one result, we should trim the command literal and respond only with the command number.

To accomplish this, we will keep the response of the compgen command in an array variable and if its size is 1 we will trim the one and only element to keep just the number. Otherwise, we’ll let the array as is.

Change the completion script to this:

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  # keep the suggestions in a local variable
  local suggestions=($(compgen -W "$(fc -l -50 | sed 's/\t/ /')" -- "${COMP_WORDS[1]}"))

  if [ "${#suggestions[@]}" == "1" ]; then
    # if there's only one match, we remove the command literal
    # to proceed with the automatic completion of the number
    local number=$(echo ${suggestions[0]/%\ */})
    COMPREPLY=("$number")
  else
    # more than one suggestions resolved,
    # respond with the suggestions intact
    COMPREPLY=("${suggestions[@]}")
  fi
}

complete -F _dothis_completions dothis

Registering the completion script

If you want to enable the completion just for you on your machine, all you have to do is add a line in your .bashrc file sourcing the script:

source <path-to-your-script>/dothis-completion.bash

If you want to enable the completion for all users, you can just copy the script under /etc/bash_completion.d/ and it will automatically be loaded by Bash.

Fine tuning the completion script

Some extra steps for better results :)

Displaying each entry in a new line

Well, in the bash completion script I was working on, I too had to present suggestions consisting of two parts. I wanted to display the first part with the default color and the second part with a gray color to point out that it was just help text. In this tutorial’s example, it would be nice to present the numbers in the default color but the command literal in another one, less fancy.

Unfortunately, this is not possible at least for the time being because the completions are displayed as plain text and color directives are not processed (for example: \e[34mBlue).

What we can do though to improve user experience (or not :D) is to display each entry in a new line. This solution is not that obvious since again we can’t just append a new line character in each COMPREPLY entry. We will follow a rather hackish way and pad suggestion literals to such a width that fills the terminal’s width.

Enter printf. If you want to display each suggestion on each own line, change the completion script to the following:

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  local IFS=$'\n'
  local suggestions=($(compgen -W "$(fc -l -50 | sed 's/\t//')" -- "${COMP_WORDS[1]}"))

  if [ "${#suggestions[@]}" == "1" ]; then
    local number="${suggestions[0]/%\ */}"
    COMPREPLY=("$number")
  else
    for i in "${!suggestions[@]}"; do
      suggestions[$i]="$(printf '%*s' "-$COLUMNS"  "${suggestions[$i]}")"
    done

    COMPREPLY=("${suggestions[@]}")
  fi
}

complete -F _dothis_completions dothis

Source and test:

dothis <tab><tab>
...
499 source dothis-completion.bash                   
500 clear
...       
503 dothis 500

Customizable behavior

In our case, we hard-coded to display the last 50 commands for completion. This is not a good practice. We should first respect what each user might prefer and if he/she hasn’t made any preference then we should default to 50.

To accomplish that we will check if an environment variable DOTHIS_COMPLETION_COMMANDS_NUMBER has been set.

Change the completion script one last time:

#/usr/bin/env bash
_dothis_completions()
{
  if [ "${#COMP_WORDS[@]}" != "2" ]; then
    return
  fi

  local commands_number=${DOTHIS_COMPLETION_COMMANDS_NUMBER:-50}
  local IFS=$'\n'
  local suggestions=($(compgen -W "$(fc -l -$commands_number | sed 's/\t//')" -- "${COMP_WORDS[1]}"))

  if [ "${#suggestions[@]}" == "1" ]; then
    local number="${suggestions[0]/%\ */}"
    COMPREPLY=("$number")
  else
    for i in "${!suggestions[@]}"; do
      suggestions[$i]="$(printf '%*s' "-$COLUMNS"  "${suggestions[$i]}")"
    done

    COMPREPLY=("${suggestions[@]}")
  fi
}

complete -F _dothis_completions dothis

Source and test:

export DOTHIS_COMPLETION_COMMANDS_NUMBER=5
$ dothis <tab><tab>
clear
source ./dothis-completion.bash
dothis clear
clear
export DOTHIS_COMPLETION_COMMANDS_NUMBER=5

Useful links

Code and Comments

You can find the code of this tutorial on GitHub.

For feedback, comments, typos etc. please open an issue in the repository.

Long post, cat photo

Let me introduce you to my debugger.

That’s all folks!

Creating a Linux Desktop application with Ruby

Thu, 25 Jan 2018 21:00:00 +0200

Lately I was experimenting with GTK and its Ruby bindings and I decided to write a tutorial introducing this functionality. In this post we are going to create a simple ToDo application (something like what we created here with Ruby on Rails) using the gtk3 gem a.k.a. the GTK+ Ruby bindings.

Note: The code of the tutorial is available at GitHub.

What is GTK+

Quoting the toolkit’s page:

GTK+, or the GIMP Toolkit, is a multi-platform toolkit for creating graphical user interfaces. Offering a complete set of widgets, GTK+ is suitable for projects ranging from small one-off tools to complete application suites.

..and about its creation:

GTK+ was initially developed for and used by the GIMP, the GNU Image Manipulation Program. It is called the “The GIMP ToolKit” so that the origins of the project are remembered. Today it is more commonly known as GTK+ for short and is used by a large number of applications including the GNU project’s GNOME desktop.

Prerequisites

GTK+ version

Make sure you have GTK+ installed.

The OS in which I developed the tutorial’s application is Ubuntu 16.04 which has GTK+ installed by default (version: 3.18).

You can check yours with the following command:

dpkg -l libgtk-3-0

Ruby

This tutorial is using Ruby 2.4.2. You can check yours using: ruby --version or via RVM with rvm list.

Glade

Again, quoting the tool’s page

Glade is a RAD tool to enable quick & easy development of user interfaces for the GTK+ toolkit and the GNOME desktop environment

We will use Glade to design the user interface of our application. If you are on Ubuntu, install glade with:

sudo apt install glade

gtk3 gem

This gem provides the Ruby bindings for the GTK+ toolkit. In other words, it allows us to talk to the GTK+ API using the Ruby language.

Install the gem via:

gem install gtk3

The application specs

We will build an application that:

it will have a user interface (desktop application)
it will allow users to set miscellaneous properties to each item (such as priority)
it will allow users to create and edit ToDo items
- all items will be saved as files in the user’s home directory in a folder named .gtk-todo-tutorial
it will allow users to archive ToDo items
- archived items should be put in their own folder archived

The application structure

gtk-todo-tutorial # root directory
  |-- application
    |-- ui # everything related to the ui of the application
    |-- models # our models
    |-- lib # the directory to host any utilities we might need
  |-- resources # directory to host the resources of our application
  gtk-todo # the executable that will start our application

Let’s start!

Building the ToDo application

Initializing the application

Create a directory in which we will save all files that the application will need. As shown in the section above, I named mine gtk-todo-tutorial.

In there create a file named gtk-todo (that’s right, no extension) and add the following:

#!/usr/bin/env ruby

require 'gtk3'

app = Gtk::Application.new 'com.iridakos.gtk-todo', :flags_none

app.signal_connect :activate do |application|
  window = Gtk::ApplicationWindow.new(application)
  window.set_title 'Hello GTK+Ruby!'
  window.present
end

puts app.run

This is going to be the script that will start the application.

Note the shebang in the first line. This is how we define which interpreter must be used to execute the script under UNIX/Linux operating systems. This way, we don’t have to use ruby gtk-todo but just the script’s name gtk-todo.

Don’t try it yet though because we haven’t changed the mode of the file so as to be executable. To do so, type the following command in a terminal after navigating to the application’s root directory:

chmod +x ./gtk-todo # make the script executable

Now from the console execute:

./gtk-todo # execute the script

Ta daaaaaa

Notes

The application object we defined above and all of the GTK+ widgets in general, emit signals to trigger events. Once an application starts running for example, it emits a signal to trigger the activate event. All we have to do is to define what we want to happen when this signal is emitted. We accomplished this by using the signal_connect instance method and passing it a block whose code will be executed upon the given event. We will be doing this a lot throughout the tutorial.
When we initialized the Gtk::Application object we passed two parameters:
- com.iridakos.gtk-todo: this is our application’s id and in general it should be a reverse DNS style identifier. For more information about its usage and best practices check here.
- :flags_none: this is a flag defining the behavior of the application. In our case, we used the default behavior. Check here all the flags and the type of applications they define. You can use the Ruby equivalent flags as defined in Gio::ApplicationFlags.constants. For example, instead of using the :flags_none we could instead use Gio::ApplicationFlags::FLAGS_NONE

Suppose the application object we previously created (Gtk::Application) had a lot of things to do when the activate signal was emitted or that we wanted to connect to more signals. We would end up creating a huge gtk-todo script file making it hard to read/maintain. Time to refactor.

As described in the The application structure section, create a folder named application along with its sub-folders ui, models and lib.

In the ui folder we will place all files related to our user interface.
In the models folder we will place all files related to our models.
In the lib folder we will place all other files that don’t belong to either of the aforementioned folders.

We are going to define a new subclass of the Gtk::Application class for our application. Create a file named application.rb under application/ui/todo with the following contents.

module ToDo
  class Application < Gtk::Application
    def initialize
      super 'com.iridakos.gtk-todo', Gio::ApplicationFlags::FLAGS_NONE

      signal_connect :activate do |application|
        window = Gtk::ApplicationWindow.new(application)
        window.set_title 'Hello GTK+Ruby!'
        window.present
      end
    end
  end
end

and change the gtk-todo script accordingly:

#!/usr/bin/env ruby

require 'gtk3'

app = ToDo::Application.new

puts app.run

Much cleaner, isn’t it? Yeah, but it doesn’t work. You should be getting something like:

./gtk-todo:5:in `<main>': uninitialized constant ToDo (NameError)

The problem is that we haven’t required any of the ruby files placed in the application folder. Change the script file as follows and execute it again.

#!/usr/bin/env ruby

require 'gtk3'

# Require all ruby files in the application folder recursively
application_root_path = File.expand_path(__dir__)
Dir[File.join(application_root_path, '**', '*.rb')].each { |file| require file }

app = ToDo::Application.new

puts app.run

You should be fine.

Resources

At the beginning of this tutorial we said that we would use Glade to design the user interface of the application. Glade actually produces xml files with the appropriate elements and attributes that reflect what we designed via its user interface. We somehow need to make use of these files so that our application gets the UI we designed.

These files are resources for the application and the GResource API provides a way for packing them all together in a binary file and afterwards accessing them from inside the application with advantages as opposed to manually having to deal with already loaded resources, their location on the file system etc. Read more about the API here.

Describing the resources

First, we need to create a file describing the resources of the application. Create a file named gresources.xml and place it directly under the resources folder.

<?xml version="1.0" encoding="UTF-8"?>
<gresources>
  <gresource prefix="/com/iridakos/gtk-todo">
    <file preprocess="xml-stripblanks">ui/application_window.ui</file>
  </gresource>
</gresources>

In this “description” we actually say: we have a resource which is located under the ui directory (relative to this xml file) with name application_window.ui. Before loading this resource please remove the blanks. Thanks. Of course this is not going to work now since we haven’t created the resource via Glade yet. Don’t worry though, one thing at a time.

Note: the xml-stripblanks directive will use the xmllint command to remove the blanks. In Ubuntu you have to install the package libxml2-utils to obtain it.

Building the resources binary file

In order to produce the binary resources file, we are going to use another utility of the GLib library called glib-compile-resources. Check if you have it installed with dpkg -l libglib2.0-bin. You should be seeing something like this:

ii  libglib2.0-bin     2.48.2-0ubuntu amd64          Programs for the GLib library

If not, then install the package (sudo apt install libglib2.0-bin in Ubuntu).

Let’s build the file. We will add the code in our script so that the resources are getting built every time we execute it. Change the gtk-todo script as follows.

#!/usr/bin/env ruby

require 'gtk3'
require 'fileutils'

# Require all ruby files in the application folder recursively
application_root_path = File.expand_path(__dir__)
Dir[File.join(application_root_path, '**', '*.rb')].each { |file| require file }

# Define the source & target files of the glib-compile-resources command
resource_xml = File.join(application_root_path, 'resources', 'gresources.xml')
resource_bin = File.join(application_root_path, 'gresource.bin')

# Build the binary
system("glib-compile-resources",
       "--target", resource_bin,
       "--sourcedir", File.dirname(resource_xml),
       resource_xml)

at_exit do
  # Before existing, please remove the binary we produced, thanks.
  FileUtils.rm_f(resource_bin)
end

app = ToDo::Application.new
puts app.run

and execute it. This happens in the console and it’s fine, we’ll fix it later on:

/.../gtk-todo-tutorial/resources/gresources.xml: Failed to locate 'ui/application_window.ui' in any source directory.

What we did:

added a require statement for the fileutils library so that we can use it in the at_exit call
defined the source and target files of the glib-compile-resources command
executed the glib-compile-resources command
set a hook so that before exiting the script (before the application exits) the binary file gets deleted so next time it gets build again

Loading the resources binary file

Ok, we described the resources, we packed them in a binary file. Now we have to load them and register them in the application so that we can use them. This is so easy as adding the following two lines before the at_exit hook.

resource = Gio::Resource.load(resource_bin)
Gio::Resources.register(resource)

That’s it. From now on, we are able to use the resources from anywhere inside the application (we’ll see how later on). Well now the script fails since it can’t load the binary which is not produced but…be patient, we are going to get to the interesting part soon. Actually now.

Designing the main application window

Introducing glade

Open glade.

A quick description of what you see.

On the left section there is a list of widgets which you can drag and drop in the middle section given that a widget can be placed there. For example, you can’t add a top level window inside a label widget. I will be calling this as the Widget section from now on.
On the middle section you see your widgets as they will appear (most of the times) in the application. I will be calling this as the Design section from now on.
On the right section there are two subsections:
- the top section contains the hierarchy of the widgets as added to the resource. I will be calling this as the Hierarchy section from now on.
- the bottom section contains all the properties that you can configure via Glade for a given selected widget of the aforementioned top section. I will be calling this as the Properties section from now on.

I will try to describe the steps for building this tutorial’s UI using Glade but if you are interested in building GTK+ applications you should take a look at the resources & tutorials for using the tool on the official page.

Create the application window design

We are going to create the application window. As you can guess, all we have to do is drag the widget ‘Application Window’ from the widget section to the design section.

Gtk::Builder is an object used in GTK+ applications to read textual descriptions of a user interface (like the one we will build via Glade) and build the described objects-widgets.

In the properties section, the first property is the ID and it has a default value applicationWindow1. If we let this property as is, then later on, through our code we would create a Gtk::Builder that would load the file produced by glade and in order to obtain the application window we would have to use something line:

application_window = builder.get_object('applicationWindow1')

application_window.signal_connect 'whatever' do |a,b|
...

The application_window object would be of class Gtk::ApplicationWindow and thus whatever we had to add to its behavior (like setting its title) would take place out of the original class. Also, as shown in the snippet above, the code to connect to a signal of the window would be placed inside the file that instantiated it.

Good news though, a GTK+ feature introduced in 2013 allows the creation of composite widget templates which among other advantages allows as to define the custom class for the widget (which eventually derives from an existing GTK::Widget class in general). Don’t worry if you are confused. You are going to understand what is going on after we write some code and view the results.

Now, in order to define our design as a template, check the Composite checkbox in the property widget. Note that the ID property changed to Class Name. Fill in there TodoApplicationWindow. This is the class we are going to create in our code to represent this widget.

Save the file with name application_window.ui in a new folder named ui inside the resources. If you open the file from an editor you will see this:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Generated with glade 3.18.3 -->
<interface>
  <requires lib="gtk+" version="3.12"/>
  <template class="TodoApplicationWindow" parent="GtkApplicationWindow">
    <property name="can_focus">False</property>
    <child>
      <placeholder/>
    </child>
  </template>
</interface>

As you can see, our widget has a class and parent attribute. Following the parent class attribute convention, obviously, our class has to be defined inside a module named Todo. Before getting there, let’s try to start the application by executing the script (./gtk-todo).

Yeah! It starts!

Create the application window class

While running the application, if you check the contents of the application’s root directory you can see the gresource.bin file there. Even though the application starts successfully because the resource bin is present and it can register it, we do not use it yet. We still initiate an ordinary Gtk::ApplicationWindow in our application.rb file and that all we show. Time to create our custom application window class.

Create a file named application_window.rb, place it under application/ui/todo folder and add the following content.

module Todo
  class ApplicationWindow < Gtk::ApplicationWindow
    # Register the class in the GLib world
    type_register

    class << self
      def init
        # Set the template from the resources binary
        set_template resource: '/com/iridakos/gtk-todo/ui/application_window.ui'
      end
    end

    def initialize(application)
      super application: application

      set_title 'GTK+ Simple ToDo'
    end
  end
end

We defined the init method as a singleton method on the class after opening the eigenclass in order to bind the template of this widget to the previously registered resource file.

Before that, we called the type_register class method which registers and make available our custom widget class to the GLib world.

Finally, each time we create an instance of this window, we set its title to GTK+ Simple ToDo.

Now, let’s go back to the application.rb file and use what we just implemented:

module ToDo
  class Application < Gtk::Application
    def initialize
      super 'com.iridakos.gtk-todo', Gio::ApplicationFlags::FLAGS_NONE

      signal_connect :activate do |application|
        window = Todo::ApplicationWindow.new(application)
        window.present
      end
    end
  end
end

Execute the script.

Define the model

For simplicity, we are going to save the ToDo items in files in json format under a dedicated hidden folder in user’s home directory. Of course, in a real life application we would use a database but this is out of the scope of this tutorial.

Our Todo::Item model will have the following properties:

id: the id of the item
title: the title
notes: any notes
priority: its priority
creation_datetime: the date & time the item was created
filename: the name of the file that an item is saved to

Create a file named item.rb under the application/models directory, with contents:

require 'securerandom'
require 'json'

module Todo
  class Item
    PROPERTIES = [:id, :title, :notes, :priority, :filename, :creation_datetime].freeze

    PRIORITIES = ['high', 'medium', 'normal', 'low'].freeze

    attr_accessor *PROPERTIES

    def initialize(options = {})
      if user_data_path = options[:user_data_path]
        # New item. When saved, it will be placed under the :user_data_path value
        @id = SecureRandom.uuid
        @creation_datetime = Time.now.to_s
        @filename = "#{user_data_path}/#{id}.json"
      elsif filename = options[:filename]
        # Load an existing item
        load_from_file filename
      else
        raise ArgumentError, 'Please specify the :user_data_path for new item or the :filename to load existing'
      end
    end

    # Loads an item from a file
    def load_from_file(filename)
      properties = JSON.parse(File.read(filename))

      # Assign the properties
      PROPERTIES.each do |property|
        self.send "#{property}=", properties[property.to_s]
      end
    rescue => e
      raise ArgumentError, "Failed to load existing item: #{e.message}"
    end

    # Resolves if an item is new
    def is_new?
      !File.exists? @filename
    end

    # Saves an item to its `filename` location
    def save!
      File.open(@filename, 'w') do |file|
        file.write self.to_json
      end
    end

    # Deletes an item
    def delete!
      raise 'Item is not saved!' if is_new?

      File.delete(@filename)
    end

    # Produces a json string for the item
    def to_json
      result = {}
      PROPERTIES.each do |prop|
        result[prop] = self.send prop
      end

      result.to_json
    end
  end
end

As you can see, we defined methods to:

initialize an item
- as new by defining the :user_data_path in which it will be saved later on
- as existing by defining the :filename to be loaded from. The filename must be a json file previously generated by an item
load an item from a file
resolve whether an items is new or not (saved at least once in the :user_data_path or not)
save an item by writing its json string to a file
delete an item
produce the json string of an item as a hash of its properties

Add a new item

Create the button

Let’s add a button to our application window for adding a new item. Open the resources/ui/application_window.ui file in glade.

Drag a Button from the widget section to the design section.
In the properties section, set its ID value to add_new_item_button.
Near the bottom of the General tab in the properties section there’s a text area just below the Label with optional image option. Change its value from button to Add new item
Save the file and execute the script

Don’t worry, we will improve the design later on. Now, let’s see how to connect functionality to our button’s clicked event.

First, we have to update our application window class so that it learns about its new child, the button with id add_new_item_button. Then, we can access the child to alter its behavior.

Change the init method as follows:

def init
  # Set the template from the resources binary
  set_template resource: '/com/iridakos/gtk-todo/ui/application_window.ui'

  bind_template_child 'add_new_item_button'
end

Pretty simple, right? The bind_template_child method does exactly what it says and from now on every instance of our Todo::ApplicationWindow class will have an add_new_item_button method to access the related button. So, let’s alter the initialize method as follows.

def initialize(application)
  super application: application

  set_title 'GTK+ Simple ToDo'

  add_new_item_button.signal_connect 'clicked' do |button, application|
    puts "OMG! I AM CLICKED"
  end
end

As you can see, we access the button by the add_new_item_button method and we define what we want to take place when clicked. Restart the application and try clicking it. In the console you should see the message OMG! I AM CLICKED every time you click the button.

What we want though to happen when we click this button is to show a new window through which we will save a ToDo item. You guessed right. Glade o’clock.

Create the new item window

Create a new project in Glade by pressing the most left icon of the top bar or by selecting File > New from the application menu.
Drag a Window from the widget section to the design area.
Check its Composite property and name the class TodoNewItemWindow.

Drag a Grid from the widget section and place it in the window we added in the previous steps.
Set its rows number to 5 and its columns number to 2 in the window that popped up.
In the General tab of its properties window, set its Rows spacing and Columns spacing to 10 (the numbers are in pixels).
In the Common tab of the properties section, set the Widget Spacing > Margins > Top, Bottom, Left, Right all to 10 so that the contents are not stuck to the borders of the window.

Drag four times a Label widget from the widget section and place them in each row of the Grid.
Change their Label property from top to bottom as:
- Id:
- Title:
- Notes:
- Priority:
In the General tab of the properties section, change the Alignment and Padding > Alignment > Horizontal property from 0.50 to 1 for each property. This will align the label text on the right.
This step is optional but I suggest that you do it: We will not bind these labels in our window since we don’t need to alter their state or behavior. So in this context, we don’t need to set a descriptive id for each of them like we did for the add_new_item_button button in the application window. BUT. We are going to add more elements to our design and the hierarchy of the widgets in Glade will be hard to read with all the label1. label2. So set a descriptive id to each to make our lives easier (like id_label, title_label, notes_label, priority_label). I even set the grid’s id to main_grid cause I don’t like seeing numbers in ids or variable names :)

Drag a Label from the widget section to the second column of the grid’s first row. The id is automatically generated by our model thus we won’t allow editing so a label to display it is more than enough.
Set the ID property to id_value_label.
Set the Alignment and Padding > Alignment > Horizontal property to 0 so that the text aligns on the left.
We are going to bind this widget to our window class so that we can change its text each time we load the window so setting a label through glade is not needed but doing so makes the design look closer to what it’ll look like when rendered with actual data. So you can optionally set a label here to whatever suits you better. I set mine to id-of-the-todo-item-here.

Drag a Text Entry from the widget section to the second column of the second row of the grid.
Set its ‘ID’ property to title_text_entry. As you may have noticed, I prefer obtaining the widget type in the id so that the code in the class becomes more readable later on.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal checkbox and turn on the switch which is right next to it. This way, the widget will expand horizontally every time its parent (a.k.a. the grid) is resized.

Drag a Text View from the widget section to the second column of the third row of the grid.
Set its ID to notes. Nope, just testing you. Set its ID property to notes_text_view.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal, Vertical checkboxes and turn on the switches which are right next to them. This way, the widget will expand horizontally and vertically every time its parent (a.k.a. the grid) is resized.

Drag a Combo Box from the widget section to the second column of the forth row of the grid.
Set its ID to priority_combo_box.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal checkbox and turn on the switch which is right next to it. This way, the widget will expand horizontally every time its parent (a.k.a. the grid) is resized.
This widget is actually a drop down element and we are going to populate its values that can be selected by the user when it shows up inside our window class.

Drag a Button Box from the widget section to the second column of the last row of the grid.
On the popped up window select 2 number of items.
In the General tab of the properties section set the Box Attributes > Orientation property to Horizontal.
In the General tab of the properties section set the Box Attributes > Spacing property to 10.
In the Common tab of the properties section set the Widget Spacing > Alignment > Horizontal to Center.
Again, this widget won’t be altered by our code but you can give it a descriptive ID for readability. I named mine actions_box

Drag a Button widget twice and place it to each of the two boxes of the button box widget we added in the previous step.
Set their ID properties to cancel_button & save_button respectively.
In the General tab of the properties window, set their Button Content > Label with option image property to Cancel and Save respectively.

The window is ready. Save the file under resources/ui/new_item_window.ui.

Time to port it in our application.

Implement the new item window class

Before implementing the new class, we must update our GResource description file a.k.a. resources/gresources.xml to obtain the new resource:

<?xml version="1.0" encoding="UTF-8"?>
<gresources>
  <gresource prefix="/com/iridakos/gtk-todo">
    <file preprocess="xml-stripblanks">ui/application_window.ui</file>
    <file preprocess="xml-stripblanks">ui/new_item_window.ui</file>
  </gresource>
</gresources>

Now we can create the new window class. Create a file under application/ui/todo named new_item_window.rb and set its contents as follows.

module Todo
  class NewItemWindow < Gtk::Window
    # Register the class in the GLib world
    type_register

    class << self
      def init
        # Set the template from the resources binary
        set_template resource: '/com/iridakos/gtk-todo/ui/new_item_window.ui'
      end
    end

    def initialize(application)
      super application: application
    end
  end
end

Nothing special here. We just changed the template resource to point to the correct file of our resources.

We have to change the add_new_item_button code that executes on the clicked signal to show the new item window. Go ahead and change that code in application_window.rb to this:

add_new_item_button.signal_connect 'clicked' do |button|
  new_item_window = NewItemWindow.new(application)
  new_item_window.present
end

Let’s see what we have done. Start the application and click on the Add new item button. Ta daaaaaaa

Of course, nothing happens when pressing the buttons. We will change that.

First we will bind the ui widgets in the Todo::NewItemWindow class.

Change the init method the to this:

def init
  # Set the template from the resources binary
  set_template resource: '/com/iridakos/gtk-todo/ui/new_item_window.ui'

  # Bind the window's widgets
  bind_template_child 'id_value_label'
  bind_template_child 'title_text_entry'
  bind_template_child 'notes_text_view'
  bind_template_child 'priority_combo_box'
  bind_template_child 'cancel_button'
  bind_template_child 'save_button'
end

This window is going to be shown either when creating a new Todo item or editing an existing one. Thus the new_item_window naming is not very valid. This was intentional though so that we refactor the code later (No it was not :D I made a mistake when writing the tutorial. In any case, we’ll refactor later on).

For now, we will update the initialize method of the window to require one extra parameter, the Todo::Item to be created or edited. We can then set a more meaningful window title and change the children widgets to reflect the current item.

Change the initialize method to this:

def initialize(application, item)
  super application: application
  set_title "ToDo item #{item.id} - #{item.is_new? ? 'Create' : 'Edit' } Mode"

  id_value_label.text = item.id
  title_text_entry.text = item.title if item.title
  notes_text_view.buffer.text = item.notes if item.notes

  # Configure the combo box
  model = Gtk::ListStore.new(String)
  Todo::Item::PRIORITIES.each do |priority|
    iterator = model.append
    iterator[0] = priority
  end

  priority_combo_box.model = model
  renderer = Gtk::CellRendererText.new
  priority_combo_box.pack_start(renderer, true)
  priority_combo_box.set_attributes(renderer, "text" => 0)

  priority_combo_box.set_active(Todo::Item::PRIORITIES.index(item.priority)) if item.priority
end

and add the constant PRIORITIES in the application/models/item.rb file just below the PROPERTIES constant as:

PRIORITIES = ['high', 'medium', 'normal', 'low'].freeze

What did we do here?

We set the window’s title to a string containing the id of the current item and the mode depending on whether the item is now being created or edited.
We set the id_value_label text to display the current item’s id.
We set the title_text_entry text to display the current item’s title.
We set the notes_text_view text to display the current item’s notes.

We create a model for the priority_combo_box whose entries are going to have only one String value. At a first sight, a Gtk::ListStore model might look a little confusing. I will try to explain how it works now.

Suppose we want to display in a combo box a list of country codes and their respective country names.
We would create a Gtk::ListStore defining that its entries would consist of two string values: one for the country code and one for the country name. Thus we would initialize the ListStore as:
```
model = Gtk::ListStore.new(String, String)
```

In order to fill the model with data we would do something like this (make sure you don’t miss the comments in the snippet):

[['gr', 'Greece'], ['jp','Japan'], ['nl', 'Netherlands']].each do |country_pair|
  entry = model.append
  # Each entry has two string positions since that's how we initialized the Gtk::ListStore
  # Store the country code in position 0
  entry[0] = country_pair[0]
  # Store the country name in position 1
  entry[1] = country_pair[1]
end

We also had to configure the combo box to render two text columns/cells (again, make sure you don’t miss the comments in the snippet):

country_code_renderer = Gtk::CellRendererText.new
# Add the first renderer
combo.pack_start(country_code_renderer, true)
# Use the value in index 0 of each model entry a.k.a. the country code
combo.set_attributes(country_code_renderer, 'text' => 0)

country_name_renderer = Gtk::CellRendererText.new
# Add the second renderer
combo.pack_start(country_name_renderer, true)
# Use the value in index 1 of each model entry a.k.a. the country name
combo.set_attributes(country_name_renderer, 'text' => 1)

I hope I made it a little more clearer…

We add a simple text renderer in the combo box and instruct it to display the one and only value of each model’s entry (a.k.a. position 0). Imagine that our model is something like [['high'],['medium'],['normal'],['low']] and 0 is actually the first element of each sub-array. I will stop with the model-combo-text-renderer explanations now…

Configure the user data path

Remember that when initializing a new Todo::Item (not an existing one) we had to define a :user_data_path in which it would be saved. We are going to resolve this path when the application starts and make it accessible from all the widgets.

All we have to do is check if the .gtk-todo-tutorial path exists inside the user’s home ~ directory. If not, then we will create it. Then we set this as an instance variable of the application. All widgets have access to the application instance. Sooooo….all widgets have access to this user path variable.

Change the application/application.rb file to this:

module ToDo
  class Application < Gtk::Application
    attr_reader :user_data_path

    def initialize
      super 'com.iridakos.gtk-todo', Gio::ApplicationFlags::FLAGS_NONE

      @user_data_path = File.expand_path('~/.gtk-todo-tutorial')
      unless File.directory?(@user_data_path)
        puts "First run. Creating user's application path: #{@user_data_path}"
        FileUtils.mkdir_p(@user_data_path)
      end

      signal_connect :activate do |application|
        window = Todo::ApplicationWindow.new(application)
        window.present
      end
    end
  end
end

One last thing that we have to do before testing what we have done so far is to instantiate the Todo::NewItemWindow when the add_new_item_button is clicked complying with the changes we made a.k.a. change the code in application_window.rb to this:

add_new_item_button.signal_connect 'clicked' do |button|
  new_item_window = NewItemWindow.new(application, Todo::Item.new(user_data_path: application.user_data_path))
  new_item_window.present
end

Start the application and click on the Add new item button. Ta daaaaaa (note the - Create mode part in the title).

Cancel the item creation/update

In order to close the Todo::NewItemWindow window when user clicks the cancel_button all we have to do is to add this to the window’s initialize method:

cancel_button.signal_connect 'clicked' do |button|
  close
end

close is an instance method of the Gtk::Window class that surprisingly enough closes the window.

Save the item

Saving an item involves two steps:

Update the item’s properties based on the widgets’ values
Call the save! method on the Todo::Item instance

Again, our code will be placed in the initialie method of the Todo::NewItemWindow:

save_button.signal_connect 'clicked' do |button|
  item.title = title_text_entry.text
  item.notes = notes_text_view.buffer.text
  item.priority = priority_combo_box.active_iter.get_value(0) if priority_combo_box.active_iter
  item.save!
  close
end

Note that we again close the window after saving the item.

Let’s try that out.

Pressing save and navigating to your ~/.gtk-todo-tutorial folder you should see a file there. Mine had the following contents:

{
	"id": "3d635839-66d0-4ce6-af31-e81b47b3e585",
	"title": "Optimize the priorities model creation",
	"notes": "It doesn't have to be initialized upon each window creation.",
	"priority": "high",
	"filename": "/home/iridakos/.gtk-todo-tutorial/3d635839-66d0-4ce6-af31-e81b47b3e585.json",
	"creation_datetime": "2018-01-25 18:09:51 +0200"
}

Don’t forget to try out the cancel button as well.

Awesome!!!

View ToDo items

We have left the Todo::ApplicationWindow to contain only one button. Time to change that.

We want the window to have the Add new item on the top but below it there should be a list with all of our todo items. To accomplish that we are going to add a Gtk::ListBox in our design which can contain any number of rows.

Update the application window

Open the resources/ui/application_window.ui file in Glade.
If you try to drag a List Box widget from the widget section directly on the window nothing happens. That is normal. First we have to split the window in two parts. One part for the button and one for the list box. Bear with me.
Right click on the new_item_window in the hierarchy section and select Add parent > Box.
In the popped up window, set that you need 2 items.
The orientation of the box is already vertical so we are fine.

Now, drag a List Box and place it on the free are of the previously added box.
Set its ID property to todo_items_list_box
Set is Selection mode to None since we won’t provide such a functionality.

Design the ToDo item list box row

Each row of the list box that we created in the previous step is going to be more complex than a row of text. It is going to contain widgets that will allow the user to expand an item’s notes, and to delete or edit the item.

Create a new project in Glade as we did for the new_item_window.ui. Save it under resources/ui/todo_item_list_box_row.ui.
Unfortunately, at least in my version of Glade, there is no List Box Row widget in the widget section so in order to add one directly as the top level widget of our project, we will do it in a kinda hackish way.
Drag a List Box from the widget section to the design area.
Inside the hierarchy section right click on the List Box and select Add Row

Inside the hierarchy section right click on the newly added List Box Row which is nested under the List Box and select Remove parent. There it is. The List Box Row is the top level widget of the project now.

Check the widget’s Composite property and set its name to TodoItemListBoxRow.
Drag a Box from the widget section to the design area inside our List Box Row.
Set 2 items in the popped up window.
Set its ID property to main_box

Drag another Box from the widget section to the first row of the previously added box.
Set 2 items in the popped up window.
Set its ID property to todo_item_top_box.
Set its Orientation property to Horizontal.
Set its Spacing (General tab) property to 10.

Drag a Label from the widget section to the first column of the todo_item_top_box.
Set its ID property to todo_item_title_label.
Set its Alignment and Padding > Alignment > Horizontal property to 0.00.
In the Common tab of the properties section, check the Widget Spacing > Expand > Horizontal checkbox and turn on the switch which is right next to it so that the label expands to available space.

Drag a Button from the widget section to the second column of the todo_item_top_box.
Set its ID property to details_button
Check the Button Content > Label with optional image radio and type ... (three dots).

Drag a Revealer widget from the widget section to the second row of the main_box.
Turn off the Reveal Child switch in the General tab.
Set its ID property to todo_item_details_revealer.
Set its Transition type property to Slide Down.

Drag a Box from the widget section to the reveal space.
Set its items to 2 in the popped up window.
Set its ID property to details_box.
In the Common tab, set its Widget Spacing > Margins > Top property to 10.

Drag a Button Box from the widget section to the first row of the details_box.
Set its ID property to todo_item_action_box.
Set its Layout style property to expand.

Drag two Button widgets to the first and second column of the todo_item_action_box respectively.
Set their ID properties to delete_button and edit_button respectively.
Set their Button Content > Label with optional image properties to Delete and Edit respective.

Drag a Viewport widget from the widget section to the second row of the details_box.
Set its ID property to todo_action_notes_viewport.
Drag a Text View widget from the widget section to the todo_action_notes_viewport that we just added.
Set is ID to todo_item_notes_text_view.
Uncheck its Editable property in the General tab of the properties section.

Create the ToDo item list box row class

Now we will create the class reflecting the user interface of the list box row which we just created.

First we have to update our GResource description file to include the newly created design. Change the resources/gresources.xml file as follows:

<?xml version="1.0" encoding="UTF-8"?>
<gresources>
  <gresource prefix="/com/iridakos/gtk-todo">
    <file preprocess="xml-stripblanks">ui/application_window.ui</file>
    <file preprocess="xml-stripblanks">ui/new_item_window.ui</file>
    <file preprocess="xml-stripblanks">ui/todo_item_list_box_row.ui</file>
  </gresource>
</gresources>

Create a file named item_list_box_row.rb inside the application/ui folder and add the following content.

module Todo
  class ItemListBoxRow < Gtk::ListBoxRow
    type_register

    class << self
      def init
        set_template resource: '/com/iridakos/gtk-todo/ui/todo_item_list_box_row.ui'
      end
    end

    def initialize(item)
      super()
    end
  end
end

We will not bind any children at the moment here.

When starting the application, we have to search for files in the :user_data_path and for each file we must create a Todo::Item instance. For each instance, we must also add a new Todo::ItemListBoxRow to the Todo::ApplicationWindow’s todo_items_list_box list box. One thing at a time.

First of all, let’s bind the todo_items_list_box in the Todo::ApplicationWindow class. Change the init method as follows:

def init
  # Set the template from the resources binary
  set_template resource: '/com/iridakos/gtk-todo/ui/application_window.ui'

  bind_template_child 'add_new_item_button'
  bind_template_child 'todo_items_list_box'
end

Next, we will add an instance method in the same class that will be responsible to load the todo list items in the related list box. Add this code in Todo::ApplicationWindow.

def load_todo_items
  todo_items_list_box.children.each { |child| todo_items_list_box.remove child }

  json_files = Dir[File.join(File.expand_path(application.user_data_path), '*.json')]
  items = json_files.map{ |filename| Todo::Item.new(filename: filename) }

  items.each do |item|
    todo_items_list_box.add Todo::ItemListBoxRow.new(item)
  end
end

and then call this method at the end of the initialize method.

def initialize(application)
  super application: application

  set_title 'GTK+ Simple ToDo'

  add_new_item_button.signal_connect 'clicked' do |button|
    new_item_window = NewItemWindow.new(application, Todo::Item.new(user_data_path: application.user_data_path))
    new_item_window.present
  end

  load_todo_items
end

Note: We first make sure we empty the list box of its current children rows and we refill it. This way, we will call this method after saving a Todo::Item via the signal_connect of the save_button of the Todo::NewItemWindow and the parent application window will be reloaded! Here’s the updated code (in application/ui/new_item_window.rb):

save_button.signal_connect 'clicked' do |button|
  item.title = title_text_entry.text
  item.notes = notes_text_view.buffer.text
  item.priority = priority_combo_box.active_iter.get_value(0) if priority_combo_box.active_iter
  item.save!

  close

  # Locate the application window
  application_window = application.windows.find { |w| w.is_a? Todo::ApplicationWindow }
  application_window.load_todo_items
end

Previously, we used this code:

json_files = Dir[File.join(File.expand_path(application.user_data_path), '*.json')]

in order to find all the names of the files that exist in the application user data path with json extension.

Let’s see what we’ve created. Start the application and try adding a new ToDo item. After pressing the Save button you should see the parent Todo::ApplicationWindow being automatically updated with the new item!

What’s left to do, is to complete the functionality of the Todo::ItemListBoxRow.

We will first bind the widgets. Change the init method of the Todo::ItemListBoxRow class as follows:

def init
  set_template resource: '/com/iridakos/gtk-todo/ui/todo_item_list_box_row.ui'

  bind_template_child 'details_button'
  bind_template_child 'todo_item_title_label'
  bind_template_child 'todo_item_details_revealer'
  bind_template_child 'todo_item_notes_text_view'
  bind_template_child 'delete_button'
  bind_template_child 'edit_button'
end

Then, we are going to setup the widgets based on the item of each row.

def initialize(item)
  super()

  todo_item_title_label.text = item.title || ''

  todo_item_notes_text_view.buffer.text = item.notes

  details_button.signal_connect 'clicked' do
    todo_item_details_revealer.set_reveal_child !todo_item_details_revealer.reveal_child?
  end

  delete_button.signal_connect 'clicked' do
    item.delete!

    # Locate the application window
    application_window = application.windows.find { |w| w.is_a? Todo::ApplicationWindow }
    application_window.load_todo_items
  end

  edit_button.signal_connect 'clicked' do
    new_item_window = NewItemWindow.new(application, item)
    new_item_window.present
  end
end

def application
  parent = self.parent
  parent = parent.parent while !parent.is_a? Gtk::Window
  parent.application
end

As you can see, when the details_button is clicked, we instruct the todo_item_details_revealer to swap the visibility of its contents.
After deleting an item, we find the application’s Todo::ApplicationWindow in order to call its load_todo_items as we did after saving an item.
When clicking to edit a button, we create a new instance of the Todo::NewItemWindow passing as item the current item. Works like a charm :D
Finally, we had to reach at the application parent of a list box row so we defined a simple instance method application that navigates through the widget’s parents until it reaches a window from which it can obtain the application object.

Save and run the application. There it is.

This has been a really long tutorial and even though there are so many items that we haven’t covered I think we better end it here.

Long post, cat photo.

Useful links

Using elasticsearch in a Rails application

Sun, 03 Dec 2017 08:30:00 +0200

In this tutorial we are going to create a Ruby on Rails application that will use elasticsearch to allow users to store and search their content. The sample application will be a stupid simple blog and the data will be, what else, posts. The integration with elasticsearch will be accomplished via the elasticsearch gems for Rails and we will use Kibana to view and check our index configuration.

Note: The code of this tutorial is available at GitHub.

What is elasticsearch?

Elasticsearch is an opensource JSON-based search engine that allows us to search indexed data quickly and with options that are not provided by classic data stores. You will better understand the tool’s usage once we start working on it later on on this tutorial. For now, just keep that we will store data to a database and will index them to an elasticsearch instance and afterwards we will search the indexed data via a Rails application.

Prerequisites

Elasticsearch

Providing instructions for installing elasticsearch is out of this tutorial’s scope. I recommend visiting the tool’s related documenation site.

After the installation, just make sure to start the service if it’s not started.

You can check that with curl http://localhost:9200.

It should give something like:

{
  "name" : "asdfasdf",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "asasdfdf-as",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "asdf",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

As you can see, we will use elasticsearch version 6.0.0 in this tutorial.

Ruby

This tutorial is using Ruby 2.4.2. You can check yours using: ruby --version or via RVM with rvm list.

Rails

Given that you have ruby installed, we will need the rails gem as well. We will use the latest stable version at the moment this turorial is written, which is 5.1.4.

gem install rails -v 5.1.4

You can check the version of your installed rails gem with:

rails --version

Important note: when working on tutorials, always try to use the same versions of software as described in them to avoid losing time figuring out what is wrong if you don’t see the same results as provided in the tutorial. With that in mind, make sure that you have the following versions installed:

Elasticsearch 6.0.0
Ruby 2.4.2
Rails 5.1.4

Create the rails application

Time to create our application. Let’s name it elastic_rails.

Navigate to your development directory (mine is under ~/development/projects/ruby for ruby projects) and execute the following command:

rails new elastic_rails

After a bunch of logs, the application has been created. Let’s start it to confirm that everything is good.

cd elastic_rails
rails s

Note: the s in the command rails s is an alias for server. You can start the server with rails server too.

As declared in the puma startup log, the server is listening on port 3000. Let’s visit the application from the browser. Navigate to http://localhost:3000.

You should be seeing this:

Define the model

Let’s say that we had a blog and we wanted to allow our visitors to search our posts. We are going to define the Post model.

By default, our new Rails application is configured to use the sqlite3 database and that works fine in our case since setting up a database and customizing rails to use it, is out of this tutorial’s scope.

Generate the model with the following command:

bundle exec rails generate model Post author:string title:string body:text tags:string published:boolean published_on:timestamp

and create the database along with the table for the posts with this command:

bundle exec rake db:create db:migrate

To make sure we’re good with the database, open a rails console with the command:

bundle exec rails c # `c` is an alias for `console`

Create a post, assign your desired values to its attributes and save it.

post = Post.new
post.title = 'My first post'
post.body = '<h1>Hello world! This is my first post!</h1>'
post.author = 'Lazarus Lazaridis'
post.tags = 'first, salute, hello'
post.published = true
post.published_on = DateTime.now
post.save

Success!

Define the controller

We will also define a controller to handle posts. We are going to scaffold the controller instead of creating all the files ourselves to keep within the scope of the tutorial. So, from the command line execute:

bundle exec rails generate scaffold_controller post

Tadaaaaa

Now, we are going to define the post routes and set the index page of posts as the application’s homepage.

Open the routes file config/routes.rb and edit as shown below:

Rails.application.routes.draw do
  # For details on the DSL available within this file, see http://guides.rubyonrails.org/routing.html
  root 'posts#index'

  resources :posts
end

Let’s start the server (if it is already started, restart it) and navigate to http://localhost:3000.

The links don’t do much at the moment but we are going to change this later on.

Time to talk elastic.

Elasticsearch gem

As described in the overview, we are going to use existing gems to integrate with elasticsearch, the elasticsearch-rails & elasticsearch-model gems.

Open your Gemfile and add the following lines:

gem 'elasticsearch-model', git: 'git://github.com/elastic/elasticsearch-rails.git', branch: 'master'
gem 'elasticsearch-rails', git: 'git://github.com/elastic/elasticsearch-rails.git', branch: 'master'

Important note: at the moment this tutorials was being written, we had to use the master branch of the elasticsearch-rails repository so that we could use version 6.0.0 of elasticsearch. If you have problems in the future due to releases of newer versions of elasticsearch (and the master branch is not compatible with version 6.0.0), make sure to update the branch in the Gemfile above accordingly.

We must then fetch the gem by executing the following command from withing the application’s root directory:

bundle

Perfect. We are now ready to tell our model that it’s going to be indexed in elasticsearch.

Open the app/models/post.rb file and edit it to look like this:

class Post < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks
end

What we did here?

We actually defined that Post is not only an application record in the rails world but also a model in the elasticsearch world.
We added callbacks so that each time a post is saved/updated/deleted, the related indexed data get updated accordingly too. Before explaining what does this mean, let’s call another elasticsearch friend to help.

Kibana

Kibana is a tool mainly allowing visualization of elasticsearch data. In our case, we will use kibana to see the effects of our code on elasticsearch.

Once again, follow the official instructions for installing kibana at your machine’s OS. I followed this guide for installing the tool via a repository in Ubuntu.

Start the kibana service and navigate to its home page at http://localhost:5601. We will use heavily the Dev Tools which is a powerful console talking to our elasticsearch engine.

On the left section of the console we write our commands and on the right part we see their execution output.

To view what indexes are currently present in our elasticsearch instance, execute the following command in Kibana:

GET _cat/indices?v

Well, at the moment there is only one index with the name .kibana which is obviously not ours. Lets go back to rails console and execute the command that will import our current posts (one post actually) to elasticsearch to see what happens.

Post.import

As you can, since we haven’t created the index for the posts, the command failed to execute. Reading the error we do understand that the gem automatically tried to find an index named as the plural form of our model name posts. Following the hint, we will execute the command, this time demanding the creation of the index if it doesn’t exist.

Post.import(force: true)

Back to kibana now, let’s see what has happened to our instance’s indices.

As you can see, the posts index was created successfully. But keep this in mind: the index was created automatically, we didn’t have the opportunity to define what fields should be indexed, what should each record in the index contain or set any other kind of configuration. Don’t worry though, later on we will change this approach. We will re-create the index explicitly the way we want it to be.

The Post.import command we execute before was supposed to also index all of our posts. If you remember, we only have inserted one single post so far. Let’s see what Kibana has to say about this. Type the following command in the Dev Tools:

GET posts/_search

The command will search for post records without any filters/terms/criteria in the posts index so we expect to get one result. And there it is.

We are interested in the hits attribute of the response’s JSON and escecially its _source attribute. As you can see, all of our model’s columns have been serialized in the document that has been indexed for that post in elasticsearch.

But do we need all these attributes there? And what if we needed attributes from children models there too? All these are configurable via specific commands of the elasticsearch-model gem which we already use. We are going to customize this behavior later on.

Let’s open the rails console once more and search in the elasticsearch data this time via our model’s methods.

results = Post.search 'world'
results.size
results.first

The first command executes the simplest kind of search in our index. And it brought results! The same will happen if you search for salute or ‘lazarus’ or post since our unique model contains these words in the title/body/tags/author fields.

Now try to search with worlds or posts:

Post.search('worlds').size
Post.search('posts').size

Hmmm. No results even though we would like elasticsearch to understand that we used the plural of words contained in our model. We are going to alter the configuration to match our needs.

Define the index

We are going to define the index ourselves instead of letting the elasticsearch gem define it automatically with defaults. To do so, edit the app/models/post.rb file to look like this:

class Post < ApplicationRecord
  include Elasticsearch::Model
  include Elasticsearch::Model::Callbacks

  settings do
    mappings dynamic: false do
      indexes :author, type: :text
      indexes :title, type: :text, analyzer: :english
      indexes :body, type: :text, analyzer: :english
      indexes :tags, type: :text, analyzer: :english
      indexes :published, type: :boolean
    end
  end
end

What did we do here?

When creating the index, we instructed to use only the fields that we actually want to be indexed skipping the published on date and the active record’s automatically generated created_at and updated_at attributes.
We used the english analyzer for the fields containing “normal text” so that cases like singular/plural version of terms are treated the same way (and much more).

Since our index had already been created automatically before, we have to re-create it. In the rails console execute to delete the index:

Post.__elasticsearch__.delete_index!

and execute to create it again this time with our configuration:

Post.import force: true

All good. Now try searching for ‘posts’:

> Post.search('posts').size
 => 1
> Post.search('salutation').size
 => 1
> Post.search('worlds').size
 => 1

As you can see, we don’t only fetch results for either singlular/plural versions of a term but also we can get results for both salute and salutation. This magic is provided due to the english analyzer that we defined in the english earlier.

All good so far but what if wanted to introduce filters on our search, for example searching only published posts?

Define searches

To demonstrate the filtered search, we will first add another post in our database, this time with the published flag set to false.

In the rails console:

post = Post.new
post.title = 'Elastic search and rails [WIP]'
post.author = 'iridakos'
post.body = '<h1>Integrating elasticsearch in your rails application</h1> In this post [wip]'
post.published = false
post.save

Reminder: since we included the Elasticsearch::Model::Callbacks in our model, the save command above triggered the related elasticsearch functionality that indexes the new post.

Now try searching for the word ‘post’ that exists in both of our posts:

> Post.search('post').size
 => 2
>

For convenience, we are going to create a class method in the Post model that will take the search query as argument but it will fetch only published posts. Add the following method in the app/models/post.rb file:

def self.search_published(query)
  self.search({
    query: {
      bool: {
        must: [
        {
          multi_match: {
            query: query,
            fields: [:author, :title, :body, :tags]
          }
        },
        {
          match: {
            published: true
          }
        }]
      }
    }
  })
end

What we actually did here is to define a query that must comply with two rules:

match the given term in any of the fields author, title, body & tags
have true value at the published field

Now, searching again for the post term we get only the published post.

> Post.search('post').size
 => 2
> Post.search_published('post').size
 => 1  

Show me what you did

Well, the application’s UI is very basic and we won’t improve it much. We will though create a new page to demonstrate the functionality we implemented. The general idea is to create a search page with a text field that will accept the user’s term and a button to proceed to post searching.

Add the route

First we have to update our routes with the new action that will serve the search functionality. Update the resources :posts of the config/routes.rb line to this:

resources :posts do
  collection do
    get :search
  end
end

Add the controller action

Now, in the posts controller we have to implement the new search action which will, what else, use the search_published action that we added in the Post model. Add the method before the private declaration.

def search
  query = params[:search_posts].presence && params[:search_posts][:query]

  if query
    @posts = Post.search_published(query)
  end
end

As you will see in the next section (with the view), the search form will have its method attribute set to get in order to be able to have bookmarkable search links. So, either user visits the search page for the first time or after a form submit, the same search action of the controller will be executed. To distinguish the cases, we check the presence of the related query parameter to see if we’re going to proceed to search in elasticsearch.

Add the view

Create a new file named search.html.erb under app/views/posts and add the following code in it:

<h1>Elastic search for posts</h1>

<%= form_for :search_posts, url: search_posts_path, method: :get do |form| %>
  <%= form.text_field :query, value: params[:search_posts].presence && params[:search_posts][:query] %>
  <%= form.submit 'Search' %>
<% end %>

<% unless @posts.nil? %>
  <% if @posts.empty? %>
    <h2>No posts found matching your query.</h2>
  <% else %>
    <h2>Search results</h2>

    <table style="width: 100%;">
      <thead>
         <tr>
           <th>Author</th>
           <th>Title</th>
           <th>Tags</th>
           <th>Published on</th>
           <th>Actions</th>
         </tr>
      </thead>

      <tbody>
        <% @posts.each do |post| %>
          <tr>
            <td><%= post.author %></td>
            <td><%= post.title %></td>
            <td><%= post.tags %></td>
            <td><%= post.published_on %></td>
            <td><%= link_to 'View', post_path(post.id) %></td>
          </tr>
        <% end %>
      <tbody>
    </table>
  <% end %>
<% end %>

Pretty self explanatory:

We add the form
We render nothing else if the @posts variable is nil (user hasn’t searched)
We render a message if we don’t find any posts - @posts.empty? == true
We render a table with the found posts - @posts.empty? == false
Note: even though the @posts variable doesn’t contain Post instances but elasticsearch results, we can still access the properties as if it was a Post. Whatever added in the _source property is accessible on the elasticsearch result as is.

Finally, let’s add a link to the homepage of the application so that we can use it to navigate to our search page. Open and edit app/views/posts/index.html.erb to look like this:

<p id="notice"><%= notice %></p>

<h1>Posts</h1>

<p>
  Press <%= link_to 'here', search_posts_path %> to search for posts.
</p>

<table>
  <thead>
    <tr>
      <th colspan="3"></th>
    </tr>
  </thead>

  <tbody>
    <% @posts.each do |post| %>
      <tr>
        <td><%= link_to 'Show', post %></td>
        <td><%= link_to 'Edit', edit_post_path(post) %></td>
        <td><%= link_to 'Destroy', post, method: :delete, data: { confirm: 'Are you sure?' } %></td>
      </tr>
    <% end %>
  </tbody>
</table>

<br>

<%= link_to 'New Post', new_post_path %>

Restart the server and navigate to home page.

Clicking on the here link, you reach at the search page.

Submitting a query you are still on the search page but this time with the results found on elasticsearch.

Conclusion

Elasticsearch is an awesome and powerful tool. This tutorial was written only to give an introduction of how to integrate Elasticsearch in a Rails application and does not cover the great and super cool stuff you can do with elasticsearch.

Useful links

Long post, cat photo. Here’s my elastic cat.

Rails sample API & UI application consuming it

Sun, 21 Jun 2015 10:00:00 +0300

I recently created a simple Rails 4 API named LofoCats API and a simple UI application consuming it named LofoCats UI (yes, I like cats, what else is new?)

You can find the code for the:

API: https://github.com/iridakos/lofocats_api
UI: https://github.com/iridakos/lofocats_ui

The concept of the app is pretty simple: register & view lost or found cats.

There are instructions on how to setup both apps in the projects’ README files.

Screenshots

Creating a simple ToDo application with Ruby on Rails - Part 4

Sun, 22 Dec 2013 09:00:00 +0200

In the previous three posts we created a simple ToDo application with Ruby on Rails. In this last part we are going to deploy the application to OpenShift.

You can find the complete code of the tutorial here.

OpenShift

OpenShift is a cloud application platform (by Red Hat).

Few months ago, when I was discovering Ruby & Ruby on Rails I started developing an application in order to practice.

When I completed the first version I wanted to have it online for demonstration purposes and thus I started exploring my options. I decided really fast since I had a requirement that only OpenShift (at least at that time) could fulfill: local storage. I wanted to allow users to upload files without using any other online storage service. I needed to have the files on the same server.

Now, after 4 months, I’m completely satisfied with my decision and not just because of the local storage:

I deploy my application in 5 minutes with a single command
great documentation and support
awesome tools for managing my “remote” machine - rch client tools
no surprises, what I deploy is what I see with no sudden downtimes, errors etc
there’s a free plan with 3 gears :)

So, let’s start.

We are going to use git so if you don’t have it yet, install it now.

Create an account to OpenShift for free here.

We will install the client tools (rhc). From the command line:

gem install rhc

and we are going to configure them with:

rhc setup

Follow the instructions and type your credentials:

Choose to generate the token so that you don’t have to login each time you interact with your remote machine (from now on we will call this gear):

If you don’t have any SSH keys, OpenShift will automatically generate one for you (nice) and will ask you if you want it to upload it to the server so that you can access your code. Say yes.

Next, you will be asked to create a domain under which your applications will be grouped. Choose one (this is going to be used in your applications public url ex: todo-yourdomain.rhcloud.com). I chose arubystory:

That’s all. We configured the tools to use our account.

OpenShifting the ToDo

Time to “create” the ToDo application on OpenShift.

When we will fire the command, the tool is going to create the application on OpenShift and then it will try to clone the remote repo to a local folder with the same name as the application. We want to skip the last part because we already have this folder. To do so, we are going to execute the command from the directory containing our project’s directory.

My todo project is under /Users/developer/development/projects/todo so I will execute it from /Users/developer/development/projects Doing so, the git clone will fail and we will have to manually connect our local repo with the remote later.

From the command line:

cd /to/the/directory/containing/the/todo/directory
rhc create-app todo ruby-1.9

Great. Now, let’s initialize our local git repo. From the command line:

cd todo
git init

We will add the remote git using the information provided with the following command:

rhc app-show todo

Copy the value of the Git URL because you are going to use it in the following command:

git remote add openshift [paste the value here]

Let’s commit. From the command line:

git add .
git commit -m "First commit"

And let’s merge the remote:

git pull openshift master

As you can see, we have a conflict in config.ru. Edit this file and change it to the following (we overwrite the remote changes with ours):

# This file is used by Rack-based servers to start the application.

require ::File.expand_path('../config/environment',  __FILE__)
run Rails.application

Commit the changes, from the command line:

git add config.ru
git commit -m "OpenShift merge"

And time to deploy the application! With one line. This:

git push openshift master

Visit the application url from your browser: http://todo-yourdomainhere.rhcloud.com

Don’t worry, we expected this. We haven’t setup the database for production. We will do this “sshing” to our gear. From the command line:

rhc ssh todo

Yes, this is your gear. Let’s setup the database. From the gears command line:

cd app-root/repo
RAILS_ENV=production rake db:setup

Exit the gear:

exit

and from your command line (not the gear’s one) restart the application:

rhc app-stop todo
rhc app-start todo

Give it some time and refresh your browser. There it is!

Now, if you try to sign up you’ll see the error message again.

This is expected too. Our email configuration was only for the development environment (we had it only in development.rb file and also we were using the mailcatcher for simulating an smtp server).

We obviously want emails to be really sent to our users in production so we must go with a real configuration. I will show you how to configure mailer to use a gmail account. Add the following in config/environments/production.rb

config.action_mailer.raise_delivery_errors = true
config.action_mailer.default_url_options = { host: 'http://todo-yourdomainhere.rhcloud.com/' }
config.action_mailer.asset_host = 'http://todo-yourdomainhere.rhcloud.com/'
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {
  address: 'smtp.gmail.com',
  port: 587,
  user_name: 'your_username@gmail.com',
  password: 'your_password',
  authentication: 'plain',
  enable_starttls_auto: true
}

Let’s deploy the fix. From the command line:

git add .
git commit -m "Mail setup in production"
git push openshift master

After the gear starts, refreshing the browser you’ll see the error message again. This time the problem is once again the database. But why? Take a look at your db/database.yml

As you can see, the database is saved under db/production.sqlite3. This file doesn’t exist in our repo. We are going to set this value pointing to a file that is not in the repo but to another directory of the gear! That directory already exists and OpenShift has taken care so that we can see what it is with an env variable: OPENSHIFT_DATA_DIR . Change the production database line to this:

database: <%=ENV['OPENSHIFT_DATA_DIR']%>/production.sqlite3

Once again:

git add .
git commit -m "Production database fix"
git push openshift master

Ssh to the gear again and re-setup the database:

rhc ssh todo

From the gear’s command prompt:

cd app-root/repo
RAILS_ENV=production rake db:setup

Let’s see that the database was created (gear’s command line).

cd $OPENSHIFT_DATA_DIR
ls

You should see the production.sqlite3 file there.

That’s it. We’re done! The application is successfully deployed to OpenShift.

Creating a simple ToDo application with Ruby on Rails - Part 3

Fri, 20 Dec 2013 09:00:00 +0200

This is the third part of the tutorial. In this part, we are going to use Devise & CanCan in order to add authentication and authorization features to the application.

In order to continue, you must complete the previous two parts if you haven’t done this already:

Done? Cool, let’s start.

You can find the complete code of the tutorial here.

MailCatcher

Our users will register to the application using their email address and they will receive a confirmation email in order to activate their account.

Before introducing Devise, you’ll get to know MailCatcher which is an awesome gem for simulating a real smtp server. We will configure the application to use it when in development mode.

Install the gem from the command line:

gem install 'mailcatcher'

and start the smtp server:

mailcatcher

As you can see from the output, we will be sending our mail through:

smtp://127.0.0.1:1025

and we will browse the sent emails via the web interface at:

http://127.0.0.1:1080

Visit the last one, you should be seeing something like this:

Now open the development.rb file which is located under config/environments and add the following lines:

config.action_mailer.default_url_options = { :host => 'localhost:3000' }
config.action_mailer.delivery_method = :smtp
config.action_mailer.smtp_settings = {:address => "localhost", :port => 1025}

The first line defines which host is going to be used while converting the relative paths in our emails to absolute ones.

The second ones defined the protocol to be used and the last one sets the settings of the smtp server.

Devise

Devise is a gem that will provide us with almost anything about authentication.

Open your Gemfile with your text editor and add the devise gem:

gem 'devise'

and then, from the command line execute the bundle installer:

bundle install

In order to have the gem’s initialization stuff automatically generated for us, execute from the command line again:

rails generate devise:install

This command will create two files:

config/initializer/devise.rb

and

config/locales/devise.en.yml

The first one configures the devise and second one has localized messages used by the gem.
Open the first one and comment out the line since we won’t be using this option:

# config.reconfirmable = true

Now, we are going to instruct devise to generate the model that will be used for our users:

rails generate devise User

With this command, we created our User model & the migration that will create the table for the model and we also added automatically the appropriate routes for registration/sign in/confirmation etc.

Before applying the migration, we will create another one because we want to enable the confirmable option so that our users confirm their account before start doing anything with the application.

To do so, we will generate a custom migration:

rails generate migration add_confirmable_to_users

Now open the generated file (xxxx_add_confirmable_to_users.rb) with your text editor and change its contents to the following:

    class AddConfirmableToUsers < ActiveRecord::Migration
      # Note: You can't use change, as User.update_all with fail in the down migration
      def self.up
        add_column :users, :confirmation_token, :string
        add_column :users, :confirmed_at, :datetime
        add_column :users, :confirmation_sent_at, :datetime
        # add_column :users, :unconfirmed_email, :string # Only if using reconfirmable
        add_index :users, :confirmation_token, :unique => true
        # User.reset_column_information # Need for some types of updates, but not for update_all.
        # To avoid a short time window between running the migration and updating all existing
        # users as confirmed, do the following
        User.update_all(:confirmed_at => Time.now)
        # All existing user accounts should be able to log in after this.
      end

      def self.down
        remove_columns :users, :confirmation_token, :confirmed_at, :confirmation_sent_at
        # remove_columns :users, :unconfirmed_email # Only if using reconfirmable
      end
    end

This migration will add columns to our users table to keep information about confirmations.
We are ready to migrate, from the command line:

rake db:migrate

We updated our users table but we need to update the model too. Open user.rb which is located under app/models/ and change it to this:

class User < ActiveRecord::Base
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :trackable, :validatable,
         :confirmable
end

We just added the :confirmable option.

Let’s check what is going on with our routes, from the command line:

rake routes

Wow, way too many new routes. Time to start the server:

rails server

Navigate to http://localhost:3000/users/sign_in

We have a sign in form!

Where did this view come from?

Well, devise has some default views that we use by default. But what if we want to use our own? This page doesn’t look good…

Thankfully, devise gives us the option to generate the default views inside our application’s views folder so that we can make changes on them.

By default, the generated views are of erb type and since we are using haml we will follow one more step.

From the command line:

rails generate devise:views

At this point, it would be a good exercise for you to convert all generated views to haml (don’t forget to change the file extension to .html.haml) but there is another way.

From the command line:

gem install html2haml
for file in app/views/devise/**/*.erb; do html2haml -e $file ${file%erb}haml && rm $file; done

I got a bunch of warnings from the last command but the conversion was successful.

Ok, now we can edit these files and make the views look what we want.

But instead of “centralizing” all devise’s forms file by file we have also another option. We can create another layout (besides our application’s default) and configure the application to use it if the executed action belongs to a devise controller.

Create a file with name devise.html.haml into app/views/layouts and add the following:

!!!
%html
  %head
    %title Todo
    = stylesheet_link_tag    "application", media: "all"
    = javascript_include_tag "application"
    = csrf_meta_tags
  %body
    .row-fluid
      .span10.offset1
        .hero-unit.text-center
          %h1
            ToDo
          %p
            Welcome to the tutorial's ToDo application
    .row-fluid
      .span4.offset4
        = yield

and add the following to the application_controller.rb:

layout :layout_by_controller

protected

def layout_by_controller
  devise_controller? ? 'devise' : 'application'
end

Refresh your browser, much better:

Let’s bootstrap that sign in button, edit the new.html.haml file located under app/views/devise/sessions and add the bootstrap class to the button:

= f.button :submit, "Sign in", class: 'btn btn-primary'

Now, I want you to visit the root path of the application, http://localhost:3000

Hmm, where is the authentication? We are going to add a filter in our application_controller.rb so that all actions require an authenticated user. Edit the file and add this line:

before_filter :authenticate_user!

Refresh, you should be automatically redirected to the sign in page.

Let’s create a user. Follow the sign up link (as an exercise, on each new devise view missing the bootstrap stuff, find the file and make the appropriate changes):

Press the sign up button and open a new tab to your browser to see the MailCatcher stuff (http://localhost:1080):

Magic!

Confirm you account following the link and login.

So far so good, but something wasn’t quite nice. We should be seeing some default devise generated messages for the actions we executed (like “Successfully registered” etc) but nothing was displayed.

Devise uses flash messages and currently we have done nothing about showing them.

We will create a simple partial under a new folder located at app/views/shared named _flash.html.haml with the following contents:

- flash.each do |type, value|
  .flash{ :class => type.to_s }
    = value

Trying to sign in with invalid credentials should show you an error message now:

Nice. Let’s add some user related stuff to our application layout so that when logged in a user can see his username and a logout link. Edit application.html.haml and add the following inside the hero-unit.

.text-center
  %strong= current_user.email
.text-center
  = link_to 'logout', destroy_user_session_path, method: :delete

We added the current user’s email (using the helper method current_user provided by devise) and a logout link pointing to the destroy session path from our routes.

Now, after a user logs in, you should see something like the following:

Ok, we have added user authentication. Now before accessing the home page and all of its task related actions, an authenticated user is required. But, which task should a user be able to view/edit/destroy?

CanCan

Before messing with CanCan, we should think a little about our models.

We have users and tasks. We want each user to manage his/her own tasks without being able to do anything to tasks that belong to another user.

So, we need to add a new attribute to our Task model that will keep track of the User it belongs to. We will do this firstly by creating a migration to add a user_id column to the users table:

rails generate migration add_column_user_id_to_tasks user_id:integer

Open the generated file and confirm that rails understood exactly what we wanted to do:

Now we must tell to both our models about their association.

In user.rb:

has_many :tasks

and in task.rb:

belongs_to :user
validates_presence_of :user

Rails understands in both cases which model is being associated by the value of the symbol.

Apply the migration to the database:

 bundle exec rake db:migrate

CanCan o’Clock. We will use CanCan in order to protect a user’s tasks from being accessed by another user.

Add the gem to your Gemfile:

gem 'cancan'

and bundle from the command line:

bundle install

Next, we are going to generate the file in which we are going to describe the “abilities” of a user:

rails generate cancan:ability

Open the generated file with your text editor and replace its contents with the following:

class Ability
  include CanCan::Ability

  def initialize(user)
    can :manage, Task, user_id: user.id
  end
end

Here we define that a user can execute any kind of action on a Task only if the task’s user_id attribute matches the current user’s id.

Time to authorize our actions. There is a shortcut for this but we will to do it the tedious way. Open the tasks controller and review the actions:

new: we don’t need to check anything here
create: the task’s user_id must be the one of the current user
edit: the task’s user_id must be the one of the current user
update: the task’s user_id must be the one of the current user
delete: the task’s user_id must be the one of the current user

We will apply the limitations above using the CanCan’s authorize! action. This method accepts two parameters, the action (symbol) that will be executed and the object on which it will be executed.

Change the controller’s actions as below:

class TasksController < ApplicationController
  def new
    @task = Task.new
    render :show_form
  end

  def create
    @task = Task.new(task_params)
    @task.user = current_user
    authorize! :create, @task
    save_task
  end

  def edit
    @task = Task.find(params[:id])
    authorize! :edit, @task
    render :show_form
  end

  def update
    @task = Task.find(params[:id])
    @task.assign_attributes(task_params)
    authorize! :update, @task
    save_task
  end

  def destroy
    @task = Task.find(params[:id])
    authorize! :destroy, @task
    @task.destroy
    @tasks = Task.accessible_by(current_ability)
  end

  private

  def save_task
    if @task.save
      @tasks = Task.accessible_by(current_ability)
      render :hide_form
    else
      render :show_form
    end
  end

  def task_params
    params.require(:task).permit(:title, :note, :completed)
  end
end

All actions in the authorized! calls could be replaced by the :manage symbol but I wanted to show you your options in case you had defined different abilities per action.

Also notice that in the create action we added the line

@task.user = current_user

This line will “fill” the user_id of each newly created task with current user’s id.

After all these changes, if a user tries to edit/update a task that belongs to another user, an AccessDenied error is going to be raised.

Note: the task_params method we defined in the previous part of the tutorial protects our task model from being assigned with a user_id from the parameters.

So, all cool? Nope.

In the home page we show all the tasks. ALL the tasks.

This isn’t right. So there is a last thing to do. Filter the tasks. Thankfully, CanCan provides us with a very nice way to do this. A model scope that uses the ability.rb permissions!

Open the pages_controller.rb and change it as below:

class PagesController < ApplicationController
  def home
    @tasks = Task.accessible_by(current_ability)
  end
end

And now we are fine. I suggest you register more users and start messing with the user_id attributes of your tasks to see the authorization being applied.

This is the last part of the tutorial that we actually implement stuff for the ToDo application.

There is going to be a final part in which I am going to help you deploy the application to OpenShift!

Creating a simple ToDo application with Ruby on Rails - Part 2

Tue, 17 Dec 2013 09:00:00 +0200

This is the second part of the tutorial for creating a simple ToDo application. In this part, we are going to:

implement the edit/delete actions accordingly
add some validations on the Task model

You can find the complete code of the tutorial here.

Let’s start. Our home page looks like this:

After creating a Task via the modal we have already implemented:

we now want to update the table of tasks in the Home page without having to redirect to it.

In order to be able to do this, we need to do two things first:

have the table being rendered inside an element with a known id so that we can change its contents with javascript
create a partial that will be displaying a list of tasks instead of having them explicitly rendered row by row each time

So, open the home.html.haml in your text editor and replace its contents with the following:

#task-list.container
  = render partial: 'tasks/task_list', locals: {tasks: @tasks}
#modal.modal.fade

Here, we added an id #task-list in the container div and we instructed the view to render inside it the partial with name task_list which is located under tasks aka: tasks/_task_list.html.haml

The partial doesn’t exist yet so create the file under app/views/tasks and add the following to it:

- if tasks.empty?
  %span.text-warning There are no tasks!
- else
  %table.table.table-hover.table-bordered
    %thead
      %tr
        %th Title
        %th Created at
        %th Completed
    %tbody
      - tasks.each do |task|
        %tr
          %td
            %strong= task.title
          %td.text-info= task.created_at
          %td.text-success= task.completed

As you can see, this is the code we already had in the home page but we changed the @tasks to tasks. From now on, this partial will be rendering the tasks passed to it using the locals: {task: @foo} way instead of always using the instance variable @tasks.

Refresh your browser, you should be having your home page displayed exactly as before. We are ready to change the way the create action behaves after completion.

Edit the _task_form.html.haml file and change the line where we defined the form so that it posts its content via javascript instead of using normal post requests:

.modal-header
  %h1 New Task
= simple_form_for task, class: 'clearfix', remote: true do |f|
  .modal-body
    = f.input :title
    = f.input :note
    = f.input :completed, as: :string, input_html: { class: 'datepicker' }
  .modal-footer
    = f.submit 'Save', class: 'btn btn-primary'

We did that just by adding the option remote: true. If you try to create a Task now, you will see that even though the task is being created, no redirection takes place and that’s because you can’t redirect via javascript using the normal rails redirect_to method.

So far, the create action just redirected to another page so there was no need for an additional view. Now that we want to do some additional stuff, we need to create a view. Create a create.js.erb file under app/views/task folder and add the following:

$('#task-list').html('<%= j(render partial: 'tasks/task_list', locals: {tasks: @tasks}) %>');
$('#modal').modal('hide');

Here, we replaced the contents of the container div with the task_list partial and we hide the modal which contained the task form.

Edit the tasks’ controller and change the create action to the following:

    def create
      @task = Task.create(task_params)
      @tasks = Task.all
    end

Check it out, after creating the Task, the modal closes and the task table is being updated correctly.

Deleting the tasks

We are going to add a link to each task row of the partial to provide the ability of deleting the task.

Change the contents of the task list partial to this:

- if tasks.empty?
  %span.text-warning There are no tasks!
- else
  %table.table.table-hover.table-bordered
    %thead
      %tr
        %th Title
        %th Created at
        %th Completed
        %th Actions
    %tbody
      - tasks.each do |task|
        %tr
          %td
            %strong= task.title
          %td.text-info= task.created_at
          %td.text-success= task.completed
          %td
            = link_to task_path(task), remote: true, method: :delete do
              %i.icon-remove

We added a new column (Actions) to the header of the table and one at each row of a task. The link is “marked” as remote so that we trigger ajax requests and the method is set to delete.

Why? Let’s see our routes, from the command line:

    rake routes

As you can see, the destroy method that we will create in order to delete a task is mapped under a delete method (second column).

Refresh your browser, your should be seeing something like this:

Pressing the x link nothing happens. At least on the browser because in the server’s console you’ll see the error:

AbstractController::ActionNotFound (The action 'destroy' could not be found for TasksController):

Let’s create the action. Add this to the tasks controller right after the create action:

def destroy
  @task = Task.find(params[:id])
  @task.destroy
  @tasks = Task.all
end

We now want to have the table updated upon each deletion and we’ll do this by re-rendering the task_list partial.

Create the view destroy.js.erb under app/views/tasks and add the following:

$('#task-list').html('<%= j(render partial: 'task_list', locals: {tasks: @tasks})%>');

Go back to the browser and check it.

The tasks are being deleted and the table is updated correctly but something doesn’t feel right, right? Right.

It’s too easy to delete a task by mistake. We will add a confirmation just to avoid this. The only thing we need to do is add the appropriate option to our link and rails will take care of it.

Change the link in the task_list partial to this:

= link_to task_path(task), remote: true, method: :delete, data: {confirm: "Are you sure you want to delete task #{task.title}?"} do
  %i.icon-remove

And there it is:

Editing the tasks

Once again, we will add a new link to the task list partial so that we can edit a task.

This time we won’t need another column on the table, we will add it to the existing “Actions” one. When the user clicks on it, a modal with the task form will be rendered. We can use the same partial we created for the new action but we will change some stuff.

Edit the _task_list.html.haml partial and change it to the following:

- if tasks.empty?
  %span.text-warning There are no tasks!
- else
  %table.table.table-hover.table-bordered
    %thead
      %tr
        %th Title
        %th Created at
        %th Completed
        %th Actions
    %tbody
      - tasks.each do |task|
        %tr
          %td
            %strong= task.title
          %td.text-info= task.created_at
          %td.text-success= task.completed
          %td
            = link_to task_path(task), remote: true, method: :delete, data: {confirm: "Are you sure you want to delete task #{task.title}?"} do
              %i.icon-remove
            = link_to edit_task_path(task), remote: true do
              %i.icon-edit

In the controller, the only thing we need to do in the edit action is to load the task. Edit the tasks_controller.rb and add:

def edit
  @task = Task.find(params[:id])
end

Our edit view will just render the task form and since now the @task instance variable is an existing one, the fields will already have the respective values.

Create the file edit.js.erb under app/views/tasks and add the following (don’t do it yet):

    m = $('#modal');
    m.html('<%= j(render partial: 'task_form', locals: {task: @task})%>');
    m.modal('show');
    $('#modal').integrateDatepicker();

But wait a minute… Isn’t this exactly the code the new.js.erb file has? Yes.

Instead of creating a duplicate, we will rename the new.js.erb to show_form.js.erb and we will instruct both the new & edit actions to use it instead of the defaults.

So rename the new.js.erb to show_form.js.erb and change the controller’s corresponding actions to the following:

    def new
      @task = Task.new
      render :show_form
    end

    def edit
      @task = Task.find(params[:id])
      render :show_form
    end

Nice. Now, try to edit a task from your browser:

Hmm, the title of the modal isn’t right. We are editing, we’re not creating.

To fix this, change the the _task_form.html.haml partial to the following:

.modal-header
  %h1
    = task.new_record? ? 'New Task' : "Editing task #{task.title}"
= simple_form_for task, class: 'clearfix', remote: true do |f|
  .modal-body
    = f.input :title
    = f.input :note
    = f.input :completed, as: :string, input_html: { class: 'datepicker' }
  .modal-footer
    = f.submit 'Save', class: 'btn btn-primary'

Here, we added a condition in the header of the modal so that when the task of the partial is a new record, New Task will be rendered otherwise, Edit task xxxx where xxxx is the title of the existing task.

Since the task is already existing, the form has automatically been configured to make a put(patch) request to the appropriate path and we don’t need to change something. We only need to implement the update action in the controller. So, edit the tasks_controller.rb and add the following just after the edit action:

def update
  @task = Task.find(params[:id])
  @task.update_attributes(task_params)
end

After the update, we want to hide the form and re-render the task list.

Yes, once again, this code will be exactly the one we have in create.js.erb so in order to avoid duplicates, we will rename the view to hide_form.js.erb and we will fix both actions (create & update) to render it instead of the defaults.

Rename the create.js.erb to hide_form.js.erb and change the controller’s actions as here:

def create
  @task = Task.create(task_params)
  @tasks = Task.all

  render :hide_form
end

def update
  @task = Task.find(params[:id])
  @task.update_attributes(task_params)
  @tasks = Task.all

  render :hide_form
end

That’s it. Everything seems to be working fine.

Nope. We have a little problem with the datepicker.

Create a task having it’s completed field filled in and then try to edit it. The field is broken:

The problem here is that the date format used by our application and the one that the datepicker expects are not the same.

In an upcoming part of this tutorial we are going to add Users in our application and we will have preferences about the locales, date formats etc but for the time being we can just use a fixed one for anyone.

I will write an article with more details on date and time handling in Rails in the future.

Edit environment.rb which is located under config and add the following at the end of the file:

Date::DATE_FORMATS[:default]="%d/%m/%Y"

Caution: restart the server in order to view the changes

Edit pages.js.coffee and change the way we initialized the datepicker:

$.fn.extend {
  integrateDatepicker: (selector)->
    selector = selector || '.datepicker'
    $(@).find(selector).datepicker({format: 'dd/mm/yyyy'})
}
$(document).ready () ->
  $('body').integrateDatepicker()

and finally, edit _task_form.html.haml and change the “completed” input to this:

= f.input :completed, as: :string, input_html: { class: 'datepicker', value: task.completed.present? ? localize(task.completed, format: '%d/%m/%Y') : nil }

Now we should be fine.

Validating the tasks

We are fine, we can create/edit/delete tasks. One important thing missing though is validation.

Currently, we can save a task with no title or with a completed date in the future. In order to prevent this kind of data in the application, we will add validations to the task model. It’s pretty easy in Rails.

Edit the task.rb model and change it to this:

class Task < ActiveRecord::Base
  validates_presence_of :title
  validate :future_completed_date

  private

  def future_completed_date
    if !completed.blank? && completed > Date.today
      self.errors.add(:completed, "can't be in the future")
    end
  end
end

Now we need to change the create/update actions so that when the task is not valid the form is re-rendered.

Change the actions of the controller as below (don’t do it yet):

def create
  @task = Task.new(task_params)
  if @task.save
    @tasks = Task.all
    render :hide_form
  else
    render :show_form
  end
end

def update
  @task = Task.find(params[:id])
  @task.assign_attributes(task_params)
  if @task.save
    @tasks = Task.all
    render :hide_form
  else
    render :show_form
  end
end

We don’t duplicate code. We will create a private method to do the saving of the instance variable @task regardless of the action calling it.

Change the actions to the following (this time do it):

def create
  @task = Task.new(task_params)
  save_task
end

def update
  @task = Task.find(params[:id])
  @task.assign_attributes(task_params)
  save_task
end

Then, after the private keyword of the controller (right before the task_params method), add the save_task method:

def save_task
  if @task.save
    @tasks = Task.all
    render :hide_form
  else
    render :show_form
  end
end

You should now see the following (in firefox) when trying to submit the form without the title:

Keep in mind that this is not the rails error message.

Simple form scanned the Task model and since it found that the title attribute is required, it automatically added the html5 attribute “required” in the input.

The post request doesn’t reach our server unless the browser you are using doesn’t support the required attribute. In that case, you should see a message can’t be blank which is the rails error message.

Let’s see what happens with an invalid date:

Cool. That’s what we wanted. Let’s make the error look at least red. In your pages.css.scss:

span.error {
  color: #ac1414;
  margin-left: 5px;
}

And that’s all!

Thank you for the feedback you’ve given on the previous posts, I’d be glad to have your comments and corrections for this one too.

In the next part of the tutorial we are going to use devise in order to have users logging in the application.

Creating a simple ToDo application with Ruby on Rails - Part 1

Sat, 07 Dec 2013 09:00:00 +0200

So, you learned how to say “Hello world” with Ruby on Rails and it’s time to move on since that app is not something you will use unless you are the only one survived on earth.

In this part of the tutorial you will get familiar with the following items.

saving objects (from now on we will use the rails term model for these)
the rails console
using haml for your views
using Twitter Bootstrap CSS to make your application look nice

You can find the complete code of the tutorial here.

Let’s start. Navigate to your workspaces directory from the command line and create the application:

rails new todo

Open the newly created directory todo with your favourite text editor (I use Sublime).

First, we will add the gems we will need. Open the Gemfile that is located at the root of the app’s directory and add the following lines:

gem 'haml'
gem 'bootstrap-sass', '~> 2.3.2.0'
gem 'simple_form'

and comment out the line:

gem ‘turbolinks’

adding a # at the beginning of the line (we won’t use this gem for now).

Also, open application.js which is located under app/assets/javascripts and remove the following line about turbolinks:

//= require turbolinks

If you “speak” Java, contents of the Gemfile are something like a classpath. You define which libraries you are going to use. Rails (bundler actually) is going to download and install these gems unless you already have them installed. We will do this now, from the command line:

cd todo
bundle install

Ok, let’s see if we are good so far. Start the server:

rails server

From your browser navigate to localhost:3000. Are we OK?

Yes, we are.

Open with your text editor the file database.yml which is located under the config directory:

config/database.yml

As you can see, the application has already been configured to use the SQLite database engine for persistence. This is ok for now but on an upcoming tutorial we are going to change this file in order to use MySQL.

Our application is going to have a single model, Task, with the following properties:

title: task’s title
note: task’s note (details etc)
completed: task’s completion date

Let’s create the model. From the command line:

rails generate model Task title:string note:text completed:date

We decided to use:

string for the title attribute (small data type)
text for the note attribute (longer pieces of textual data)
date for the completed attribute

From the command’s output, we are currently interested for the first two files:

db/migrate/xxxxxxxxxxxxxxxxx_create_tasks.rb This is the migration. Migrations are files which change your database’s schema upon execution. This specific file has the appropriate commands that will create the table for our tasks. View the contents to get an idea of the commands.
app/models/task.rb This is the model. Models are classes defining persistent objects. This specific file defines our task model. If you open the file you will see that is actually an empty subclass of the ActiveRecord::Base. We are going to add stuff there later on.

At this point we are going to get familiar with the rails console - your best friend while developing rails applications.

I suggest opening a new window/tab of the console application you’re using (I use iTerm2) so that you can have access to both the rails console and the shell prompt of our application’s root folder. I have a third one for the rails server command.

From the command line:

rails console

or the equivalent

rails c

Imagine the rails console as an application in which you can directly execute your ruby code having all your application’s files available (and some of them already loaded).
Play a little with it, for example:

Don’t be scared by the example, read here.

Now, let’s check our Task model. Just type Task:

Hmmm.. Refresh your browser (at http:://localhost:3000).

That’s right. We generated the model migration but we didn’t ask to apply it so that our database schema gets updated.

Let’s do it now. From the command prompt:

rake db:migrate RAILS_ENV=development

Note: you may skip the RAILS_ENV=development part since this is the default.

Great. The table has been created. If you refresh your browser you’ll see the default rails application page again.

But re-typing Task in the rails console you’ll see that you get the same error. This happens because your environment was loaded the moment you executed the command rails console and at that moment the table indeed didn’t exist. You may reload the environment at any point using the command:

reload!

And there you have it:

Since our Task class is empty, what are all these properties? Where did they come from?

Well, ActiveRecord scans our table and automatically maps the column names to our model’s attributes. Cool? Cool.

As you may notice, there are two attributes there that we didn’t ask for: created_at and updated_at There were added automatically by our model generation command and ActiveRecord automatically assigns values to these columns upon creation & updates of a model. Cool? Cool.

Let’s create our first task. Within the rails console:

Task.create(title:  'First task',  note:  'This task was created inside the rails console')

That’s it. You just saved your first task. Now will load it in order to edit/update it and then to delete it.

task =  Task.first

We loaded our task by selecting the first and only record of the Task model.
Let’s edit it:

task.title =  'First task - edited'
task.save

And let’s delete it:

task.destroy

Confirm that we have no tasks anymore.

Task.count

Ok, our model seems to be working fine. Time to start giving a face to our application.

The default views of a rails application are erb files that in the case of an html page you can imagine them as jsps but instead of Java you write ruby code. We already have such a file in our application, application.html.erb, open it in your text editor to view its contents (it’s located under app/views/layouts).

This file is a layout meaning that you may use it to display content that you wish to be present to a lot (if not to all) of your application’s pages. Each page you create will be embedded at line 11. You may add whatever you want before & after the yield directive.

As you can see, the file contains both static and dynamic content (enclosed in <%= %> tags) and looks very much like a jsp page. We are going to change this now by using haml. Why? Because our views will look clearer and more beautiful. It will be easier to read and update them.

The haml’s general idea is:

use indentation to express hierarchy skipping end tags etc
ruby commands start with -
ruby embedded content starts with =
html tags start with %
use # for setting id’s to your html elements
use . for setting css classes to your html elements

Now, rename the file to application.html.haml and replace the contents with the following:

!!!
%html
  %head
    %title Todo
    = stylesheet_link_tag    "application", media: "all"
    = javascript_include_tag "application"
    = csrf_meta_tags
  %body
    = yield

As you can see we also removed the turbolink related stuff.

IMPORTANT: Make sure to set your text editor to use 2 spaces instead of tabs!!!

Got it? Isn’t it simpler? You’ll see more haml code below that will help you understand it better but you may follow a simple tutorial later.

So, we will create our static pages controller now as we did at the previous tutorial. From the command line:

rails generate controller pages

Add a home action in the generated controller pages_controller.rb which is located under app/controllers:

class PagesController < ApplicationController
  def home
  end
end

Create the corresponding (empty) home page file home.html.haml under app/views/pages directory.

Remove the comments from the routes.rb file which is located under the config directory and add the following line in order to have our home page being rendered as the root path of the application:

root to:  'pages#home'

Refreshing your browser, you should now view an empty page (since our home.html.haml is empty). But if you view the source of the page from your browser, you’ll see the html of our layout.

Let’s edit our layout to add some cool “bootstrapish” stuff.

We will add a hero unit with our application’s name. If we didn’t use haml, we would add something like:

<div class="row-fluid">
  <div class="span10 offset1">
    <div class="hero-unit">
      <h1>ToDo<h1>
      <p>Welcome to the tutorial's ToDo application</p>
    </div>
  </div>
</div>

The equivalent haml:

.row-fluid
  .span10.offset1
    .hero-unit.text-center
      %h1
        ToDo
      %p
        Welcome to the tutorial's ToDo application

So our application.haml.html now looks like:

In order to have the Boostrap magic enabled:

rename the application.css file which is located under app/assets/stylesheets to application.css.scss
add the following line to the bottom of the file:

@import  'bootstrap';

in application.js which is located under app/assets/javascript add the following before the line //require_tree .
```
//= require bootstrap
```

Done. Refresh your browser and you should see this:

Time to add some task functionality.

In our home action of the pages controller we will keep all of our tasks in an instance variable named @tasks:

def home
  @tasks = Task.all
end

and now we have them available to our view. We are going to show a table with all the tasks and an appropriate message if none exists. Add the following to your home.html.haml:

.container
  - if @tasks.empty?
    %span.text-warning There are no tasks!
  - else
    %table.table.table-hover.table-bordered
      %thead
        %tr
          %th Title
          %th Created at
          %th Completed
      %tbody
        - @tasks.each do |task|
          %tr
            %td
              %strong= task.title
            %td.text-info= task.created_at
            %td.text-success= task.completed

If you refresh you browser now, you should be seeing this:

Add a task as we did before via the rails console just to see how it is being displayed:

We created our pages controller in order to handle the static pages of the application. The “static” word here doesn’t mean static content but actions that do not belong in the context of a resource. This controller is described as a non-resourceful controller.

On the other hand, now, we need to create a controller for managing our model(resource) Task. We want this controller to handle actions (view, create, new, update & destroy) on our model. This kind of controllers is described as resourceful.

Rails helps us route these actions all at once with the resources method in our routes.rb file. By default, this method will route all available actions (see here) but we can limit them to our needs as you’ll see below. Add this line to routes.rb:

resources :tasks

Now, from the command line execute this command to view the current routes:

rake routes

As you can see, a bunch of new routes have been created besides our root one. Notice the forth column of them. These are the actions that are mapped because of the addition of the resources method in routes.rb.

We actually want all of them except the index since we are going to display our task index in the home page instead of a separate one. It’s pretty clear what the others are supposed to do. So, let’s remove the index action.

Change the resources call to the following:

resources :tasks,  except:  [:index]

Gone.

Also notice that the actions are supposed to belong to a tasks controller(tasks#action). Guess what’s next:

rails generate controller tasks

We created the tasks controller (app/controllers/tasks_controller.rb).

Now it’s time to describe what we want to do. On the home page, we want to have the list of our tasks giving the ability to update/destroy each of them. We also want to give the ability to create new tasks. We want all these actions to be executed the ajax way and not with redirects to new pages etc…

We will start with the tasks#new action. In the tasks controller:

def new
  @task = Task.new
end

Here we initialize a Task and we keep it at the @task instance variable. In our home action, we were rendering a whole page but now we want just a modal with a form inside it to be shown. We will respond with javascript to accomplish this so the corresponding view of the new action won’t be a new.html.haml but a new.js.erb one. Create it under app/views/tasks.

Before continuing, we will add a modal window in the home.html.haml so that it is be available in the page for our scripts to access it. Add the following line to the end of you the file:

#modal.modal.fade

Edit the new.js.erb and add the following:

m = $('#modal');
m.html('Foo');
m.modal('show');

We will correct the code above later but for the time being, let’s see how does the modal appear in our page.

We need to add a button/link in the home page for the new action. We can add it to the layout actually so it is available on all pages (even though we are going to have only the home page in this tutorial). Edit the application.html.haml file in your text editor and add the following line in the hero unit:

= link_to 'New task', new_task_path,  class:  'btn btn-primary'

Now refresh your page in the browser:

Nice. Press the button.

Hmmm.. Error… Missing template…

What happened here since we have a view for the new action? In simple words, normal clicks of links are requesting the html format of our views. We have to request the js format and to do so change the link we added to the layout as below:

= link_to 'New task', new_task_path, class: 'btn btn-primary', remote: true

If you press the button now, you will see a modal with the word Foo as its content.

Before continuing, you should know what a partial is: a re-usable part of code (imagine it as a part of a view) which you may embed into other partials/pages/layouts etc. So, if we have a part of a view that we want to use to other places too, we usually create a partial.

In our case, we will create a partial that will be our form for creating a task. Then, in our new.js.erb we are going to replace the Foo string with this partial so the modal will display the Task form. Partial files start with an underscore so create a file named _task_form.html.haml under app/views/tasks directory and add the following contents:

.modal-header
  %h1 New Task
= simple_form_for task, class: 'clearfix' do |f|
  .modal-body
    = f.input :title
    = f.input :note
    = f.input :completed
  .modal-footer
    = f.submit 'Save', class: 'btn btn-primary'

Also, open the page.css.scss file which is under app/assets/stylesheets and add the following rule to fix the bottom margin of the form displayed in a modal:

.modal form {
  margin-bottom: 0;
}

Finally, let’s change the new.js.erb in order to render the partial in the modal:

m = $('#modal');
m.html('<%= j(render partial: 'task_form', locals: {task: @task}) %>');
m.modal('show');

And now we can press the button New task:

As you can see, simple_form picked to render the title as a simple input text, note as a textarea and, well, the completed date field as a combination of three selects. Let’s change the last one because it doesn’t look nice.

To achieve this we will tell simple_form to render it as a simple text field, then we will add a specific class to this input and using javascript we are going to create a nice calendar.

First change the line of the task form from:

= f.input :completed

= f.input :completed, as: :string, input_html: {class: 'datepicker'}

We will need a new gem, so add it to the end of your Gemfile:

gem  'bootstrap-datepicker-rails'

Edit application.css.scss which is under app/assets/stylesheets and add the following before line *= require_tree .

*=  require bootstrap-datepicker

Edit application.js and add the following before line *//= require_tree .*

//= require bootstrap-datepicker

Now, we have to create a javascript function that will apply the datepicker behaviour of the gem to the appropriate inputs. We’ll do this in a fancy way. Open the pages.js.coffee file and add the following:

$.fn.extend {
  integrateDatepicker: (selector)->
    selector = selector || '.datepicker'
    $(@).find(selector).datepicker()
}
$(document).ready () ->
  $('body').integrateDatepicker()

What did we do here?

We extended the jQuery prototype in order to add a function (integrateDatepicker) that will search the children of the current element with a given selector and apply the datepicker behaviour on them. We also call this function when the document is ready on the body in order to avoid the explicit call on freshly rendered pages. Now, since our form was not present during the ready callback of the document, the behaviour hasn’t been added to our completed input. Thus, we must add the following line at the end of the new.js.erb:

$('#modal').integrateDatepicker();

Refresh and there it is:

Since this form was created for the @task instance variable of the controller and since that @task does not exist but it is a new one, its action (the form’s action) has automatically been resolved to the create action of the controller. Don’t believe me? Press the save button:

Open the tasks controller and add the following:

def create
  @task = Task.create(task_params)
  redirect_to root_path
end
private
def task_params
  params.require(:task).permit(:title, :note, :completed)
end

Here we defined the create action that creates the task and then redirects to the root path aka our home page. We also created the task_params private method so that we filter the params of the request in case someone tries to pass parameters that we don’t expect. We only allow values for the title, note and completed attributes of our model. There will be cases that your model will have attributes you don’t want to be set by the user and this is the way to control them.

It’s about time to end this part of the tutorial here. Very soon I will write the next part in which we will:

refactor the create action so that it doesn’t redirect to the root path but it closes the modal and re-renders the table of the tasks
add validation to our model so that empty titles are not allowed for the tasks showing the appropriate messages to the user
implement the edit/update and delete functionality

Saying hello world with Ruby on Rails

Sun, 24 Nov 2013 09:00:00 +0200

If you are new to Ruby I suggest you first read my previous post which in short explains why you should not give up on learning Ruby.

Starting with this post, I’ll be writing some tutorials covering the first steps you could follow in order to get familiar with both the language (Ruby) and the framework (Ruby on Rails). My posts will be based on my own experience so I hope that anyone new here can understand them easily.

First things first, install ruby and rails on your machine. There are plenty of installation tutorials out there from which you can pick the one matching your machine’s operating system. I followed this one in order to install Ruby 2.0 & Rails 4 on OS X Mavericks.

Development environment

You will need:

Text editor: I prefer using Sublime Text which is an excellent text editor with many plugins. There are a lot of other options but this is something you don’t have to decide now. Give them a try and you’ll find the one you like the most. Update 2016: I now prefer using GitHub’s Atom editor.
Console: I use iTerm2 which is a powerful replacement for OS X’s Terminal application.
Browser: Your favorite one. I use firefox (with its firebug & httpfox plugins) for development. Update 2016: I now prefer using Google Chrome with its great development tools

Let’s begin

Create a folder in which you are going to put the tutorial’s code and navigate to it from the command line:

mkdir tutorials
cd tutorials

Now let’s create our new application with the name hello_world:

rails new hello_world

The command above will create a folder with the name hello_world. Inside this folder you will find various files and subfolders which actually constitute a Rails application.

In this post I will only explain the files/folders to which we are going to make changes on (a thorough explanation will be provided in an upcoming one).

You already have a web server installed in your system that comes bundled with Ruby and its name is WEBRick. Let’s start it to see what we have created by the last command.

From the command line, navigate to the created folder:

cd hello_world

Start WEBrick with the following command:

rails server

You can also use s instead of server in the previous command which is just an alias and will do the exact same thing:

rails s

You should see something similar to this:

By default, the server listens to port 3000 but you may change to the one you desire by using the -p option:

rails server -p 8080

So, the web server started and our application is successfully deployed. Open your browser and visit the following address:

http://localhost:8080

Tadaaa… You should see something like this:

Before we continue, I will try to explain in a simple way how a rails application works.

Every request to the application is served by methods (called actions) which are defined in special files called controllers.
These actions execute what has to be executed and define what the response will be (for example an html page, a JSON object etc). The responses (e.x. a page) are defined in special files called views but there are cases (such as when responding with a JSON object) that an action’s view is not required to be defined.
The rules (routes) that define which action of which controller will serve each request are placed in a special file called routes.rb

These are enough for the time being. Let’s create our first controller named pages. From the command line:

rails generate controller pages

You will see a bunch of lines coming out from this command but focus on the first one:

Yes, pages_controller.rb is our controller. In this file we are going to create our action that will serve the “root” (home) request to our application.

So, open this file with your text editor. Its location is: app/controllers/pages_controller.rb

As you can see, the controller is empty. We will add an action to it, let’s call it home:

def home
end

As you can see, our action does nothing since all we want to do is show a static page with text Hello world. Let’s add something though just to see it being executed from the server’s log:

class PagesController < ApplicationController
  def home
    puts "Honey, I'm home!"
  end
end

If you speak Java, puts is similar to Java’s System.out.println();

If not told otherwise (which is our case), the home action will try to respond with a view named home too. So let’s create this view in the appropriate folder which is: app/views/pages/

Create a file in this folder with the name home.html.erb

The erb extension implies that this file will be processed by Rails in order to embed any dynamic content.

Edit this file and add:

<h1>Hello world!</h1>

What’s left? Yes, we must route the root requests to our controller’s action.

Open with your text editor the routes.rb file which is located here: config/routes.rb

You will find a bunch of help comments (text starting with #) there which you can delete.

Now, add the following line which tells that the root path / will be served by our controller’s home action:

root to: 'pages#home'

If you refresh your browser, you will see the amazing and super complex page we just created:

We could stop here but it would be nice if we had the Hello world! text being passed to the view from the controller’s action, yes? Yes.

Open the pages controller again and change the home action:

def home
  @greeting = "Home action says: Hello world!"
end

Here, we declared an instance variable to our controller with name greeting and value Home action says: Hello world!.

This variable is available for use from our corresponding view so open the home.html.erb view and change its contents to the following:

<h1><%= @greeting %></h1>

Refresh the page in your browser and there it is:

There, you just created your first Ruby on Rails application!

What’s next? How about creating a simple ToDo application?

iridakos - tutorials

Mongoid - Inheritance: change embedded document's type via nested attributes

Use case

Item

Book

BoardGame

Person

The problems

Failing attribute assignments

Document attribute leftovers

Wrong validators

Investigation

Desired behavior

Solution

Explanation

How to alias and navigate to directories with autocomplete in Linux

goto

Installation

Usage

Change to an aliased directory

Example

Register an alias

Example

Notes

Unregister an alias

Example

Notes

List aliases

Expand an alias

Example

Cleanup

Help

Version

Extras

Push before changing directories

Revert to a pushed directory

Notes

Behind the scenes

Dockerizing a Rails application

Prerequisites

Install docker

Important note

Clone the rails chat tutorial from GitHub

Configure the application to use PostgreSQL for the production environment

Building the docker image

Create the Dockerfile

Define the parent image

Copying the application code

Install application dependencies

Asset compilation

Start the server

Compacting the Dockerfile

Running the application in the Docker environment

Creating the PostgreSQL container

Creating the Redis container

Creating the application’s container

What’s next

Thanks

Creating a chat application from scratch using Rails and WebSockets

Introduction

What are WebSockets

Why WebSockets

Building the application

Setting up the environment

Install ruby

Install rails

Create the rails application

Users and devise

Rooms and messages

Adding authentication

Add bootstrap

Add simple_form

Devise views with bootstrap and simple form

Cleanup unused components

Adding a navigation bar

Room management

Room index

Room new/edit

Room page

Creating room messages