In my opinion, the top 3 things that Russian IT specialists gave this world are:

• Tetris
• Nginx
• Telegram

Telegram is great, it has already become the main source of news for many people these days. Especially, its channels which are absent in Whatsapp or Viber by the way. But the constant reading of many channels through the day could be tiresome. What if someone reads the news for you? Or something instead of someone. I just have Yandex Station Light with voice helper Alice inside. It is like Alexa or Cortana in a box. So what if!

Alice allows writing user programs that can be reviewed, published, and then activated by voice interface. Yandex calls these programs “skills” or “dialogs”. So the goal here is to dive into writing new dialog, figure out how to work with Telegram API, and then connect these two things together.

Dialogs for Alice

Yandex has quite comprehensive docs for dialogs. Creating dialog is a simple process that you can do even from a console. The idea behind the dialog is that the station receives some phrase from the user, packs it into a request, and sends it to the dialogs backend. When the response is received, station speaks back to the user. When the next phrase is received, the state may be changed already, so this is the way for the dialog to evolve.

The main things you have to consider for new dialog:

1. Unique name
2. Activation phrases: how to start working with dialog
3. Intents: different ways people will interact with the dialog
4. Hosting: cloud function or webhook URL
5. Access type: public or private

While 1, 2, and 5 seem to be clear, lets get closer to 3 and 4.

Intent

When you design the way communication is going to happen you have some scenarios in mind. At every step of any scenario Alice should know what exactly you want from her. This means that your phrase should contain meaningful information in terms of your specific dialog. Think of it like communication protocol you should follow. The intent is the way to extract this information related to some protocol. For example, “what is the weather for tomorrow in Helsinki” and “order me a taxi for 10 pm” are two phrases with different intents.

Yandex uses NLP (Natural Language Processing) to do the main job, but we need to help it in parsing phrases. Even though the difference between the two intents above looks quite obvious, we need to describe valuable pieces of information there. So the first dialog will start looking like calling function weather with arguments when=”tomorrow” and where=”Helsinki”. The second one will look like calling function orderTaxi with argument where=<my_current_location> (default value) and when=”10 pm”.

Intents and their grammar can be described via the web interface. This process is not complicated: there are only a few terms you need to get familiar with. But please keep in mind that there are only 4 types for named entities right now: names, locations, dates, and numbers. Yandex provides type description for them and can effectively recognise these type of values from your voice input. When your intents are fully described, all you need to do is to describe corresponding structures within your backend and start working with data coming from dialog.

Hosting

Yandex Cloud Functions (or just YCF) is a technology from a serverless family. It is one of the options to host the backend for your dialog. Processing time for the function that serves Yandex dialog is not charged by the company. This is the good part. The bad part is that “serverless” type of processing has some limitations. I will get back to these limitations later.

Another option is using a webhook URL, which means that you can host your dialog on any public server, grab the URL of your deployed backend, and put it into dialog settings. It is quite universal but requires more effort to deploy your code. This is the option I had to choose for now.

Telegram API

I had decided to try using Go for working with Telegram (or just Tg). I wanted to practice this language and it was the best option for me in the list of languages compatible with YCF. I was lucky to find a full-native go implementation of Tg API. Unfortunately, the last version (1.0.0) was released almost a year ago. But I was able to find all the functionality that I needed in this package.

When you want to work with Tg API, you have to create a new application first. You will get the application id, application secret hash, and server host address. All these parameters are required to initialise the Tg client. Along with this initialisation Tg client also inits the storage for session data and public keys. This storage should be local and this is where the problem comes from.

YCF does not allow you to store any data locally because of its serverless nature. This means you can not use this Tg client and YCF in pair. This is the reason I had to use local/public server to host backend for dialog that uses Tg API and switch my dialog to using webhook URL.

Except procedures of logging in and confirming phone number with code I need to work with some other Tg entities to make the whole logic work. The most essential of these entities are:

• Chat
  I am mostly interested in channels, but channel is the type of chat in Tg world. In order to get chats I use getAllChats endpoint.
• Message
  Every chat contains messages. A list of messages is available through history, which can be fetched using getHistory endpoint.

So my backend is just some processing around Tg chats and messages: find chat, get number of messages from this chat etc.

Other obstacles

• Tg API has the option InitWarnChannel for client config.
  Switch this one to “false” and you won’t see different client warnings in the process output
• When a connection with Tg server is created the first time, the client starts “keepalive” pinging and reading responses from the server. When the received message is not “nil”, “context.Canceled”, or “io.EOF” we get panic which is not I expect to happen.
  This is a bug that was not fixed at the moment and I don’t know if it is fixed now. Replace checking error with something else within startReadingResponses function to avoid this behaviour.

Some results

At least, this thing works. It was deployed to Amazon Lightsail server as a Golang API in a Docker container using some Ansible scripts. By the way, I found it quite useful to manage some of my pet project with one common Ansible repository.

After all I have managed this to work along with self-hosted backend. Managing several Tg sessions remains tricky part for me: since I only have one Yandex/Alice account, I can easily test with one session only. Of course, public dialog has to manage many sessions. Thus, publishing of this dialog does not seem to happen soon. Maybe I should leave it private and be a sole user here. Haha!

At the same time some services can give you a lot of data without account specifying. When “News and Trends” topic arises the very first thing that comes to my mind is Twitter. I can realise that dozens of tweets contain media and so we need rather to see content than to read it. But text is still important for many people who would like to receive news. So againg: what if!

Plans

At a first glance this new dialog should provide these main options:

• read home timeline (most recent Tweets, Retweets, and replies posted by the authenticated user and the accounts they follow)
• read specific user timeline (published by a specific Twitter account)
• search by tag, keyword or whatever

If you have any ideas with respect to these plans, please let me know in replies. Thank you for reading.

Hello. In this post I would like to cover one more topic related to my previous post theme. It is all about feature tests for web applications with reach frontend.

Sometimes we have to deal with requests to third-party services. If request goes from backend we can stub it, we can use VCR or whatever. But things differ when this request goes from frontend. We assume, that our backend is written on Rails. Request fired on frontend side does not go though Ruby part of application. But we do not want to have real requests in test environment. What to do then?

We need to emulate target server for requests coming from frontend. Literally, this means that we need to provide second test server in a way. Luckily, this can be done with the following snippet of code:

Here we describe Sinatra application with just one POST route. Don’t forget to provide some headers in order to bypass CORS protection. Additional gem is also used for this purpose. Finally, ‘discoball’ gem is used here to spin separate test server. But you can do this with simple threads if you would like to.

One more thing you have to do is to replace your 3rd-party server with our new Sinatra server for frontend. This was the reason to save second server url in application config. So now you can write in you html layout (slim template in my case):

javascript:
DEMO_API_URL = “#{Rails.configuration.x.demo_app.api_url}”

and then in javascript simply use:

window.DEMO_API_URL

That’s it. Happy testing!

If you have web application with reach frontend, you are better to have some feature tests, right? In Ruby world we use to have Capybara/Rspec and some JS driver for these purposes. But it often happens that your frontend logic is complicated enough to have random failures in tests. And you see messages, for example, like this:

Failure/Error: expect(page).to have_selector(‘[data-testid=homepage]’) expected to find css “[data-testid=homepage]” but there were no matches

or this:

Capybara::Ambiguous: Ambiguous match, found 2 elements matching visible css “[data-testid=button]” with text “Create group”

or better this:

Failure/Error: expect(page).to have_content(‘Captcha Verification failed. Please try to sign up again.’)

Captcha verification can be failed successfully on next run. You just need to rerun your tests. But time after time this becomes boring. Yes, it is better to get rid of this cases in your tests, but if you just need to save some time at current stage, here is the tool: rspec-retry. Simple setup in your test helper may look like this:

This setup will rerun your failed feature js tests automatically. Env variable is used here in order not to retry tests on your local machine unless you really want it. So that’s it: CI is going to rerun your tests instead of you from that moment. And of course, once you are done with random failures, this little snippet can be easily removed. Happy testing!

Lately I’ve been working with the product team. I mean the team working on its own product, not something like outsource and product owner or product manager on the other side. I was told before that this should be “completely different experience”. Well, I wouldn’t say so, for me it’s not like “completely” different. But many things differ in interesting way. And today I wanted to share some of my thoughts.

Planning and estimation.

Really, team does not care too much about velocity. Task tracker (Pivotal in this specific case) is doing all the job itself. So it knows exactly what tasks can fit into sprint. This helps team to focus on important goals for current iteration and not interrupt it. As usual, nothing special.

The difference is that team rarely have to spend time thinking about performance decrease and its reasons — there are too many causes, it really makes unreasonable this spending of time. So the only thing team needs to do (when all the tasks are in place) is to think about priorities.

Stories and epics.

Fast growing product has to involve more and more new functionality. Which means big features. Which, in its turn, means that you are going to face epics quite often. Just to confirm: epic here means huge story (can not be estimated using given limit of story points) that you can split into several small stories (can be estimated using given limit of story points) but can not deliver until every story is fully finished.

One important thing related to epics is accepting it in terms of QA (quality assurance). Let’s suppose you have staging environment in your applications (several apps within one big project, for example), QA specialist and somebody who sets a task (just like product owner) are two different persons, epic story includes tasks for design, for different applications etc. Well, things get complicated. Looks like you need to develop a special flow for accepting tasks within epic and the epic itself. Maybe some items in the list above are not so relevant for you, but accepting may become a problem if you don’t keep things in mind.

Support and bugs.

This is quite obvious, but still: if you want to have loyal customers you need both QA and customer support. In every live product bugs are coming constantly into backlog and, on the other hand, you are trying to check features before deployment. But when you are in product team, you have to prioritise bugs.

This becomes your and your peers’ responsibility, not someone’s else. So you can face the situation when something needs to be fixed “right now” (or even “yesterday”) and other thing may not be fixed at all (I mean ever, really). Just because at any given time you understand that there is something more important. Sound weird on the first glance, but so it is.

Keep hacking and stay true!

Working with product team: inside view was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you do any type of software, logs are very important part of ecosystem. They help to understand what happens at any given time. They are quite irreplaceable when you need to find root cause of a bug. This works for web applications as well. But when you have one app with hundreds and thousands of users, you have huge amount of logs. And it is a good practice to manage it somehow.

There are many services for log management nowadays. And they allow to use different types of integration with your application. Usually, the most convenient way for developer is language specific library. Because who wants to use API directly, right?)

Let’s say we are working on Rails application. We have number of solutions for Ruby integration. But they often have some inconveniences. For example, Logentries, being a very popular service, offers Ruby gem, which just replaces standard logger. This means that with this integration all the logs will go to the service and you won’t have text backups on your server. This sounds inconvinient for me, almost inappropriate. Though I have to say that for Heroku hosting option I would consider it quite useful.

Another example is Coralogix that I’ve tried recently. They suggest different approach. There is a feature in ActiveSupport that allows message broadcasting to many loggers. So your custom logger can just be added to the system in order to start receiving messages. The problem is that these messages will have pure text. For example, if you you are using tagged logging in Rails app, tags are not going to be included in text that you send to Coralogix. So you’ll have two version of logs: rich text on your server and plain text on Coralogix server. A little bit less inconvenient for me, but still.

Ok, so how do we do integration at the end? When you have several different services working on different platforms solution comes by itself. Usually, we have our servers running on Linux-based machines, most of them have rsyslog (or syslog-ng) service included. Here is the quote from official site:

It offers high-performance, great security features and a modular design. While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to accept inputs from a wide variety of sources, transform them, and output to the results to diverse destinations.

RSYSLOG can deliver over one million messages per second to local destinations when limited processing is applied. Even with remote destinations and more elaborate processing the performance is usually considered “stunning”.

What you need to do with this approach is to setup config for rsyslog. It includes several options, but the most important parts are:

1. template that describes message format (goes within “template” section)
2. number of filters that allows you, for example, to pick only one service’ messages from syslog (goes within “filters” section)
3. destination that describes where your templates are going to be sent to (goes within “destination” section)

First and third are usually specific to the service you use when second usually describes your personal needs to filter data. There is configuration section as well, but default settings are often fine there.

I have to say that when you need smart filtering, for example, config setup may become very tricky. But in many cases you just create new simple config, put it with other configs, restart rsyslog service. And that’s it! You may want to include this into some deployment configuration that you have for your servers (ansible role, for example). But general steps are very simple.

Rsyslog and log management was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Semaphore 2.0 was officially released recently. It brings “customizable CI/CD pipelines, autoscaling and more” to us. Let’s try to figure out, how can we switch to brand new Semaphore and do we need it at all.

Note: this is a quick view, many new sem commands, dashboard setup and other things were not covered at all.

According to official guides, we need first of all to get new CLI tool called “sem”. It will help to connect to Semaphore endpoint and initialise new project. New project means .semaphore/semaphore.yml file only because this is all you need to start. If we open new Semaphore’s home page in browser, there are “Dashboards” and “Projects” sections in the sidebar. Not sure if I need any dashboards so let’s go to projects. We can see pushed branches here and commit statuses. How do we get tests to pass?

Well, instead of project setup in browser you can describe everything with YAML configuration now. It is supposed for you to have blocks of tasks that have jobs inside. Blocks/tasks go one by one and jobs (within tasks) can run in parallel. In general, you need to make jobs based on your previous Semaphore project settings. Let’s try.

“In fact, this is the simplest version of Semaphore ever.” — Semaphore team.

When you start new project configuring, you’ll quickly figure out, that everything that were in settings before now needs to be described explicitly: not just you have to set Ruby version or DB type, you need to start DB server, do checkout, you even need to install imagemagick and some other packages yourself (in case if it’s required by setup, of course). Env vars can not be encrypted anymore, but more important, that you have to set the same vars for every block where you use them — they just can not be moved to upper block. This goes to duplication that is especially disturbing when you have to have many env vars.

Being able to configure everything with YAML file should bring a wide range of possibilities. At least, in theory. In practice, I expect CI to duplicate my regular testing flow. And here is the question: do you stop DB server between test run? Maybe you constantly switch Ruby versions? I don’t. So including these “obvious” steps seems weird to me.

Being user of this service (I mean Semaphore here) and having some projects working already, I would like to have some tool for config generation. Or maybe some magic button to dump config from existing setup. But this is not what we have. In this case migration guide should be really comprehensive. But this is not what we have either.

It was quite easy for me to describe a block for application setup. Just several iterations. But next I had to run tests and code quality check. I got stuck with the fact that gems were not found in next block. Debugging a job with ssh did not help since my folder was not there. I still don’t know why it happens. But I know, that Semaphore creates it’s own .env file there and my setup with dotenv doesn’t work anymore (script does not copy file if it is already in place). As for the gems, support guy suggested to include “bundle install” string as one of block commands. Well, good point of course. But why do I need these tricks with bundle cache then?

And why to I need to do db setup, redis start and other things if it was done in previous block? Probably, this is not how it works and I don’t need a “setup” block at all: everything can go into prologue step of main block. It was not supposed from the docs, but ok. Can I run my tests now? Not yet, let’s look into some brand new issues with rmagick first. Sorry guys, but I had to stop at some point.

As a conclusion, here is what you can get with new Semaphore:

1. You have “customizable CI/CD pipelines, autoscaling and more”
2. You have “flexibility” and wide range of options for setup
3. You are forced to duplicate config (env vars, prologue etc)
4. You are forced to use prologue if you don’t want to duplicate commands too
5. You can not encrypt env vars anymore (at least, easily)
6. You need to do things that you do not repeat in usual flow (start DB server, change ruby version etc)
7. You don’t have well-written migration guide
8. You don’t have config generator
9. You can not use dotenv as before
10. You have to fight with new issues. Who would doubt that? ;)

Do not despair! Keep hacking!

What did we get from Semaphore 2.0 was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.

TLDR If you use GoDaddy for DNS management, do it with caution. If possible, do not use it at all.

The number of sites switching from HTTP to HTTPS as the default connection protocol has grown drastically in the last few years. It’s sufficient to have a look at the Let’s Encrypt progress report for June 2016 to get a sense of how fast the web is moving to HTTPS.

The increasing adoption of HTTPS as the default connection protocol for websites has introduced a few new challenges to developers and system administrators. One of them is the need to properly handle redirects to the canonical address of the website to avoid the risk of content duplication. More importantly, configuring the appropriate redirect is necessary to offer a consistent and secure browsing experience.

This is what DNSimple blog tells us. They say that we often need to support up to 4 main redirects if we want to forse ssl with canonical domain. And one common way to do this is domain forwarding.

Also, when we use Heroku and need a custom domain, it tells us to use DNS hosting with ALIAS/ANAME records support. If you don’t have any (like you are on GoDaddy, for example), here is the basic manual which describes how to do similar thing using domain forwarding. The point is the following: is does not work correctly. It looks like it was fine before. But now they switched the way of forwarding domains via a URL Shortener. So it forwards main domain only. But if you forward path (and query params) it can bring you to the page shown on the image above.

Guys from support can say that this is how domain forwarding should really work. In fact, everything is perfectly described is this post, where it’s confirmed that service was changed to “increase stability and performance as well as address other internal issues”. And that’s why “the path match no longer works for any forwarding”. So please, be careful.

In our case we had to implement some workaround solution and add subdomain to “long” links so it does not require redirection. But this is a temporary solution and general plan is to choose new DNS hosting with ALIAS records support.

Happy hacking!

Domain forwarding in Rails applications was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Once I was searching for a bug in code and found this:

The problem with this code is in the line 12: when you use find_or_initialize_by method block is called only for initialized instance. So existing timelapse instance is not passed to build_timelapse and not updated in our case.

Ok, let’s move out this block from find_or_initialize_by call. We should have something like this:

The most obvious way is to move build_timelapse method into Timelapse model — in this case we’d just call t.build_timelapse or t.build_from_schedule i.e. renamed version. But here I started to torture myself: we need to do this only for scheduled timelapses, so why do we add this method to every instance? Let’s find another way even if it is going to be tricky![1]

Ok, if we find a way to call this method in context of timelapse we won’t even need this nasty local variable. Just this:

But how do we do this? The very famous way is instance_eval, but this is not exactly what we need here since we are going to set values in our method (probably replacing timelapse with self) which means passing parameters and instance_eval is not suitable for this task [2]. instance_exec to the rescue?

Not so fast! Documentation says that this method executes given block within the context of the receiver and self is set to receiver object. And it handles arguments for the block too. Great, but we still need to pass a block somehow. Yes, we can turn method into block of code: something like &method(:build_timelapse).to_proc. But this won’t work. In this case method will not start behave like proc. This method is bounded to instance of ScheduleTimelapse, not Timelapse, and it won’t accept call to self properly. Even if we do unbound it can only be bounded back to the instance of ScheduleTimelapse class.

This was my main upset — there was no way to do exactly what I’d planned to do and I had to convert this method to lambda accepting one argument schedule. Finally, I was able to rewrite the code this way:

[1] We can do build_timelapse(timelapse, schedule) and just make this method return timelapse as a result. This should work, but I don’t like this way since it is going to prevent method from being kind of “pure function”.

[2] There is a note about arguments in the docs, but difference is not indicated explicitly. You can find additional info in article about FactoryGirl gem (which is FactoryBot now).

Happy hacking! Thanks.

instance_exec to the rescue? not so fast! was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recently, we had a small task: API with couple of endpoints making authentication based on asymmetric encryption and JWT. Application was required to perform relatively big number of requests per second. That’s why we decided to try Elixir. Here are some notes based on our attempts to create such an app and measure performance.

In our case execution cycle within endpoint includes: DB operations (quite simple, but still), Redis operations (save/read challenge message) and encryption/verification logic (verify encrypted challenge message, create JWT). So measuring relative performance was kind of goal. What did we learn so far?

1. Making such a prototype is not a big deal. Having tools like ex_crypto (to verify signed messages), exredis (to work with Redis), rsa_ex (to generate keypairs) and, of course, guardian (swiss army knife for Elixir/Phoenix projects, in our case — JWT source) made the process very smooth.
2. Without OOP we can still use modules to encapsulate code responsibilities and provide some analogue of SRP in our application. Having pattern matching mechanic, pipelines, fallbacks controller mechanics and views separated from templates is not an obstacle but rather good facility to create json api following described protocol.
3. Generate a million of records on Heroku is not a big deal. But with keypairs generation this task becomes time consuming. I was unable to to do this directly from console because of Heroku timeouts. Having this task run with detached console, I was unable to get 1_000_000 records as well — it stopped after ~300_000 for some reasons.
4. You can easily create mix tasks and run it on Heroku. In my case I had two tasks: one to get public key stored in random record and one to use it in wrk benchmarking. Combining tasks, however, was not an easy goal. I spent some time struggling with redundant slashes in encoded string and had to give up (because of time limitations).
5. We can have 300–500 requests per second ever on free dyno. And free plan for Redis add-on (with 10 connections) seems like enough too. However, I’ve got plenty of responses with 500 code while benchmarking. I’m going to play with dyno types to check if it can help.
6. We used benchee for internal benchmarking. Just to compare execution time for different parts of our endpoints. Running benchmark from mix task, I got results showed on the image below. It just proved that for the first endpoint working with DB is the most time consuming operation. I’m going to try the same thing later to compare DB operations time with key verification operations time.
7. You can play with all these things not only on Heroku. Well, in theory. There is a new service called Gigalixir (you can watch the demo from this screencast). It is supposed to behave very the same as Heroku, but for Elixir apps. It has account settings, manageable apps, PG tool etc — everything is managed though the cli tool (which is installed via pip). Finally, I was able to run application (response from support guys was very quick). Unfortunately, I was unable to run benchmarks. Service is quite raw at the moment. There is no web dashboard, no add-ons available from the console (we need Redis for this app) and I just couldn’t run seeds.

[I suppose that I will be able to solve problems on Gigalixir with help of support team. But now there is no luck after couple of days spent.]

Thanks for reading and happy hacking!

Elixir. Some performance remarks was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Let’s suppose you have a huge chunk of data in file (csv, for example): tens of thousands of rows. And you need to import it in your DB. Let’s also suppose that your application lives on Heroku. How would you do that? Let’s go from more obvious to less obvious.

If there are no sensitive data, you may want to attach data files to your repo and then import it from rake task or even from database seeds, for example.

1. You go through the rows and create DB records one by one.

Main problem here is time. Inserting of so many records will take a lot of time. And one-off dyno that is going to do this for you (when you run related task) has one hour timeout. So you may fail here (if you have enough data to import) very easy.

2. You go through the rows, collect data and then create DB record all at once.

Ok, let’s try to combine all inserts in one. This still can take a lot of time, but let’s think that timeout is not a problem anymore. Next problem you can face is lack of memory. When you collect data for your request, you may spend a huge amount of memory that is going to be freed only when task is finished.

What if your dyno does not have enough memory? In this case you may not see any record being created at all. You will only see R14 errors in log. That’s why you may need next approach.

3. You go through the rows, collect data in batches and create DB records in batches.

Couple of options that were considered by me: plain sql and activerecord-import. SQL itself is always an option. But the gem could be used even on previous step. Here it can be used as well: just import data with little chunks(*).

4. You move this logic into scheduled task.

There was an opinion, that since web dynos are usually loaded more heavily moving the whole process to worker dyno(**) may help. This does not seem to be true. As it was mentioned already, when you need to perform some task or work in console Heroku allocates separate instance for you. Properties of this instance (dyno) depend on your current setup, but this dyno is not loaded by something else. So you don’t have to “choose” between web and worker types.

* In my case reducing chuck size to 1000 was enough. I guess it’s reasonable to consider something between 100 and 1000.

** These are two separate instances on Heroku usually.

Massive data import options was originally published in Flatstack Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.