A few weeks ago I decided to try out something new. Telegram Bot caught my attention so I started playing around with it. The first thing I had to do was to set up a local development environment. The process was a bit tricky because a bot server had to be accessible from the Internet and Telegram required only secured HTTPS connections.

This article is a complete guide. Once you’ve followed all the steps you have a fully working workspace for developing a chatbot.

Create bot

Open @botfather bot and follow instructions on how to create a new bot. As a result, you should get a token. In a moment, We’ll use it to communicate with Telegram API.

Setup Ngrok tunnel

Telegram needs HTTPS endpoint available on the Internet. Whenever a user does something in a bot it sends a request containing user action info to the endpoint. Ngrok is a good solution because it is easy to set up and it provides a secure endpoint for free. One disadvantage is that when you stop Ngropk tunnel and start again the new URL is generated. So you have to set webhook again. A paid Ngrok version offers static URL.

Let’s create a docker network:

docker network create bot_ngrok

and start Ngrok container and leave it working:

docker run --rm -p 4040:4040 --net bot_ngrok \
    --name telegram_bot_ngrok wernight/ngrok \ 
    ngrok http telegram_bot:80

Find out your public address:

curl $(docker port telegram_bot_ngrok 4040)/api/tunnels

The command should print a JSON data. Find public_url field there. The content could be like this https://f2183d0.ngrok.io.

Set up telegram webhook

Now, we have a tunnel. Let’s instruct Telegram to send updates to that Ngrok URL. For that, we need to do a bit of coding, create a directory and install formapro/telegram-bot library:

docker run --rm --interactive --tty --volume $PWD:/$PWD -w $PWD --user $(id -u):$(id -g) composer:1.8 composer req formapro/telegram-bot

Create a set_webhook.php PHP script:

<?php
// set_webhook.php

include_once __DIR__.'/vendor/autoload.php';

use Formapro\TelegramBot\Bot;
use Formapro\TelegramBot\SetWebhook;
use function GuzzleHttp\Psr7\str;

$bot = new Bot(getenv('TELEGRAM_TOKEN'));
$response = $bot->setWebhook(
    new SetWebhook($_SERVER['argv'][1])
);

echo str($response).PHP_EOL;

Execute the script:

docker run --rm -v $PWD:/var/www/html -w /var/www/html \
    -e TELEGRAM_TOKEN=telegram_token \
    formapro/nginx-php-fpm:latest \
    php set_webhook.php "https://f2183d0.ngrok.io/get_updates"

If it goes well you should see:

HTTP/1.1 200 OK

...

{"ok":true,"result":true,"description":"Webhook was set"}

Pay attention that if you stop ngrok container and start again the URL will change. So, you have to set up webhook again.

Create updates handler

Create index.php. It’s a heart of your bot — a script that contains the bot logic:

<?php
// index.php

use Formapro\TelegramBot\Bot;
use Formapro\TelegramBot\Update;
use Formapro\TelegramBot\SendMessage;

include_once __DIR__.'/vendor/autoload.php';

if ('/get_updates' !== $_SERVER['REQUEST_URI']) {
    echo 'Telegram Bot Demo'.PHP_EOL;

    exit;
}

try {
    $requestBody = file_get_contents('php://input');
    $data = json_decode($requestBody, true);

    $update = Update::create($data);

    $bot = new Bot(getenv('TELEGRAM_TOKEN'));
    $bot->sendMessage(new SendMessage(
        $update->getMessage()->getChat()->getId(),
        'Hi there! What can I do?'
    ));

    http_response_code(200);
    echo 'OK'.PHP_EOL;
} catch (\Throwable $e) {
    file_put_contents('php://stderr', (string) $e);

    http_response_code(500);
    echo 'Internal Server Error'.PHP_EOL;
}

Start the bot server and leave it working:

docker run --rm -p 80:80 -v `pwd`:/var/www/html --net bot_ngrok \
    -e TELEGRAM_TOKEN=telegram_token \
    --name telegram_bot \
    formapro/nginx-php-fpm:latest

Test the bot

Open your bot in Telegram, type /start and in return, you should get Hi there! What can I do?.

This is it. Now you have a working playground and can continue observing other features of chatbot universe.

You can also visit our corporate blog to learn more about up-to-date IT trends and download some useful insights.

Setup Telegram Bot local workspace in minutes. was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

I had a project that ran on Docker Swarm cluster with MongoDB and in a while, I had to dump and restore DB. I tried to solve it by:

1. Adding HTTP endpoint to the application that returned all the collections data. The same for restoring. (cons: polluted app code, complexity, in-house solution).
2. Hassle with shell scripts that connect to the prod server, make a dump and download it. (good enough for simple Docker but not for Docker Swarm, cuz it makes it hard to get into a container and fetch dump file from it).

I wanted to use official Mongo commands: mongodump and mongorestore but didn’t want to install mongo locally. I ran everything in docker locally so I tried to combine them together:

docker run --rm mongo mongodump --host $1 --archive --gzip | cat > dump_$(date '+%d-%m-%Y_%H-%M-%S').gz

So easy and elegant way, where you have only one command and a single-file dump.

And what about restore?

cat dump.gz | docker run --rm -i mongo mongorestore --host YOUR_HOST --archive --gzip --drop --db=YOUR_DATABASE_NAME

Here I use a little hack of mounting one file into the container: push it into STDIN and read it in the container; -i is necessary to inject it STDIN

Also added some stuff and it turned into this.

Hope will be useful for somebody. Good luck!

P.S: this approach could be applied to other databases too.

You can also visit our corporate blog to learn more about up-to-date IT trends and download some useful insights.

An elegant way to dump MongoDB using Docker was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

This tutorial will guide you through the basic steps of deploying and running a secure Docker registry on Digitalocean.

Inherently, the registry uses insecure HTTP connections. Because of that, any docker daemon refuses to fetch images from such a registry unless you force it. Of course, a registry supports secure TLS connection but you have to cook certificates manually and don't forget to update them once they expire. This is far from to be desired, so let’s automate that with Traefik and Let’s Encrypt.

1. Create a droplet on DigitalOcean (referral). The easiest way is to use “One click apps” called “Docker 18.06.1~ce~3 on 18.04”. Make sure you selected a droplet with enough space. CPU and Memory are less important. Also, make sure you added your SSH key, we will need it in a moment. I assume you named a droplet docker-registry .
2. Don’t forget to set up a firewall to restrict access.
3. Configure DSN server to map requests to your domain to a newly created droplet IP address. I assume your registry domain name is registry.foobar.com.
4. Init Swarm cluster. Once the droplet is up and running, open terminal and ssh to it. Then run swarm init command:

$ ssh root@registry.foobar.com # or ssh root@your_droplet_ip
$ docker swarm init --advertise-addr=your_droplet_ip

5. Create a docker-compose file on a local computer:

6. Now, createtraefik.toml file in the same directory:

7. Create a deploy.sh script:

#!/usr/bin/env bash 

scp "docker-compose.yml" "root@registry.foobar.com:/docker-compose.yml"

scp "traefik.toml" "root@registry.foobar.com:/traefik.toml"

ssh "root@registry.foobar.com" "docker stack deploy --compose-file /docker-compose.yml foobar_registry"

Make the script executable:

chmod u+x deploy.sh

8. Deploy registry by running ./deploy.sh

9. Test it. Open link https://registry.foobar.com/v2/_catalog in a browser.

Now you could build and push your images to the registry:

docker build --tag "registry.foobar.com/foobar/app:latest" .
docker push "registry.foobar.com/foobar/app:latest"

The code could be found here.

Deploy and Run Secure Docker Registry was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

First, install the enqueue/async-command package:

$ composer req enqueue/async-command:0.9.x-dev

Create the file enqueue_async_commands.yaml and enable the feature there:

# app/config/packages/enqueue_async_commands.yaml

enqueue:
  async_commands: true

A simple use case example, that shows how to send a command to message queue so it could be processed later in a consumer.

<?php

use Enqueue\Client\ProducerInterface;
use Enqueue\AsyncCommand\Commands;
use Enqueue\AsyncCommand\RunCommand;
use Symfony\Component\DependencyInjection\ContainerInterface;

/** @var $container ContainerInterface */

/** @var ProducerInterface $producer */
$producer = $container->get(ProducerInterface::class);

$producer->sendCommand(
    Commands::RUN_COMMAND,
    new RunCommand('debug:container')
);

Optionally, you can get a command output, as well as exit code and error stream:

<?php

use Enqueue\Client\ProducerInterface;
use Enqueue\AsyncCommand\CommandResult;
use Enqueue\AsyncCommand\Commands;
use Enqueue\AsyncCommand\RunCommand;
use Symfony\Component\DependencyInjection\ContainerInterface;

/** @var $container ContainerInterface */

/** @var ProducerInterface $producer */
$producer = $container->get(ProducerInterface::class);

$promise = $producer->sendCommand(Commands::RUN_COMMAND, new RunCommand(‘debug:container’), true);

// do other stuff.

if ($replyMessage = $promise->receive(5000)) { 
   $result = CommandResult::jsonUnserialize(
      $replyMessage->getBody()
   );
}

echo $result->getExitCode().PHP_EOL;
echo $result->getOutput().PHP_EOL;

Don`t forget that you need a consumer working in a background:

$ ./bin/console enqueue:consume --setup-broker -vvv

Neat and simple solution. Enjoy!

Symfony Async Commands was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

The worst case scenario is when you have lots of empty queues and a few with messages. For example, It consumes from five queues and only one has messages and the receive timeout is one second. In this case, it would consume messages one per four seconds. The performance degrades a hundred times. It affected all transports except AMQP.

Since Enqueue 0.8.35 supports SubscriptionConsumer. The feature has to be enabled explicitly on 0.8.x version and will be the default approach since 0.9. It improves dramatically performance of the worst case scenario and opens a room for further improvements, for instance, Doctrine DBAL transport can fetch messages from multiple queues at one go.

To try it out, make sure you have at least enqueue\enqueue of 0.8.35 version.

QueueConsumer:

<?php

use Enqueue\Consumption\QueueConsumer;

/** @var \Interop\Queue\PsrContext $context */

$queueConsumer = new QueueConsumer($context);
$queueConsumer->enableSubscriptionConsumer(true);

SimpleClient:

<?php

use Enqueue\SimpleClient\SimpleClient;

$client = new SimpleClient('null:');
$client->getQueueConsumer()->enableSubscriptionConsumer(true);

Symfony:

# app/config/services.yaml
# composer upd enqueue/enqueue-bundle:^0.8.35

enqueue.queue_consumer.enable_subscription_consumer: true

Enjoy!

Enqueue. Multi-queue consumption improvements. was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

In Symfony world a mainstream solution to work with it is Doctrine MongoODM library. Unfortunately, it is extremely slow and forces you to maintain model schema which is effectively kills all profit at the grassroots level. This happens mostly because Doctrine has to traverse an object tree to map values to corresponding object properties or in other words hydrate an object. It also has to do an inverse operation to take a storable data.

I was not comfortable with that state of things. So I tried to find a better approach that amplify power of the store. What if we use a data array provided by MongoDB as is, just wrap it a little bit by an object. The hydration and persistence would take no time and be as fast as possible. Effectively it is just about setting or getting an array from an object’s property called values. We always have storable data within reach, no need to traverse anything. On the other hand, a cover object plays a role of a contract adding objectish aspects like data encapsulation, methods\properties visibility, type checks, method definitions, interfaces and so on.

Here’s just a few examples that demonstrates the idea :

So I started developing a solution around the concept, which is later became known as a schema less, fast hydration and persistence document mapper library called makasim/yadm and the bundle makasim/yadm-bundle.

Let’s move to a practical example to see how it all works. For that we have to create a basic project and require the bundle:

$ composer create-project symfony/skeleton acme; cd acme;
$ composer require makasim/yadm-bundle

The model design is a essential starting point so take a look at the imaginary Order model below. It takes a currency, amount, and a number.

The next step is to configure the bundle:

CRUD examples:

I’ve showed just a small portion of awesomeness that comes with the bundle.

There are a lot more so try it in your next Symfony application.

Have fun and give me feedback on occasion!

An efficient way to treat MongoDB in Symfony. was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

First of all, here is our topology:

Lets have a brief overview on how retrying works.

1) Imagine something goes wrong in a comments queue consumer. Say, there is an exception or reject.

2) We acknowledge an original message, create its clone and send it to TTL EXCHANGE with routing key = retry-1

3) Message goes to corresponding TTL queue — comments-retry-1-30s (ttl queue1 30s on image above) with TTL = 30s

4) This queue does nothing, even doesn’t have any consumers connected. The message is expired there and therefore moved to corresponding DLX (Dead Letter Exchange).

5) DLX is configured to send all messages to our payload queue comments, so we have a second try now.

6) If it fails again, do repeat, but this time publish the cloned message to comments-retry-2-10m queue.

The consumer might lost the connection with RabbitMQ (consumer crashed/killed or connection issues). In this case we should do next:

1. RabbitMQ notices that connection has been lost and therefore return the message to the queue, marking it as redelivered.

You can read more about redelivery pit falls in the article

2) When the consumer recovers it handles that message again. Now it checks the redelivered flag. If it’s true it DOSN’T EVEN TRY TO HANDLE IT - just sends it to retry flow. After it goes through the whole retry flow, the message will come back without redelivered mark and will be handled as always.

Why don’t we handle redelivered messages and just send them to retry? Because we don’t know for sure what caused that redelivery. What if this message is broken (say we try to handle an old message with a new code) and each time it is handled a consumer crashes? So instead of having endless loop of crashes we have maximum 3 crashes and then the message is erased.

Each work (payload) queue (like comments and so) has its own set of retry queues and exchanges: 1 TTLX, 3 ttl queues and DLX

Implementation

First of all let’s create our comments queue and set up retry queues and exchanges:

Next, if we have a failed message we can send it to retry loop like this:

Keep in mind that we send to retry loop failed messages for both reasons: failing inside handler (because of reject OR exception) or failing because of connection issues (in such case AMQP will mark a message as redelivered). This is what happens in sendMsgToRetry: we acknowledge an original message, determine which attempt it is and send a new message (clone) to the next retry loop.

You can find a full implementation with descriptive comments here.

Also, you can visit our corporate blog to learn more about up-to-date IT trends and download some useful insights.

RabbitMQ retries on Node.JS was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

• InfluxDB is open source time series database designed to handle high write and query loads.
• Chronograf is open source web application for infrastructure monitoring, alert management, data visualization, and database management.
• Kapacitor is open source data processing framework that makes it easy to create alerts, run ETL jobs, and detect anomalies.

I’d like to share my findings on how to setup them locally using Docker and how to use it from PHP.

Luckelly, InfluxData provides us with good pre-build images. We just have to properly configure and link the services.

In a shell, open an empty directory and create a docker-compose.yml file there, as follows:

To start containers execute:

docker-compose up -d

Open http://localhost:8888 to access Chronograf UI.

Now, let’s install influxdb/influxdb-php which we use to query and store points:

composer req influxdb/influxdb-php:^1

Here’s the script that sends two points to InfluxDB:

To run it open the browser and type http://localhost:80.

As you can see the whole process was smooth and took just a few minutes. Having this working you are ready to explore great features of InfluxDB, Kapacitor. Have fun!

You can also visit our corporate blog to learn more about up-to-date IT trends and download some useful insights.

Docker setup for InfluxDB, Chronograf, Kapacitor, and PHP was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

It is time to settle a date of a stable release and possibly some intermediate steps towards it:

• The 1.0 will be released in December 2018.
• The 1.0-beta1 will be released in August 2018.
• The 1.0-alpha1 will be released in March 2018.

I don’t expect any serious changes or BC breaks. I’d be happy to gather as much feedback as possible during the next months prior to the release.

I call for feedback and comments. Feel free to open an issue to provide it.

Queue Interop roadmap. was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.

Increasing popularity of AMQP interop prompted a wave of questions regarding which adapter is the fastest. In this post, I am going to share my findings of message publishing performance. There will be a similar post about consumption side.

• php-amqplib
• amqp-ext
• bunny

as well as their interop adapters:

• enqueue/amqp-lib
• enqueue/amqp-ext
• enqueue/amqp-bunny

and one from Golang world:

• streadway/amqp

We are going to compare performance of the libraries, the cost of an interop wrappers. It would also be also interesting to compare PHP implementations with something from another world to feel the difference. I chose streadway/amqp Go’s library which seems like widely used one.

A quick note on methodology. I ran benchmarks on DigitalOcean droplet (1Gb RAM, 1 CPU, Ubuntu 16.04 x64) inside docker containers. Every run was done in new containers. The benchmarks source code could be found here.

A picture is worth a thousand words so here’s some statistics:

Check for yourself

You can run benchmarks yourself. For that, you have to download enqueue dev repository, do composer install and run bench script. Like this:

git clone https://github.com/php-enqueue/enqueue-dev.git
composer install
BODY_SIZE=10000 ./bin/bench bench/AmqpExtBench.php --report=aggregate

Conclusion

• amqp-ext is the fastest among PHP implementations.
• bunny is almost as fast as the AMQP extension.
• php-amqplib is sometimes two times slower than other solutions.
• Interestingly Go script was the real winner only for a really small payload. For the rest it showed a bit better result than amqp-ext. It should’ve been much better IMO.

PHP AMQP clients. Benchmark them all! was originally published in formapro on Medium, where people are continuing the conversation by highlighting and responding to this story.