As mentioned the original post at https://medium.com/faun/zabbix-v5-0-and-grafana-automated-installation-script-6cd1a87cc36a, you can use it to install the latest LTS version of the Zabbix on Ubuntu 20.04, 22.04 and CentOS 7

For more information and installation instructions, check out the repositoy: https://bitbucket.org/secopstech/zabbix-server/src/master/

Any bug reports, ideas and feature requests are welcome.
https://bitbucket.org/secopstech/zabbix-server/issues

It’s a simple shell (bash) script that installs docker (ce), docker-compose, and jq packages and deploys the Zabbix containers from the official Zabbix Docker images by using docker-compose.

Also, in order to bring better data visualisation support to Zabbix, the script deploys a grafana (7.2.1) service from the official docker image and imports some useful dashboards in it.

Zabbix server uses a MySQL container as a DB backend. Docker-compose creates two separate host volumes, for MySQL and grafana containers in order to hold their data on the local host permanently. So, as long as you do not delete the docker volumes, Zabbix and grafana data will be persistent.

The installation script invokes Zabbix and grafana APIs via curl to do all customizations/configurations like creating additional Zabbix items/triggers/host groups and grafana integration etc.

Here is a complete list of actions that performed by the script:

• Installs docker (ce), docker-compose, and jq.
• Creates self-signed SSL certificates for Zabbix and grafana frontends.
• Deploys Zabbix services via docker-compose.
• Adds a bunch of hosts group to Zabbix.
• Creates auto registration actions for Linux and Windows hosts.
• Creates an item to show free usage in percentage for Linux and Windows hosts.
• Adds a free memory in percent check for Windows hosts.
• Changes filesystem and netif LLD rules intervals to 30minutes for Linux and Windows hosts.
• Disables annoying Windows service items LLD rules. (If you want to get a lot of meaningless Windows service alert, just enable it.)
• Adds alexanderzobnin-zabbix-app datasource plugin to grafana.
• Adds briangann-gauge-panel, btplc-trend-box-panel and jdbranham-diagram-panel plugins to grafana.
• Invokes the grafana API to add custom dashboards for Linux and Windows hosts. (an overall system status dashboard will be also imported.)
• Sets email notification for admin user. (Optional)
• Sets slack notification for admin user. (Optional)

Usage

To invoke the Zabbix server installation, execute the “zabbix-server” script with “init” parameter, and follow the instructions.

# bash scripts/zabbix-server init

Note: You can find out a complete output of an example execution at the project README file.

Accessing Zabbix and Grafana UIs

Zabbix UI is accessible via https://host-ip:8443 and the default username and password is Admin/Zabbix

Also, grafana UI is on https://host-ip:3000 with admin/zabbix

Both services use https with a self-signed certificate.

Screenshots

These three graphs listed below are from the “Linux Server Dashboard” that imported by the script. (There is also another dashboard for Windows hosts with the same layout.) This is a ready-to-go dashboard so all metrics are collected automatically. For example, any network interfaces and disk devices will be discovered and showed up on the dashboard.

Another dashboard which also imported is “Zabbix System Status” that shows the overall status of your infrastructure.

And this is the custom alert messages layout to send Zabbix admins’ notification channels.

Any improvement ideas or feature requests are welcome.
https://bitbucket.org/secopstech/zabbix-server/issues

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Zabbix v5.0 and Grafana Automated Installation Script was originally published in FAUN.dev() 🐾 on Medium, where people are continuing the conversation by highlighting and responding to this story.

Kubernetes and Data Persistence

In the early versions of Kubernetes, we intended to use it for our stateless services as it was designed for these kind of workloads at the beginning. For those days, data persistence wasn’t a concern, which running stateful services that tightly coupled to data on Kubernetes wasn’t a common approach. (There was no durable local storage support in late 2015 yet. https://github.com/kubernetes/kubernetes/pull/1311)

However, as Kubernetes becomes more popular platform, it started to provide more reliable features that changed our mindsets about scheduling stateful services on it. Currently, we are able to use many type of storage volumes including GCE Persistent Disk, AWS Elastic Block Store, vSphere Volume, Glusterfs, NFS and Ceph. So we‘re now more comfortable to run services that need robust storage backends.

Especially, if your Kubernetes environments are placed in a cloud provider such as Google Cloud Platform or Amazon Web Services, it will be easy to use a storage backend for any data persistence requirements, since all cloud providers offer their own reliable storage backends built-in. For instance if your k8s cluster is based on GKE, then your stateful services can be backed by GCE persistent disks.

But when it comes to on-premise Kubernetes environments, there are still some challenges as you know.

On-premises Challenges

For sure, it’s a bit more difficult to implement a storage backend in on-premise Kubernetes environments unlike the cloud providers. In such situations, you need to choose a production grade software defined storage solution, which should be seamlessly integratable to your existing environment and more it should be a robust, scalable, easy to manage platform.

Fortunately, there are many good solution that can be used as storage backend on Kubernetes environments. You can find out many of them, by googling “software defined storage for Kubernetes” keywords. However, integrating a software-defined-storage (SDS) solution to a Kubernetes environment is not an easy peasy task at all. As the storage backends are one of the most important components, they should be reliable; thus you need to implement them carefully for production use. In other words, you need to know what you’re doing; otherwise, it will be inevitable to be challenged by the day 2 problems.

Well, at this point, projects like Rook, that designed to seamlessly implement these kind of important components, become a very important aspects for cloud-native environments. As a storage orchestrator, Rook aims to handle all storage related tasks on Kubernetes environments for you. This is why it is hosted by Cloud Native Computing Foundation as an incubation-level project.

So, in this article, I’ll mention Rook’s features and this will also be a “how to” guide about building and integrating a Ceph Storage Cluster on top of a Kubernetes.

What is Rook ?

As it defines itself:

“… an open source cloud-native storage orchestrator, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.

Rook turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.”

So, you can use Rook to implement software defined storage building blocks for your Kubernetes environments in an automated manner. It runs as an operator in Kubernetes cluster and enables selected storage system to run on the same Kubernetes cluster. In this way, rook automatically configures a CSI (Container Storage Interface) driver to mount storage spaces to your pods.

While, in this article I’ll explain that how to implement a high available Ceph storage system to a Kubernetes cluster for providing block storage, object storage, and shared filesystems mounts to pods, you can to take a look for other Storage provider options by visiting https://github.com/rook/rook/blob/master/README.md#project-status

Ceph Cluster on top of Kubernetes

As you know, Ceph is a highly scalable distributed storage solution that has a very extensive usage on production infrastructures just as it defines itself:

Ceph is for you! Ceph’s foundation is the Reliable Autonomic Distributed Object Store (RADOS), which provides your applications with object, block, and file system storage in a single unified storage cluster — making Ceph flexible, highly reliable and easy for you to manage.

It can be implemented Kubernetes to provide storage for applications, however -as I mention above-, to achieve this, you need to know all aspects of Ceph’s dynamics to build a production ready cluster and you should make a reliable Kubernetes integration.

This is where Rook comes in… It’ll handle all Ceph installation and management tasks for us to provide Dynamic volume provisioning for our applications.

How it works ?

Before we go further, it’s worth to mention how Rook integrates Ceph to Kubernetes clusters.

As you can see in the image below, it uses Kubernetes primitives to run on Kubernetes.

So, you basically deploy Rook on your Kubernetes and it takes care of the rest to build, manage and monitor a Ceph cluster.

Typically, Rook uses custom resource definitions (CRDs) to create and customize storage clusters and these CRDs are implemented to Kubernetes during its deployment process. Also it has an operator within Kubernetes named Rook Operator for Ceph which automates configuration of storage components and monitors the cluster to ensure the storage remains available and healthy.

Rook, also runs a daemonset named rook-discover that start a discovery agent pod on every nodes of your Kubernetes cluster to discover any raw disk devices (no partitions or formatted filesystems) or raw partitions (no formatted filesystem) that can be used as Ceph OSD disk. If minimum required number of disks (for redundancy purposes should be at least 3) are discovered, rook operator uses these disks to build the Ceph Cluster. If you add more disks to your nodes, the operator discovers them via its agents and scales the ceph cluster automatically. As well Rook configures Ceph-CSI drivers to provide Block Storage and Shared Filesystem mount support for pods.

Also, for monitoring requirements, Rook enables Ceph Dashboard and each Rook Ceph cluster has some built in metrics collectors/exporters for that can be scraped by Prometheus.

Deploying a Ceph Cluster with Rook

A Ceph cluster deployment with rook is a quite straightforward process that we only need to clone a git repo and run a few Kubernetes manifests via kubectl. That’s all.

But first it will be good to clarify requirements and deployment options.

Requirements

OSD Disks and Rook Modes
Rook needs to use some storage areas to create Ceph OSDs which will be base the cluster. In order to provide these areas, there are two different mode that named “Host-Based” and “PVC-based”.

• Host-based Cluster:
  In host based cluster mode, rook detects all raw (unformatted) devices or raw partitions on all (or selected) Kubernetes nodes and creates the ceph cluster. In this mode, all you need is to attach some raw disks to your nodes. (Note that, minimum three disks on three different nodes are required to build a high-available ceph cluster)
• Note that, minimum 3 disks are required to build a high-available ceph cluster
• PVC-based Cluster:
  On the other hand, PVC-based cluster requires a bit more things, like an existing StorageClass. Because in this mode rook dynamically claims number of PV from the StorageClass that you define, and uses them to create OSD disks. Also this mode requires, Kubernetes version =>1.13.0.

Since, Host-based cluster mode is the most simple way to build a ceph cluster, we’re gonna use this mode in this article. So on my setup, I have three raw disks are attached to three k8s nodes each.

Deployment

At the time of writing this article, the latest release of rook was 1.3.5 which we’re gonna deploy this version, but please check its releases page to be sure you’re on the latest version.

First step is cloning rook’s 1.3 branch.

At this point, we will apply three k8s manifests which reside in rook/cluster/examples/kubernetes/ceph directory to build the cluster.

So let’s apply them one by one:

common.yaml
This manifest, basicaly creates some “CustomResourceDefinitions”, “ServiceAccounts”, “ClusterRoles” that requires by cluster formation process.

So apply this yaml.

As you see that a bunch of resources are created.

Now we’re ready to deploy Rook system components by applying operator.yaml manifest.

operator.yaml
This manifest simply creates a deployment that deploys a rook-ceph-operator pod and its configmap. This operator is the main component of Rook that responsible to automatically create and manage Ceph clusters on top of our Kubernetes cluster.

So, we just create it:

After few seconds, the components should be running like below:

You can see that there is a “Deployment” named rook-ceph-operator which responsible to manage ceph clusters as I mentioned before.

Also, there are a daemon set named rook-discover which runs a three pods (as my test k8s cluster has three nodes). This service is regularly checks for any raw device or partitions on k8s nodes to use them as a member of Ceph cluster.

Note that, any rook related resources is placed in rook-ceph namespace.

cluster.yaml
Ok, rook is ready and we can deploy the ceph cluster by running cluster.yaml manifest. This manifest simply tells rook that how we wanted to create a ceph cluster.

By default it’ll use all nodes and all (raw) devices or partitions to create Ceph cluster as follows.

As I mentioned before, I have three raw disks attached to my three k8s nodes and with this setup, rook will automatically detect and use all of those disks to create Ceph OSDs.

Also note that, rook will use /var/lib/rook directory on each nodes to cache ceph mon ad osd configuration data. If you have previous rook managed ceph cluster, you need to delete the data on this directory.

So, we apply the manifest:

After a while, rook will create all necessary resources to form up a ceph cluster. When you check the status of the deployment, you should see something like below:

What we see in this output is, the ceph cluster is successfully deployed and there are a bunch of new deployment.apps are in placed. For example, rook-ceph-mgr, rook-ceph-mon and rook-ceph-osd deployments, manages pods that related to ceph internals (which I don’t mention them in detail as this is not an entry level Ceph documentation. ) Also, rook has deployed some CSI plugins named “cephfs” (for mounting ceph volumes to pods) and “rbd” (for mounting raw blocks to pods).

Checking Ceph Cluster Health

As it seems our cluster has deployed, we can check its status by either using its dashboard or using ceph commands via ceph toolbox pod.

Accessing Ceph Dashboard

By default, rook enables ceph dashboard and makes it accessible within cluster via “rook-ceph-mgr-dashboard“ service. So we use kubectl’s port-forward option to access the dashboard from our local computer.

Now, should be accessible over https://locallhost:8443

Default username is admin , and its password can be collected from rook-ceph-dashboard-password secret like below:

So, it looks like everything is OK as it says on Cluster Status section below.

Since ceph is high-available storage system, there are three Monitors is available in order to avoid create a SPOF (single point of failure). Also, we have three OSDs which uses the raw disk devices that attached to k8s nodes each. You can scale your OSDs by adding more raw devices to your nodes. As I mention before, Rook will automatically watch for new nodes and devices being added to your cluster to scale OSDs.

As you may notice, there is only one Manager daemon currently. Although you can think that it would be nice if it has some standby daemons, rook leverages Kubernetes auto healing feature on this. As manager service is about to provide additional monitoring and interfaces to external monitoring and management systems, Rook maintainers have decided to improve its eviction process and not implement standby manager daemons as they discussed it at (https://github.com/rook/rook/issues/1796)

Accessing Ceph Cluster via CLI

Another options to interact rook ceph cluster is using rook toolbox. It’s a container that contains common tools used for rook debugging and testing ceph; also the toolbox is based on CentOS, so you can install any other ceph related tools on it via yum.

In order to create this container, we can use “toolbox.yaml” manifest which resides in /rook/cluster/examples/kubernetes/ceph:

At this point, we can attach the toolbox.

Now, we’re ready to use ceph commands. For example, to get cluster status you can use “ceph status” command:

To see osds status:

For other ceph control commands, you can check its documentation at https://docs.ceph.com/docs/giant/rados/operations/control/

Ceph Cluster Storage Modes

OK, we have a ceph cluster is in place and now it can be utilized.

Typically rook ceph cluster provides three different storage mounting mode for pods. While, the mostly used ones is “block storage”, “shared filesystem” and “object storage” are also supported.

Block storage, allows a single pod to mount storage that typically used for databases, nosqls etc. (in this article we’ll use this mode as an example)

On the other hand a shared filesystem can be mounted with read/write permission from multiple pods. It’s suitable if your application requires to access to the same set of data.

And object storage exposes an S3 API to the storage cluster for applications to put and get data over http.

Creating a StorageClass for Rook Managed Ceph

In order to provision block storages a “StorageClass” and a “CephBlockPool” should be created first.

To do this we’ll apply a “storageclass.yaml” manifest which placed in “cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml”

But first, let’s take a look at the manifest:

As you see, it declares two Kubernetes resource that I mentioned.

CephBlockPool definition will create a ceph pool named “replicapool” by using three OSDs to set three replica; so our example will require at least 1 OSD per node, with each OSD located on 3 different nodes; because the “failureDomain” is set to “host” and the “replicated.size” is set to “3” in the manifest.

And StorageClass definition will create a SC which will utilize the “replicapool” CephBlockPool. So whenever you want to dynamically provision a storage space for your pods, you can request it from this storageclass to get the space from ceph cluster.

OK, we create the resources:

Now, you should see a storageclass named “rook-ceph-block”:

And also, “replicapool” ceph pool should be in place:

Note: If you didn’t run toolbox deployment, the command above won’t work obviously. In this situation you can check the pool from ceph dashboard.

Right now, we’re ready to dynamically provision storage spaces for pods.

Dynamic Provisioning Example

In this section, we’re gonna create a sample wordpress app by using example “mysql.yaml” and “wordpress.yaml” manifests reside in rook/cluster/examples/kubernetes/ directory. These manifests will dynamically provision two persistent volume which will be created by Rook for mysql and wordpress data placement.

By default, example wordpress manifest creates a service for port 80 by using LoadBalancer service type, however my test Kuberbetes hasn’t a cloud controller to interact any cloud based external load balancer; so I’ll change the type as ClusterIP which the service will be available from inside the Kubernetes cluster as this example is for demonstration purpose.

So let’s create the app:

As you see from the output above, two “PersistentVolumeClaim” has created for mysql and wordpress. Let’s see the PersistentVolumes and PersistentVolumeClaims:

It looks like, they are created and successfully bounded.

And here is the output of the deployment, pods and services related to wordpress app.

Now, we can fetch wordpress enrty page from one of our Kubernetes nodes by using wordpress service’s ClusterIP:

It seems, our wordpress app is running and writes its data (both mysql and wordpress http server) to the Rook managed ceph persistent volumes that created dynamically.

Wrapping Up

Just we’ve seen it in action, rook is a very promising project that simplify ceph cluster deployment and integration processes.

As, it’s encourages people to run stateful applications -that requires reliable storage systems within on-premises environments-, I’ve always liked such kind of good quality projects, which expands Kubernetes’ use cases and boost the ecosystem.

Rook: A Storage Orchestrator to Run Stateful Workloads on Kubernetes with Ceph was originally published in Devops & AI Türkiye☁️ 🐧 🐳 ☸️ on Medium, where people are continuing the conversation by highlighting and responding to this story.

Though, there are many good articles related to Kong, I wanted to share my opinions and write a “yet another tutorial” about installing, configuring and securing Kong for production environments. So, this article will be such a “quick start guide” that includes bit more information for production use.

If you already know what Kong is and its features, just jump to Installation & Configuration section.

What is Kong!

Simply, it’s an API gateway, that sits in front of your backend services and forwards client requests to those services. Since Kong is the first point that client requests hit, it can transform and/or routes any requests or responses based on your configuration. So you can manipulate any requests or responses by adding, appending or removing headers on the fly and route them to wherever you want.

What’s more, as an abstraction layer it provides additional features with its plugins such as authentication, rate limiting, caching, logging, bot detection, CORS, IP restriction and many more. This means, that developers don’t need to implement these features in each micro services, thus they can be more focused on business logic which improves productivity.

Kong is a kind of “platform agnostic” solution that runs on almost every platform. You can run it as a docker container or install on a GNU/Linux distribution in on-premise or cloud based environments. Even, it can run as an kubernetes ingress controller.

Architecture

Here is a simple diagram to see how Kong works and how it extends your application ecosystem with its features.

How It Works

Kong is built on top of Nginx, -this is why it’s highly extensible- and by using its own admin api, it configures the nginx server to provide proxying for backend services.

Kong uses OpenResty which is a kind of enhanced version of nginx that includes lua-nginx-module. Thus Kong is distributed along with OpenResty to leverage its lua scripting capability to configure and extend nginx.

In other and simple words, Kong accepts your reverse proxying directives via its own Admin API over http/s, translates them to nginx configuration using lua scripting language and builds a gateway platform for micro service backends.

Admin API

When you install Kong on a host, the Admin API will be accessible on tcp/8001(http) and tcp/8444(https) ports. By interacting the Admin Api , you can configure Kong dynamically.

As a security best practice, Admin API ports are for internal use and should be firewalled.

Proxying

On the other hand Kong listens tcp/8001(http) and tcp/8443(https) ports to receive end-user requests and according the configuration you define, forwards to your endpoints. Those ports are publicly-facing services that must allow external connections.

Datastore

Kong uses an external database to store configurations data. You passes your configuration directives over admin API and Kong stores them in its datastore. And multiple kong servers can use a central database. (Which makes the gateway platform high-available and scalable. I’ll mention this in detail on Clustering section.)

Supported datastores are Cassandra and PostgreSQL. Though Cassandra is preferred datastore to build a multi-regional Kong environment, PostgreSQL can also be used for production without any issues. (In this post, we’ll use a Postgres replication for the sake of simplicity.)

Cache

As you know, when it comes to performance, databases can be easily become a main bottleneck. In order to avoid this and improve performance, Kong maintains a cache to hold the configuration data in the node’s memory.

No matter which datastore engine you use, when you call admin API to configure Kong, configuration data is written to the external datastore, then pulled from the external db and written to the node’s memory as cache; finally, configuration is applied.

At this point there is no need for a database roundtrip anymore while proxying requests. If you do any configuration change, Kong invalidates cache and pulls the new configuration to cache it again. This approach boost Kong’s performance obviously.

In order to avoid to break Kong’s cache invalidation mechanism, you should not manipulate the datastore db directly.

DB-less Mode and Declarative Configuration

As I mentioned in Datastore section, kong requires a database for the configuration data, however there is another option called DB-less mode. In this mode the configuration directives is declared as a JSON or YAML formatted configuration file, thus Kong reads this file to get configuration directives and write it to node’s memory.

With this mode, you can manage Kong by using a CI/CD tool. Since, configuration directives are stored in a static file, you can keep it in a git repository and distribute to multiple kong nodes. However in this mode Admin API is read-only, because the only way to configure Kong is using a declarative configuration file. In other words, you can’t pass any configuration directives to the admin api over RESTFull api.

Moreover, some useful plugins are not compatible with this mode, since they require a database by design. For example, rate-limiting plugin requires to a database to hold limiting values.

This mode may seem useless for production use however it is also a part of another mode called Hybrid mode that can be used create a Kong cluster. Next section I’ll explain this mode as well.

Clustering

Probably one of the most reliable service within the infrastructure should be API gateways and we don’t want to create a SPOF on this layer. Since they are act as the entry point to your backend services, any outages or performance issues couldn’t be tolerated on this layer. Needless to say, if the gateway dies, practically the application dies too. So it should be redundant and scalable.

As I mentioned before, Kong uses a datastore to holds its configurations and multiple kong node can use this central datastore to form a cluster.

A typical kong cluster looks like:

Note that, Kong is not a load balancer, you need a load-balancer in front of your Kong cluster to distribute your traffic.

So, if you need to handle more incoming requests, you can easily add more nodes horizontally by installing Kong on your new nodes and pointing them to the central datastore.

This simplicity and flexibility makes Kong to an automation suitable platform as well. For example, you can track your Kong cluster resources utilization (by collecting metrics via its prometheus plugin) and decide to scale up or down automatically. Kong installation and initial configuration is a quite straightforward process which can be easily automated.

Hybrid Mode Clustering

Classically, Kong nodes depend to the central datastore, but this is not the only option, because there is another way called Hybrid mode to build a Kong cluster.

In this mode, Kong nodes are separated into two roles called Control Plane Nodes(CP) and Data Plane Nodes(DP)

While, CP nodes -which provides Admin API-, are the only components that interact with the central datastore, DP nodes -which handle incoming requests- fetch their configuration from a CP node.

Here’s a good demonstration of two Kong cluster modes.

As you noticed, this mode is used db and db-less mode both, this is why it’s called hybrid.

It has some benefits such as reducing datastore load, improving security and easiness of management.

Though this mode is very suitable for distributed environments, I’m gonna configure Kong cluster in its classical way on this article for the sake of simplicity.

For further information about hybrid mode, please check its documentation https://docs.konghq.com/2.0.x/hybrid-mode/

Features

Kong has many useful features (known as plugin). These plugins either comes with Kong as bundled or provided by the community. As I said briefly, it’s using OpenResty’s ngx_http_lua_module to provide its core configuration management and plugin management infrastructure and this makes possible to write new plugins for Kong to extend it further. Thanks to its wise approach, there are many plugins on the Kong Plugin Hub!

Here are some useful plugins to worth mention:

Authentication Mechanism

Authentication plugins, provide bunch of authentication mechanisms to the backend services; and it’s possible to use different mechanism for each service. For example, while service-A is using basic authentication, service-B only accepts request with JWT token.

One of the well-known benefit to use an API gateways, makes it possible to offload the authentication overhead to the gateway. Thus, developers can be better focused on business logic of their backends without concerning about authentication implementation to each backend service. (Of course you need to be sure that the services are well isolated to unrelated connections nonetheless.)

Some of the mechanisms that supported by Kong is:

• Basic Auth
• LDAP integration
• OAuth 2 integration
• Key based auth
• HMAC auth
• JWT tokens

Security

Kong has some goodies to add security related capabilities to the endpoints in an effortless way. For example, by using bot detection plugin, you can detect and block any kind of requests that initiated by specific bots or custom clients. Or with its IP restriction plugin, you can easily restrict access to your services by either whitelisting or blacklisting IP addresses.

Here is built-in plugins come with Kong open source edition. (It has a commercial version named Kong Enterprise which includes more features.)

• ACME (Let’s encrypt and ACMEv2 integration)
• CORS
• IP Restriction
• Bot Detection

Traffic Control

Traffic control plugins are very useful features that everyone might be needed. For instance, Proxy cache plugin, caches mostly responded request in Kong nodes’ memory (we will mention that what a kong node is later).

As you know caching mechanisms are important things to reduces the load on the backends especially related to the data layers. If you cache frequently requested responses, that runs a bunch of heavy SQL queries on the backends, then you can reduce load on your db and related services.

Other plugins are:

• Rate limiting
• ACL (defines, who can access the services)
• Request body size limiting
• Request terminating (Terminates certain request with a specific status code.)

Analytics & Monitoring

When it comes to troubleshooting, (especially for performance related issues) one of the most important thing is “metrics” for sure! Particularly, if we’re talking about microservice environments, metrics become an imperative situation.

Though microservices offer a lot of benefits (which I won’t mention them, since this is not a post about it), also brings its challenges too. Unlike old school monolith applications, decoupled nature of microservices increase the complications of the visibility. So, on a typical microservice environment (since there are bunch of independent services that talk to others), you need to collect good quality metrics and correlate them in an understandably approach in order to understand how performant your environment is!

Well, Kong is an api gateway and it sits in between clients and services. (also you can place it in between your microservices as well.) therefor you can grab a lot of meaningful metrics from it such as latency values of your endpoints or you can collect tracing spans for any api calls. This is why Kong has these plugins below:

• Prometheus
• Zipkin
• Datadog

Logging

Needless to say, another “must have” feature for distributed applications is logging.

Logs say a lot! Based on their importance level, they should be collected, parsed, stored and visualized. Nowadays, we use logs to produce metrics that processed on the modern monitoring systems, then we use these metrics to fire alerts on serious events. They are became important resources more than ever.

So, these plugins ship logs to remote logging systems in various ways to make them to be processed!

• Syslog
• HTTP Log
• StatsD
• TCP/UDP Log

Transformation

Another useful subset of plugins are transformation related ones. These plugins can be used for manipulate any requests or responses on the fly. For example you can add, remove or replace header, body or querystrings of the requests or responses. Also there is a good plugin called Correlation ID, which injects unique ID to requests or responses in order to track http traffic flow.

• Request & Response Transformer
• Correlation ID
• gRPC Web
• Inspur Request & Response Transformer

Installation & Configuration

In this section, three separate CentOS 7 instances will be used to build a kong cluster. And in order to keep it simple, every components will be run on top of Docker.

Here are the components:

• Datastore
  As datastore, we’ll use a PostgreSQL docker container on one of the CentOS instance.

Note that your database should be redundant for production environments. So if you gonna deploy Kong for production use, you need to use a PostgreSQL replica cluster or Cassandra Cluster in order to avoid a single point of failure.

• Kong Nodes
  Two other CentOS instances will be the Kong nodes.

Docker and Kong Nodes: It’s OK to run kong nodes as docker containers in production, but they should also be redundant. So don’t run multiple kong nodes in a single instance.

Finally, my example environment ip addresses looks below:

kong-datastore: 10.10.10.11
kong-node-01: 10.10.10.21
kong-node-02: 10.10.10.22

Step 1 — Prepare Nodes

If you don’t have docker in your nodes, install its community edition as following steps:

Step 2 — Deploy PostgreSQL Container on Datastore Node

Connect your kong-datastore instance and run:

Now, we started a PostgreSQL 9.6 container on datastore node. As we passed the environment variables, a database named kong is created. So we can connect it with kong/kong credentials.

Important note: Again, this PostgreSQL deployment is just for demonstration purpose, you need to use an external PostgreSQL replica or Cassandra cluster for production use.

Step 3 — Prepare Kong Database

Next, switch to one of your kong node instance and run bootstrap command to make initial configuration on the db, such as creating schema etc:

As you noticed, we’ve pointed it to our postgres host via KONG_PG_HOST parameter.

Step 4 — Starting Kong Nodes

Finally, start kong container on your every kong node instances:

With this command set, we first tweaked the kernel to get better performance and then started kong node with some configuration parameters which we passed them as environment variables.

Here’s variable explanations:

KONG_DATABASE and KONG_PG_*
Defines which datastore that kong node will connect.

KONG_{PROXY|ADMIN}_{ACCESS|ERROR}_LOG
As a common containerized application approach, this parameters redirect admin api and proxy http logs to stdout and stderr by its category. (There are some good logging plugins to ship the logs remote, that you can enable regardless of these directives.)

KONG_{ADMIN|PROXY}_LISTEN

These environment variables, configure addresses and ports on which the proxy and admin api should listen for HTTP/HTTPS traffic.

According to our configuration, the admin api will listen to tcp/8001 for http and tcp/8444 for https traffic and proxy server will listen to tcp/8000 for http and tcp/8443 for https.

Note that, backlog=N parameter, sets the maximum length for the queue of pending TCP connections. This number should not be too small in order to prevent clients seeing “Connection refused” error connecting to a busy Kong node.

And this is related to net.core.somaxconn parameter, so you need to increase this parameter at the same time to match or exceed the backlog number. This is why we’ve tweaked kernel parameters and used “ — sysctl net.core.somaxconn=16384” line in our docker run command above.

KONG_ANONYMOUS_REPORTS

By default, anonymous reporting feature that sends anonymous usage data such as error stack traces to help improve Kong, is enabled. But I think it’s not good for any production environments. So we disabled it.

Step 5 — Check if Kong Nodes are running

Now, kong node should be run and provides its admin api via tcp/8001 or tcp/8444 on localhost.

So you can call it from any of kong node’s localhost address with curl like:

If everything is OK, it should return a JSON output that shows current configuration.

Securing Kong Admin API

As you noticed, when we run node kong container, we exposed admin api port on localhost with “-p 127.0.0.1:8001:8001 and -p 127.0.0.1:8444:84444“ parameters which means the admin api wouldn’t be accessible other than localhost. Since the admin api is the most important component that provides a RESTful interface for administration and configuration purposes, it should be well protected. So it’s a good habit to not expose it to network by default as we did.

But at the end of the day you will need to use the admin api remotely. In order to expose the admin api securely, adding fine-grained access control measures such as authentication and authorization is a good idea.

In this section, we will leverage kong itself to provide proxy to its own admin api. This will be also a good practice to learn how to use Kong.

Step — 1 Create a service

Services are used to refer to the upstreams which the incoming requests will be passed.

So with this command below, we’ll create a service called admin-api which refers the localhost’s port 8001 as an upstream.

Step — 2 Create a route to pass requests to the admin-api service

Routes specify how (and if) requests are sent to their Services after they reach the API gateway.

So we’re gonna create a route under the admin-api service to tell Kong that if it receives any incoming requests for the /admin-api endpoint, pass it to the admin-api service.

As our admin-api service is ready, we can call it via curl from a remote host.

As you can see, this time we access the Admin API by sending our request to Kong’s proxy port on its LAN address. So our request hits to Kong proxy and it returned response by fetching from the admin-api upstream (http://localhost:8001)

Step 3 — Apply Authentication Control to admin-api Service

At this point, we will apply an authentication control to our service, in order to deny any unauthorized accesses.

Though, Kong has a bunch of security related plugins that we mentioned above, I’ll use basic authentication for demonstration purpose. To see other options please take a look at https://docs.konghq.com/hub/#security

Step 3.1 — Enable basic-auth Plugin

This plugin adds basic authentication to a service or a route with username and password protection. You can enable it for each services or routes individually. As we want to protect the Admin API, we enable it for admin-api service:

Notice that we define our service name (admin-api) in the URL and this will enable basic-auth plugin for service called admin-api.

Step 3.2 — Create a consumer

Consumers are associated to individuals that uses services. You can think them as users. Now, we create a consumer called my-user for our admin-api service:

Note: consumer_id is a field for storing an existing unique ID for the consumer.

Next we create s credential for our consumer:

As you can see, our username is my-user and password is my-password

Step 3.3 — Using the credential

OK we got a consumer and its credentials, so we can call admin-api service with our username and password pair.

To do this, we need to use Authorization (or Proxy-Authorization) header in our request. This authorization header must be base64 encoded version of username:password

So, we can first encode our credentials, and then send request with the proper header to kong node.

As we enable basic-auth plugin on admin-api service, any unauthorized access will be denied; for example if you don’t define the auth header on your request, you’ll be rejected with the message below :

Well, at this point we added basic-auth for the Admin Api. Though, it can be still accessible via localhost:8001(or :8444) on Kong nodes, you need to provide your credentials to interact from a remote location.

Wrapping Up

There are many other things that we can talk about, however this quick start guide will be useful at the basics I think. If you want to get more information, you can check the docs at https://docs.konghq.com/

Also, in my next article, I’ll write about Konga -which is a nice user interface for Kong-. So stay tuned.

Kong Api Gateway: Installing, Configuring and Securing. was originally published in Devops & AI Türkiye☁️ 🐧 🐳 ☸️ on Medium, where people are continuing the conversation by highlighting and responding to this story.

So here’s the details:

1. What is Ingress ?

As it explained in kubernetes documentation, Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

In other words when you want to expose your applications that reside in the k8s cluster, you configure an ingress resource that defines which inbound traffic will be go to which application.

A typical flow of an ingress is as below:

As you can see, it’s a load balancer that routes specific traffics to the specific targets in an automated fashion. You just create an ingress resource with desired definitions, it does its magic for you.

In this way you can expose as many services as you want with the same IP address. Since, this is a L7 load balancer, it supports host based (ie: app1.domain.com) or path based (i.e: /basket) redirections and SSL termination etc.

Note: Ingress is not the only option to expose applications to outside of the cluster. But since, this is not a kubernetes networking post, I won’t cover everything in here. If you want to learn more about this topic, reading https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0 will be a good start.

A form of an ingress resource manifest is something like below:

In this example, any http requests that contain cafe.example.com as hostname will be redirected two separated service, based on their paths. While, http://cafe.example.com/tea will be routed to tea-svc service, http://cafe.example.com/coffe will be hit to coffe-svc.

2. What is Ingress Controller ?

As its name implies, ingress controller is a component that responsible to manage ingress resources. In other word, if you want your ingress resources to work, you need to have an ingress controller on your cluster.

The most cloud provider kubernetes services, have an implementation of an ingress-controller based on their load balancer solution default. For example when you run your cluster on Google Kubernetes Engine (GKE), the default ingress controller manages Google HTTP/S Load Balancers. Whenever you create an ingress resource, GKE ingress controller spins up a GLB to handle the routes that defined in the ingress resource. So if you are on a cloud provider, you probably don’t need to think about it; but if your environment is on-premise, then you have to implement an ingress controller for your cluster.

Though, there are many ingress controllers out there, one of the most popular ones is nginx ingress controller imho. So we’ll integrate it in this post.

3. Nginx Ingress Controller

Basically, nginx ingress controller manages a bunch of nginx pods in order to do automatic nginx configurations that defined by ingress resources. When you create an ingress, the controller translates the ingress definitions as nginx configuration parameters and applies them to the nginx pods. So it actually does automatic nginx configurations based on your ingress resources.

There are two different versions of the controller. One is maintained by the kubernetes community and published from a github repo. (its documentation sits on https://kubernetes.github.io/ingress-nginx/). The other one is maintained by the Nginx community that can be reached from https://github.com/nginxinc/kubernetes-ingress.

Although, we’ll use kubernetes version of the controller in this story, you can use any of them based on your requirements. If you want to take a look at their differences, just check the comparison document.

3.1. How it works ?

To setup nginx ingress controller on your cluster, all you need is to run a k8s manifest provided by the project. So, for a classic integration with very default configurations you can run the manifest published at https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

This manifest creates a namespace for ingress controller resources, then creates some configmaps to hold default nginx configurations, a service account and related RBAC resources and a deployment to run nginx pods, also it creates a service for nginx deployment to route traffic vianodePort.

Actually, this setup is OK and there is nothing wrong with it. But if we use a DaemonSet instead, then we don’t have to use an additional nodePort typed service, since we can use hostPort parameter with a DaemonSet resource. In order to do this, I’ve just cloned the official manifests to my bitbucket repository and converted it to use DaemonSet.

If you check the related manifest, you can see our definition:

As you know, a DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected.

Also in the ports section we define that we want to use hostPort, so that the pods will be reachable via the node’s IPs. In other words, our nginx pods will be accessible via nodeIP:80 or nodeIP:443 notation.

Note: Again, if you don’t know much about nodePort and hostPort differences, please check https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-ingress-when-should-i-use-what-922f010849e0

3.2 Integration

OK lets create the necessary resources to integrate the controller in our cluster.

First, clone the repo:

git clone https://bitbucket.org/secopstech/nginx-ingress-controller.git

Then, apply the manifests that sit in the configs directory:

kubectl apply -f nginx-ingress-controller/configs/

After few seconds, all resources needed by ingress-controller should be created in ingress-nginx namespace.

To check them, run:

kubectl get all -n ingress-nginx

Output should be something like:

As you see, we got 6 nginx ingress controller pods (because my test environment has 6 nodes in total). Also there is another pod to be used as default backend which is a service that handles all URL paths and hosts the nginx controller doesn’t understand (i.e., all the requests that are not mapped with an Ingress).

3.3 Test the controller

In order to check the controller functionality, we create a test application which resides in test-app directory of the repo. This folder has some manifests to deploy a simple web service with an ingress definition.

Lets look at this example ingress manifest:

This will create two ingress resources to handle routes for web.domain.com and api.domain.com requests. It also provides SSL termination support for our services (with a self signed certificate for *.domain.com)

Now, create the resource by using kubectl:

kubectl apply -f nginx-ingress-controller/test-app/

And check its status:

So we got two deployments, services and other related resources for our test-app. If your nginx ingress controller is up and running, you can now access your web and api pods via https://api.domain.com and https://web.domain.com URLs. (Note that, our SSL terminations use a self signed certificate, so your browser gonna complain about it which you can ignore.

So, ingress-controller setup is finished. If you want to delete test-app deployment, just delete the ingress-test namespace with kubectl delete ns test-ingress

4. Considerations

In this section, I want to point out some production considerations.

4.1 DaemonSet with nodeSelector

In our example, we use a DaemonSet for nginx-ingress-controller, thus our all nodes run a nginx-ingress-controller pod. As I said before, with this setup, our ingress resources would be accessible from all node’s IP addresses.

For example my k8s node’s ip addresses are 10.10.10.1{1,2,3,4,5,6} and api.domain.com or web.domain.com services are reachable from all of these IPs. So, I can use an external load balancer for this domains and distribute the traffic to all of these 6 nodes.

If you have few k8s nodes, it won’t be a problem, but as your cluster grows over time, load balancer management and compute resource planning might become more difficult. (You probably don’t want to run dozens of nginx ingress controller pods in total.) So, at this point using DaemonSet with “one pod per node” logic might not be a suitable way to run an ingress-controller.

In this situation, you can consider to run your pods on specific nodes by using nodeSelector parameter in your DaemonSet. For example, if we want to run our nginx pods on our three nodes only; we can label these nodes and tell DaemonSet to run the pods on the nodes which have the label.

To do this, add a label to your nodes:

kubectl label nodes node1 ingress-controller-node=true
kubectl label nodes node2 ingress-controller-node=true
kubectl label nodes node3 ingress-controller-node=true

Then change the DaemonSet manifest spec something like below:

As you noticed, we add the label as nodeSelector to container spec section. After applying this manifest, nginx pods run on our first three nodes.

4.2 Nginx Custom Configurations

When a nginx pod is starting, its configuration file is injected through a configmap; and the default configmap provided by the project is something like below:

As you can see, there is no any nginx configuration parameters in it. While, nginx pods have a default nginx configuration that comes from the image, sometimes you might want to adjust the configuration level by your needs. To do this, you can add your specific parameters within the data section:

In this example, we set some proxy related config parameters our ConfigMap. So whenever a nginx pod starts, it will consider these parameters as well.

Note: The above parameters are published as an example and they are not for production use.

All supported parameters are documented on https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/

Also, if you want to configure an ingress resource individually, you can use Annotations in ingress resource manifests as explained on https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

Kubernetes Nginx Ingress Controller for On-Premise Environments was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

As you know, automation is one of the most important aspect of the DevOPS culture. The time goes by, IT environments converted to more and more complex things and now we have more servers and network equipments that we need to manage. More importantly we need to orchestrate them without using much manpower. This is a “must have” approach to adopt DevOPS culture!

IMHO, DevOPS culture is a way to embrace “work smart, not hard” philosophy. It’s a mindset to design robust, agile and simple IT environments that produce much output with less input and a way to gaining full control over them with minimal effort!

This is why I like the clever tools that can be used to achieve to build or manage such an environment. No doubt, one of them is ansible. It’s a simple and agentless automation tool that comes with a comfortable UI like Ansible AWX (or Tower). It also has, less learning curve than others.

Anyway, Let’s focus on how to install Ansible AWX via docker-compose.

Ansible AWX

Actually AWX is an open-source version of Ansible Tower which provide a web interface to manage environments via ansible.

Here’s a quick explanation from AWX project FAQ:

The AWX Project — AWX for short — is an open source community project, sponsored by Red Hat, that enables users to better control their Ansible project use in IT environments. AWX is the upstream project from which the Red Hat Ansible Tower offering is ultimately derived.

So, if you’re following ansible’s path to manage your environments, using such a UI might be useful. Because it has some good features which mentioned in the Tower website:

Red Hat® Ansible® Tower helps you scale IT automation, manage complex deployments and speed productivity. Centralize and control your IT infrastructure with a visual dashboard, role-based access control, job scheduling, integrated notifications and graphical inventory management. And Ansible Tower’s REST API and CLI make it easy to embed Ansible Tower into existing tools and processes.

In particular, job scheduling, role-based access controlling and its REST-API, make it worth using.

AWX Installation

Basically, there are some options for deployment. It can be installed on docker (via docker-compose), kubernetes or openshift. But for the sake of the simplicity I’ll document the installation based on docker on a latest CentOS 7 host.

Prerequisites:

In order to deploy AWX on a CentOS box, you need to have at least 4GB memory and 20GB disk space. If you decided to use AWX on production, you need to take a look Tower’s requirements page for more information.

First of all let’s install docker engine and docker-compose from docker repo.

Download

Now, download the latest AWX release tarball to the home directory and extract it.

Initial Configurations

The tarball includes bunch of ansible playbooks and roles to install AWX with an inventory file that holds default configuration variables. We are going to change the inventory file (located at installer/invetory) in order to deploy AWX with additional configurations like https support, official logos etc.

By default, installer deploys AWX via docker-compose and uses official awx docker images which published on dockerhub to creates postgresql, rabbitmq, memcached, and a nginx based frontend container that required by AWX.

But if you want to do some additional configurations like https support; you need to build your local awx images by comment out dockerhub_base=ansible line as it says in the official documentation:

Remove these lines if you want to run a local image build
Otherwise the setup playbook will install the official Ansible images. Versions may be selected based on: latest, 1, 1.0, 1.0.0, 1.0.0.123
by default the base will be used to search for ansible/awx_web and ansible/awx_task

So let’s comment out that line:

Note: As of AWX version 6.1.0, the only way to enable https support is building local images. So, if you want to use official awx images from dockerhub, you SSL setup that we cover later in this post, cannot be done!

As mentioned above, AWX requires a postgreSQL database and installer will automatically create a psql container for it. But in order to keep data in a persistent location, we need to create a folder to hold the db files in the host system and tell its path to the installer via postgres_data_dir variable.

So I’ll create a folder named awx-psql-data in /opt directory and define it in the inventory.

Note: If you wish to use an external database, in the inventory file, set the value of pg_hostname, and update pg_username, pg_password, pg_database, and pg_port with the connection information.

Next parameters we’ll change is ssl_certificate which tells the installer enable https support for the frontend with the provided ssl key pair. (By default it only supports http.). With this parameter, we need to provide “a file” that should includes both a SSL certificate and its private key.

In this step, I’ll create a self-signed SSL located at /etc/awx-ssl/ folder and merge the .key and .crt files in another file at /etc/awx-ssl/awx-bundled-key.crt. Then I’ll pass the file path to ssl_certificate variable:

Next changes is use of eye-candy AWX logos from awx-logos repository. I think the default logo that comes with AWX itself is horrible, so changing it might be good choice (believe me :)

In order to replace the default logo, we need to get awx-logos repository and place it to next to your awx folder (awx-6.1.0 in our example):

So our, working directory should be look something like below:

drwxrwxr-x 9 root root 4096 Jul 18 17:08 awx-6.1.0
drwxrwxr-x 3 root root 55 Sep 1 2017 awx-logos

Note: AWX installer which resides at installer/install.yml, searches for “awx-logos” directory as “../../awx-logos”. So, your awx-logos folder should be located at where awx installer’s parent directory is placed.

After put the logo directory the correct location, we need to set awx_official parameter as true:

Next, we need to change admin_userand admin_passwordparameters. By default, installer creates a super user and you can access the AWX ui with “admin/password” credentials. However changing the defaults is a good habit.

So let’s change them as user: awx-admin and pass: CHANGE_ME

Note: Set the credentials as your standards. Also there are some other credential parameters like secret_key and rabbitmq_password etc. But I don’t cover all of them in this post. If you want to change them as well, you can take a look the inventory file since the comment lines explain everything nicely.

Installation

As I said before, AWX comes with a installer ansible playbooks/roles which makes the whole installation process simple. When you done with your configuration changes in the inventory file, you can just run the installer playbook like below:

Based on compute power of your host, installation process takes a while (Since, we build local docker images for awx). But if everything goes well it should be completed successfully and you can access to AWX UI by via https://your-hosts-ip/

After initial setup, AWX upgrades itself; and when you visit the UI, you see that notice.

And after a while, login screen appears.

And AWX is ready to use:

As you noticed, there is a demo project already deployed and you can examine it as a good usage reference. But, in the next post, I’m gonna publish a document about possible usage scenarios and best practices too.

Stay tuned.

Ansible AWX Installation was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

Zabbix v4 and Grafana Installation Script for CentOS 7 was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

So you can easily build a kubernetes cluster on many cloud platforms; but what if you need a production ready, fully redundant cluster in your on-premise environment ?

In this situation, there are bunch of solutions that can be used to build such a cluster and I personally prefer to use kubespray.

Kubespray is a project that contains a bunch of ansible playbooks which deploy kubernetes clusters in an automated way. It is very suitable installation, configuration and maintaining method for on-premise environments since you can use it to deploy kubernetes clusters on different linux distros like, Ubuntu, Coreos and Centos.

So, in this document I want to explain how to install, configure and maintain a CentOS7 based kubernetes cluster with kubespray.

Requirements

In order to deploy a fully redundant k8s cluster, you should ideally have at least nine hosts, which three hosts for masters, three for workers and three for etcd cluster. However, in the sake of the simplicity we are going to use only three nodes which all components will be installed on these three nodes, so each node will have master, worker and etcd roles.

But again, don’t forger that, if your cluster will be used as a production environment that run a critical workload, all k8s components should be installed on a separated host groups.

Also, as we now kubespray uses ansible, there should be a host with ansible installation that holds the playbooks and configurations which responsible for running playbooks against k8s-nodes over SSH to deploy the cluster. This host can be a separated or you can use one of the k8s-nodes for this purpose. In our example we’ll use three hosts named k8s-host-0{1,2,3} and ansible host will be k8s-host-01

There are two main preparation part of this document:

• Ansible host preperations
• k8s nodes preperations.

After preperation steps we’ll initiate kubespray cluster.yml playbook to deploy our first cluster.

So let’s get started with ansible preperation tasks.

Preparing ansible

As we’ve mentioned above our ansible host will be k8s-host-01 and in this section all necessary action steps will be done in this host.

First of all disable selinux and firewalld:

Then install required packages by kubespray:

Now we’ll create an user named kubespray and add to sudo (as NOPASSWD). Actually this is not a “must have” situation but I think isolating the kubespray from the root useris a good practise.

Switch to kubespray and create an SSH key pair. We’ll use this keypair to establish passwordless connections to k8s nodes.

Get the latest kubespray release from https://github.com/kubernetes-incubator/kubespray/releases and extract it. (As the date of this document has written/updated the latest version was 2.13.1)

Now we install required python packages by using pip

Next steps are related to cluster configurations and there are some variables which shoud be defined correctly in you environment.

By default, when you initiate a cluster deployment via kubespray, it sets the cluster name as cluster.local, we’ll change it via environment variables. So please change $CLUSTER_NAME and $CLUSTER_DOMAIN variables as your needs.

Create a directory that presents your cluster name. This directory will contains cluster specific ansible inventory and variable files and we get those files by copying them from the sample directory.

Now our ansible environment is ready to be configured as our specifis needs. There are two main configuration files named all.yml and k8s-cluster.yml and both files is placed in “inventory/$CLUSTER_NAME/group_vars/” directory.

Let’s change the cluster name first.

And set token auth modes true:

I prefer to use weave as CNI (for further reading, here is a good comparison https://chrislovecnm.com/kubernetes/cni/choosing-a-cni-provider/) so we set the kube_network_plugin variable as weave and define a password for encryption.

also kube_read_only_port variable should be enabled in order to metrics-server work.

rename the hosts.ini file as inventory.cfg

inventory.cfg file is main configuration file that defines the k8s host roles. So you need to be set as properly. In our example we have three nodes named k8s-host-0{1,2,3} so our inventory.cfg should look like below:

Note: You need to also set the bastion and ip variable options if your network requires it. And there is an another section related to nginx ingress controller. But we’ll deploy ingress controller separately on the next post (in a more customized way) we don’t use the ingress section of inventory so simply delete it.

A few notes about connectivity:
- Your nodes should be resolve each other hostname. So create proper DNS records or add them to the hosts file.

- Since we use a custom user for SSH connections, you need to set kubespray as remote_user in ansible.cfg file’s [default] section as below:

Not: If you use a different port for sshd, you also need to change remote_port definition.

Ansible host preparation is done and now we will configure our k8s nodes.

Preparing k8s nodes

These steps should be done on all k8s nodes.

Disable selinux and firewalld:

Swap should be disabled on all nodes otherwise kubespray will fail. So let’s disable it.

And add kubespray user and setup sudoers for it.

Now you need to distribute the ansible host’s (k8s-host-01) pubkey to the k8s nodes’ kubespray user as authorized key. And you need to also put the pubkey to host-01’s authorized_keys file, since ansible will make a ssh connection for itself too by using its primary ip address.

Deploy kubernetes cluster from the ansible instance

OK, we are ready to deploy cluster. It’s a very simple step, only run cluster.yml playbook as follows:

After deployment process the cluster should be installed properly.

Accessing kubernetes API from k8s-node with kubectl

By default kubernetes API can be accessible via kubectl from any k8s host.

Let’s check it on host-01:

As you can see our k8s cluster is ready and all nodes act as master and worker (also etcd host).

Accessing kubernetes API from a remote kubectl

When you initiate cluster deployment, by default kubespray adds an admin user to the cluster named “kube”. And its password written down to “inventory/$CLUSTER_NAME/credentials/kube_user.creds” file.

Although we’ll create a user and join it to a new admin group in order to do more granular access control (also this will interact to the kubernetes API by using its PKI key pair instead of basic auth.) on the next section, if you still want to use the default user to connect kubernetes you can follow the steps below:

Create a kubeconfig file on your local machine.

Note: Your local machine should meets the requirements listed below:

• Able to access to kubernetes API on https://kubernetes.default.svc.$CLUSTER_FQDN:6443
• Masters tcp port 6443 needs to be accessible from your client.
• $CLUSTER_FQDN resolves to one of the master’s primary IP addresses (or you can load balance them).

And paste the snippet:

And change the variables listed below:

• $CLUSTER_FQDN: should be changed as your cluster FQDN
• certificate-authority-data: It has to contains /etc/kubernetes/ssl/ca.crt information as one-line base64 encoded format. To grab it as required format, you can run something like this in one of your master node:
  cat /etc/kubernetes/ssl/ca.crt |base64 -w 0 then copy the output and paste it to $CLUSTER_CA_CRT_BASE64_IN_ONE_LINE section.
• password: This needs to be changed with the password that stored in “inventory/$CLUSTER_NAME/credentials/kube_user.creds” file.

If you prepare the kubeconfig properly, you can set this file as kubectl config via KUBECONFIG variable:

Now you can test your connectivity like below:

Well, our cluster is formed and we have connected it by using the basic authentication method. Now we’ll do some customization like creating a new admin user which will use PKI instead of basic auth as mentioned above.

Additional Configurations

In this chapter we’ll cover the additional configuration steps listed below:

• Create an admin group and an user
• Dashboard Settings
• Metrics Server Deployment
• Nginx Ingress Controller Deployment

Create an user with administrator privileges

This section consists of four phases:

1. Create a “ClusterRole” for the custom admin group
2. Create a “ClusterRoleBinding” for our newly created “ClusterRole”
3. Create an user certificate for kubectl
4. kubeconfig configuration

1. Create a ClusterRole

First of all we’re going to create a cluster wide role with full permission to any api group. In our example the ClusterRole name is cluster-admins-cr. So each binding member to this ClusterRole, can be able to access any kubernetes API group.

On one of your master hosts, run:

2. Create a ClusterRoleBinding

Next, we create a “ClusterRoleBinding” to give access “ClusterRole” named cluster-admins-cr for members of cluster-admins group.

This creates a ClusterRoleBinding named cluster-admins-crb, binds it to cluster-admins-cr ClusterRole and permits ClusterRole’s privileges (which is permitted for all API group calls) for each users who provide a certificate that issued by this kubernetes cluster CA. Also as we’ll see the next section, any user certificate organization (O) should be defined as the group name to access the cluster with admin privileges.

3. Crate an user certificate

When you create a user certificate, there are two important variable that you need to define before:

• $USERNAME: This is the username to be created
• $GROUP_NAME: This is the group name which we’ll be bind our admin “ClusterRole” that we’ve created the last section. Our example is “cluster-admins”

So let’s define them first as environment variables:

Now we ready to issue for our user by using our cluster’s CA. We’ll run our commands as root and create a hidden folder in root’s home directory to place the generated certificates which should be keep as secure as possible.

Our user certificate should be issued, let’s check it:

Now we can configure our kubeconfig by using these PKI files.

4. kubeconfig Configuration

Create a kubeconfig file on your local machine and populate it as follows and change the variables as your setup.

In order to set kubeconfig properly these variables should be defined:

• $CLUSTER_CA_CRT_CONTENT:
  This is your kubernetes cluster CA public key file which places in /etc/kubernetes/ssl/ca.crt on any master hosts. It have to be presented in one-line base64 encoded format in kubeconfig. To grab it in correct format run: # cat /etc/kubernetes/ssl/ca.crt |base64 -w 0 and copy the output to paste it as certificate-authority-data value.
• CLUSTER_FQDN:
  This is our clusters name variable which we’ve defined in installation section.
• USERNAME: In our example this is my-admins-username
• USERNAME.CRT_CONTENT: This is our user’s certificate content and its should be as one line base64 encoded format too. You can convert it as follow: # cat ~/.k8s-user-certs/my-admins-username.crt |base64 -w 0 and use it as client-certificate-data
• USERNAME.KEY_CONTENT: This is our user’s private key content and also its must be as one line base64 encoded format:
  # cat ~/.k8s-user-certs/my-admin-user.key |base64 -w 0 copy and paste it as client-key-data

If you populate the kubeconfig you can set this file as kubectl config via KUBECONFIG variable:

Now test your connectivity:

Kubernetes Dashboard Settings

Kubespray installs the dashboard by default (since k8s 1.7) and you can authenticate by using the default kube admin user. However we will create a service account with admin privileges and use its token to access the dashboard.

Create an Service account named dashboard-admin:

Then create an ClusterRoleBinding for the dashboard-admin ServiceAccount and assign it to cluster-admin ClusterRole.

Now you can extract the bearer token from the dashboard-admin ServiceAccount to use it on the dashboard authentication.

Now run kubectl proxy command then visit http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

In the login section you can select token based authentication option and provide your bearer-token to login.

Metrics Server Deployment

Please see:

Kubernetes metrics-server Installation

Nginx Ingress Controller Deployment

Nginx controller deployment post will be published soon.

Production Ready Kubernetes Installation with kubespray. was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, if you want to use k8s’ features like horizontal pod autoscaler which was depended on heapster (or even to be able to use kubectl top command) you need to use metrics-server instead.

What is metrics-server ?

Metrics Server is a cluster-wide aggregator of resource usage data. It collects metrics like CPU or memory consumption for containers or nodes, from the Summary API, exposed by Kubelet on each node. If your kubernetes is formed by kube-up.sh script then you probably have a metrics-server by default. But if you use kubespray or kops to build a production ready k8s cluster, then you need to deploy separately.

How to deploy ?

Though, I prefer to use kubespray and this document is based on k8s clusters which deployed by kubespray, installation steps are same all other solutions like kops etc.

Requirements

In order to deploy metrics-server, aggregation layer should be enabled in your k8s cluster. As of kubespray version 2.8.3, aggregation layer is already up by default. If you need to enable it especially see the link below. https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/

Deployment

Clone metrics-server git repo from https://github.com/kubernetes-incubator/metrics-server and deploy it as follow:

git clone https://github.com/kubernetes-incubator/metrics-server.git
cd metrics-server
kubectl apply -f deploy/kubernetes/

kubectl apply step sets an apiservice named v1beta1.metrics.k8s.io, creates a deployment named metrics-server and configures a service for the deployment.

If everything gone well, you can check the apiservice like below:

$ kubectl get apiservices |egrep metrics
v1beta1.metrics.k8s.io                  kube-system/metrics-server   True        11h

Also, there should be a deployment and a service named metrics-server:

$ kubectl get deploy,svc -n kube-system |egrep metrics-server

deployment.extensions/metrics-server 1 1 1 1 11h
service/metrics-server ClusterIP 10.233.31.135 <none> 443/TCP 11h

And if you run kubectl top nodecommand you should see node utilizations:

$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s012-node001 667m 8% 2683Mi 17%
k8s012-node002 828m 10% 2489Mi 16%
k8s012-node003 431m 5% 2424Mi 15%

Lastly you can call the apiservice via kubectl; for example this should return basic node metrics:

$ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" |jq .
{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": [
    {
      "metadata": {
        "name": "k8s012-node001",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s002-node001",
        "creationTimestamp": "2019-03-17T20:01:12Z"
      },
      "timestamp": "2019-03-17T20:01:10Z",
      "window": "30s",
      "usage": {
        "cpu": "1118046599n",
        "memory": "2748964Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s012-node002",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s002-node002",
        "creationTimestamp": "2019-03-17T20:01:12Z"
      },
      "timestamp": "2019-03-17T20:01:02Z",
      "window": "30s",
      "usage": {
        "cpu": "483888352n",
        "memory": "2545964Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s012-node003",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s002-node003",
        "creationTimestamp": "2019-03-17T20:01:12Z"
      },
      "timestamp": "2019-03-17T20:01:02Z",
      "window": "30s",
      "usage": {
        "cpu": "454428677n",
        "memory": "2481852Ki"
      }
    }
  ]
}

Not: jq is for good looking json output.

Troubleshooting

If kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" command returns empty response and metrics-server pod throw an error like

E0903  1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:<hostname>: unable to fetch metrics from Kubelet <hostname> (<hostname>): Get https://<hostname>:10250/stats/summary/: dial tcp: lookup <hostname> on 10.96.0.10:53: no such host

This error is related to a known issue as of metrics-server v0.3.1 which reported here https://github.com/kubernetes-incubator/metrics-server/issues/131

To fix the issue, you need to edit metrics-server-deployment.yaml and add the parameters below right after image: k8s.gcr.io/metrics-server-amd64:v0.3.1 line:

command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP

then re-apply it as $ kubectl apply -f metrics-server-deployment.yaml

After few seconds, you can get metrics via kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" or kubectl top nodecommands.