The Open Source Routing Machine (OSRM) is an open-source router designed for use with data from the OpenStreetMap project. The following services are available via HTTP API, C++ library interface and NodeJs wrapper:

• Nearest — Snaps coordinates to the street network and returns the nearest matches
• Route — Finds the fastest route between coordinates
• Table — Computes the duration or distances of the fastest route between all pairs of supplied coordinates
• Match — Snaps noisy GPS traces to the road network in the most plausible way
• Trip — Solves the Traveling Salesman Problem using a greedy heuristic
• Tile — Generates Mapbox Vector Tiles with internal routing metadata

Why OpenStreetMap over Google Maps ?

Open Street Maps (OSM) is an open source project maintained by the OpenStreetMap Community that provides free editable maps of the whole world. The most obvious advantage in using OpenStreetMap-based software over a commercial solution is economical convenience, because OpenStreetMap comes as free software.

The downside is that it takes a little configuration in order to setup a working web service. The easiest and quickest way to setup your own routing engine is to use Docker images

There are two pre-processing pipelines available:

• Contraction Hierarchies (CH)
• Multi-Level Dijkstra (MLD)

We recommend using MLD by default except for special use-cases such as very large distance matrices where CH is still a better fit for the time being. In the following we explain the MLD pipeline. If you want to use the CH pipeline instead replace osrm-partition and osrm-customize with a single osrm-contract and change the algorithm option for osrm-routed to --algorithm ch.

Generate data using Docker:

Our web service will be based on a map export from OpenStreetMap. There are several possible ways to obtain a map export.

The first option is to point your browser to the OpenStreetMap’s export page, zoom on the interested area, and click on Export. (You may need to use one of the other sources if the native export does not work; Overpass API is a good choice.) This will allow you to finely choose which areas to include in your map and, in general, to cut down on the pre-processing times.

https://www.openstreetmap.org/export#map=6/16.225/73.059

However, there are limits on the size of the export you can get from the OSRM website, so you might want to download a ready-made export. A number of services, such as Geofabrik and Mapzen offer updated and ready-made map exports of countries and selected metropolitan areas, which are good for most use cases.

In this article we will see how the OSRM data is generated for India using MLD pipeline.

To generate the routing data you need to download the .pbf file for India from Geofabrik.

wget http://download.geofabrik.de/asia/india-latest.osm.pbf

OSM data comes in a variety of formats including XML and PBF, and contain a plethora of data. The data includes information which is irrelevant to routing, such as positions of public waste baskets. Also, the data does not conform to a hard standard and important information can be described in various ways. Thus it is necessary to extract the routing data into a normalized format. This is done by the OSRM tool named extractor.

Profiles are used during this process to determine what can be routed along, and what cannot (private roads, barriers etc.).

For the MLD pipeline we need to extract (osrm-extract) a graph out of the OpenStreetMap base map

docker run -t -v $(pwd):/data osrm/osrm-backend:latest osrm-extract -p /opt/car.lua /data/india-latest.osm.pbf

The flag -v $(pwd):/data creates the directory /data inside the docker container and makes the current working directory $(pwd) available there. The file /data/india-latest.osm.pbf inside the container is referring to $(pwd)/india-latest.osm.pbf on the host

Then partition (osrm-partition) this graph recursively into cells, by using the following command

docker run -t -v $(pwd):/data osrm/osrm-backend:latest osrm-partition /data/india-latest.osrm

Note that india-latest.osrm has a different file extension.

Once the graph is partitioned in to cells, customize the cells (osrm-customize) by calculating routing weights for all cells

docker run -t -v $(pwd):/data osrm/osrm-backend:latest osrm-customize /data/india-latest.osrm

For CH the partition and customize pipeline stage gets replaced by adding shortcuts from the Contraction Hierarchies algorithm (osrm-contract)

Once the graph is customized, then spawning up the development HTTP server (osrm-routed) responding to queries

OSRM Docker Deployment

Docker image can be run standalone without Kubernetes

docker run -t -i -p 5000:5000 -v $(pwd):/data -d osrm/osrm-backend:latest osrm-routed — algorithm mld /data/india-latest.osrm

Make requests against the HTTP server

Optionally start a user-friendly frontend on port 9966, and open it up in your browser

docker run -p 9966:9966 osrm/osrm-frontend

OSRM Kubernetes Deployment

To deploy the OSRM in Kubernetes cluster, we need the routing data inside the Kubernetes cluster. There are multiple ways to achieve this. The reliable way is to use persistent volume model in a dynamic Kubernetes cluster. On a standalone k8s cluster the simple and easiest way is to use the hostPath.

To deploy on standalone cluster once the data generation is completed, copy all the generated files to the following location /data/osrm/india on the server. Make sure this data is available on all the Kubernetes nodes.

Now use the below mentioned YAML file to deploy the osrm-backend application in k8s.

osrm-backend-india.yaml

apiVersion: v1
kind: Service
metadata:
  name: osrm-india
  labels:
    app: osrm-india
spec:
  ports:
  - port: 5000
    targetPort: 5000
    name: http
  selector:
    app: osrm-india
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: osrm-india
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: osrm-india
    spec:
      containers:
      - name: osrm-india
        image: osrm-backend:latest
        command: ["/bin/sh", "-c"]
        args: ["osrm-routed --algorithm mld /data/india-latest.osm.pbf"]
        ports:
        - containerPort: 5000
          name: osrm-port
        volumeMounts:
        - name: india
          readOnly:  true
          mountPath: /data
      volumes:
      - name: india
        hostPath:
          path: /data/osrm/india
        resources:
          requests:
            memory: "512Mi"
            cpu: "0.5"
          limits:
            memory: "512Mi"
            cpu: "0.5"

Below config make sure the data is mounted to the docker container while deploying

        volumeMounts:
        - name: india
          readOnly:  true
          mountPath: /data
      volumes:
      - name: india
        hostPath:
          path: /data/osrm/india

Command to run the osrm-backend image in Kubernetes

kubectl apply -f osrm-india.yaml

To deploy the user-facing frontend with map, apply the below file

osrm-frontend.yaml

kind: Service
metadata:
  name: osrm-frontend
  labels:
    app: osrm-frontend
spec:
  ports:
  - port: 9966
    targetPort: 9966
    name: http
  selector:
    app: osrm-frontend
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: osrm-frontend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: osrm-frontend
    spec:
      containers:
      - name: osrm-frontend
        image: osrm/osrm-frontend
        ports:
        - containerPort: 9966
          name: osrm-frontend
        env:
        - name: BACKEND
          value: 'http://osrm-india:5000'

Command to run the osrm-backend image in Kubernetes

kubectl apply -f osrm-frontend.yaml

Note:

OSRM data can be generated for any countries. The data generation consumes lot of memory on your server based on the country . To generate OSRM data for India, docker runs consumes almost 7 GB of RAM on the server.

Conclusion

This blog covers the configuration and maintenance of a web service which can answer questions such as:

• What is the closest street to a given pair of coordinates?
• What’s the best way to get from point A to point B?
• How long does it take to get from point A to point B with a car, or by foot?

References

• Geofabrik Download Server
• BBBike extracts OpenStreetMap

OSRM data generation and deployment was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

I was quite fascinated by a simple principle defined by Andrew Hunt and David Thomas in their Pragmatic programmer book derived from the “Broken window theory” in programming. This book was originally published in 1999, but the principles apply to this date. Freakanomics is where I have originally encountered this theory, some years ago. I was probably more interested in the economics part of it then. It is quite brilliant in my view to apply this to software development.

The crux of the broken window theory in it’s original context was that

Disorder causes crime, Crime causes further disorder and crime.

It is not hard to imagine what this would mean in the software world. Imagine a project where there is large code base but has limited tests, when a piece of code is submitted, It is unlikely that the new code being submitted has tests to cover. Compare this to a codebase in the same project and has good unit test coverage, would the same developer submit code without tests? Extremely unlikely. We can extend this to bugs, technical debt, design, quality of pull requests, documentation and everything else that is involved in software development.

So, over sufficient time in projects that are not managed well, a lot of disorder occurs which makes it even more difficult to maintain order later on. Entropy kicks in.

Software entropy

Entropy in physics, defines the amount of disorder in a system. And the amount of disorder tends to reach maximum over time, according to laws of physics. Any software system, if not managed well through the time will tend to “rot”. Bad designs, wrong decisions or just plain badly written code if left unrepaired will cause more disorder and rot.

The most important reason for Software entropy is the culture of the team. It is the mindset of the team, regardless of the size of the team that matters. When the team maintains and fixes the broken pieces at the earliest, the rot naturally reduces. This is precisely the point of the “Broken window theory”, isn’t it?

Don’t live with broken windows

Good teams strive to fix broken windows at the earliest, and will continue to maintain the hygiene of the software. If not possible to fix and some times it really may not possible, at least have a plan and board it up temporarily. But again, the culture if the team tends to frequently resort to boarding up temporarily, then it would end up in the same rot again.

What do you do about existing broken windows?

Let’s say hypothetically (most possibly practically), we are working on a project which has lot of broken windows, what should we do now. One succesful strategy is to aggressively identify all broken windows, board them up, plan “Refactoring”, talk to your boss, the benefits of fixing it have to outweigh. I am not going to talk about how you convince your team, but usually benefits outweigh the cost of effort in long run and you can negotiate a time in upcoming sprint/release for sure.

Go ahead and chalk out a plan to aggressively fix those broken windows now.

Due credit has to go to the authors of this book for this simple and effective insight. If you have not read The Pragmatic programmer book, it is never a bad time to learn from the best.

Programming and Broken windows was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Monitoring metrics and runtime characteristics of an application server is essential to ensure the adequate functioning of the applications running on that server, as well as to prevent or resolve potential issues in a timely manner. As far as Java applications go, one of the most commonly used web application servers is Apache Tomcat, which will be the focus of this article.

Tomcat performance monitoring in Kubernetes cluster can be done either by relying on JMX beans or other tools available in the market. We have listed the prerequisite with which this monitoring can be achieved.

1. Prometheus Agent
2. Kubernetes YAML for application
3. Annotations
4. Scraping Policy
5. Prometheus Server

Prometheus Agent

Prometheus provides a collector that can scrape and expose the mBeans of a JMX target. It runs as a Java Agent, exposing an HTTP server and serving metrics of the local JVM. It can also be run as an independent HTTP server and scrape remote JMX targets, but this comes with various disadvantages, such as it being harder to configure and it being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.

The following Dockerfile adds support for the options we’ve listed.

FROM tomcat

RUN mkdir /data

ADD jmx_prometheus_javaagent-0.7-SNAPSHOT.jar /data/jmx_prometheus_javaagent-0.7-SNAPSHOT.jar

ADD prometheus-jmx-config.yaml /data/prometheus-jmx-config.yaml

CMD java -javaagent:/data/jmx_prometheus_javaagent-0.7.jar=8088:/data/prometheus-jmx-config.yaml -jar $TOMCAT_HOME/start.jar

• Connect to an exposed JMX port of the JVM (not recommended)
• Java agent version (recommended)

If you use the agent version, you’ll have to modify three configuration options:

• the Jar file location
• the port for the http(s) interface, where the metrics will be available for scraping, already in Prometheus friendly format
• additional configuration options

An example looks like this:

-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.7.jar=8088

The latest jar can be downloaded from the following link:

wget http://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.7/jmx_prometheus_javaagent-<latest-version>.jar

Create prometheus-jmx-config.yaml with following config in it

---
hostPort: 127.0.0.1:1234
lowercaseOutputName: false
lowercaseOutputLabelNames: false

The complete options for the prometheus-jmx-config.yaml is available here.

Kubernetes YAML for application

When you run the tomcat image in Kubernetes infrastructure, the JMX port needs to be exposed to scrape the data from the container. Following sample tomcat YAML file shows how to deploy the basic tomcat service in K8s infrastructure with prometheus JMX port exposed and annotated.

apiVersion: v1
kind: Service
metadata:
  name: <service name>
  labels:
    app: <service name>
spec:
  ports:
  - port: 8080
    targetPort: 8080
    name: http
  - port: 8088
    targetPort: 8088
    name: prom
  selector:
    app: <service name>
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: <deployment name>
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 1
  template:
    metadata:
      labels:
        app: <deployment name>
        type: java
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8088'
    spec:
      containers:
      - name: <container name>
        image: <docker image>
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_AGENT_PORT
          value: "8088"
        resources:
          requests:
            memory: "1024Mi"
            cpu: "1"
          limits:
            memory: "1024Mi"
            cpu: "1"

Annotations

Annotations on pods allow a fine grained control of the scraping process:

• prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation will exclude the pod from the scraping process.
• prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
• prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports (default is a port-free target if none are declared).

These annotations need to be part of the pod metadata. They will have no effect if set on other objects such as Services or DaemonSets. The above Deployment manifest will instruct Prometheus to scrape all of its pods on port 8088.

Scraping Policy

To minimise the amount of configuration required from the user, the default setup will try to scrape all pods present in the system. This may have some undesirable side-effects such as:

• Prometheus issuing HTTP GET requests to TCP ports not even exposing an HTTP server.
• Prometheus issuing HTTP GET requests to fetch /metrics on services that may not export any metric, polluting logs.

While the annotations described in the previous section allow the user to opt-out of the scraping process, we also offer an alternative Prometheus configuration that only scrapes a minimal set of system metrics and requires the user to opt-in on a per-pod basis. In other words, in this mode the prometheus.io/scrape: true annotation is required for Prometheus to scrape the pod.

Now that we have all the required configs in place to scrape the JVM data from the tomcat container. Let see how it can be scraped via prometheus server.

Prometheus Server

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Please use the following docker-compose.yaml for downloading the Prometheus Image

---
version: '2'
services:
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    restart: always
    ports:
      - "9090:9090"
    volumes:
    - /data/prometheus/prometheus.yml:/prometheus/prometheus.yml:ro
    command: [ '--config.file=/prometheus/prometheus.yml', '--web.enable-lifecycle', '--web.enable-admin-api', '--storage.tsdb.retention=1y' ]

Configuring Prometheus

The Prometheus download comes with a sample configuration in a file called prometheus.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'prometheus
    scrape_interval: 30s
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - staging
    scheme: http
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

There are three blocks of configuration in the example configuration file: global, rule_files, and scrape_configs.

• The global block controls the Prometheus server's global configuration. We have two options present. The first, scrape_interval, controls how often Prometheus will scrape targets. The evaluation_interval option controls how often Prometheus will evaluate rules.
• The rule_files block specifies the location of any rules we want the Prometheus server to load. For now we've got no rules.
• The last block, scrape_configs, controls what resources Prometheus monitors. Since Prometheus also exposes data about itself as an HTTP endpoint it can scrape and monitor its own health. Prometheus expects metrics to be available on targets on a path of /metrics. So this default job is scraping via the URL: http://<hostname>:9090/metrics.

When you scrape data from the containers running inside the Kubernetes cluster, don’t forget to mentions the namespace from where the data should be scraped as mentioned in the above config file.

Once you have all prerequisites configured and deployed, the JVM metrics will be collected and stored in the prometheus. We use Grafana to create a graphical dashboard and measure the performance as shown below.

References

• JVM overview - Prometheus dashboard for Grafana
• GitHub - prometheus/jmx_exporter: A process for collecting metrics using JMX MBeans for Prometheus consumption
• Collect Docker metrics with Prometheus

Tomcat JVM metrics monitoring using Prometheus in Kubernetes was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Logistimo, all of our applications are containerized and run as docker containers inside Kubernetes. We had noticed lot of restarts on containers with Java apps and is quite random. Docker inspection revealed that the pod was killed by OOMKiller code:137. This meant that the application is consuming more memory than allocated to the container. It didn’t sound right, since we have limits on the Java application using -Xmx and we left about 20% buffer as Kubernetes resource limit (docker container) for the Meta space and GC data.

For e.g., 2 GB for Java process, and 2.4 GB for the Kubernetes resource.

Subsequent sections cover this problem and how to solve it in detail.

JVM memory usage

Obviously the first step was to review why the container is exceeding the said limit, clearly these are sufficiently buffered.

“ps” command confirms that the Xmx is indeed in place, and is set to max of 4GB.

“top” command, however reveals that the physical memory used is 4.5 GB.

Why would Java take 500 MB more than allocated?

Since JDK 1.8.40 there is a Native memory tracker tool introduced which provides a detailed breakup of memory used by Java application with every byte accounted for. Note that NMT tool shows committed, resident might be less.

Actual usage = Heap memory + Meta space + Off heap

Off heap typically consists of class meta data, compiled code, threads and GC data. GC data is variable while rest of it should remain static for most applications. This memory is native (yes including the meta space), and JVM uses available memory on host to grow or garbage collect this data.

I would encourage you to read this excellent blog post by Mikhail to get better perspective.

Coming back to the problem at hand, JVM took 500 MB more because the underlying host had 16 GB memoy. At times this number could go higher than the buffer we had set, which would cause the container to be terminated. Shouldn’t the JVM be reading docker container’s memory limit?

Containers and Java

It turns out Java versions 9 and below do not understand containers/dockers at all (by default). It picks up available CPUs and Memory from the underlying host. Each Java app running on a host inside container relies on the host configuration. Considering that we are Kubernetes and many pods run on single node, this could lead to problems like the ones we are facing.

Java 10 supports containers out of the box, and it will lookup the linux cgroup information. This allows the JVM to garbage collect based on containers limits. This is turned on by default using the flag.

-XX:+UseContainerSupport

Thankfully, some of these features have been backported to 8u131 and 9 onwards. They can be turned on using the following flags.

-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

Summary

Older versions of Java read the underlying host, and don’t understand cgroups. This would cause mismatch between container configuration and the Java process. This mismatch is on both CPU and the memory. Java has an Off heap memory component which has a dynamic GC data component, which could grow. Best way to solve this is to use the container support features available in recent versions of Java. Do not rely on buffering (It is waste of money).

Upgrade to Java 8u131+ or Java 9, if you have to remain on these major versions and turn on the experimental flags. Even better, if you can get all the container love available Java 10 onwards.

References

• Java RAM Usage in Containers: Top 5 Tips for Not Losing Your Memory - DZone Performance
• Analyzing java memory usage in a Docker container
• Nobody puts Java in a container - JAXenter
• Java inside docker: What you must know to not FAIL - RHD Blog
• Should I set a MaxMetaspaceSize?

https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

• PermGen and Metaspace - DZone Performance
• Why does a JVM report more committed memory than the linux process resident set size?
• Java and Docker, the limitations

https://blog.docker.com/2018/04/improved-docker-container-integration-with-java-10/

OOM Killer and Java applications was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are many ways that your server can be compromised by remote systems and malicious softwares. It’s important that once you discovered the attack, you try to find out how. Knowing how the attacker got in can help you reduce the risk of future compromises. Few such malwares and vulnerabilities are rootkit, backdoor and exploit.

Rootkits are softwares that can provide the administrative access to the servers. This malware can be planted into the targeted system by an attacker through following means:

1. Network port
2. An un-patched system
3. System with a weak administrator password etc.

Rootkits replace normal programs and system libraries that are part of the operating system on victim machines with versions that superficially appear to be normal, but that in reality subvert the security of the machine and cause malicious functions to be executed. This kind of attack does not usually trigger automated network security controls like intrusion detection systems. It can be discovered only by digging deep on the affected system for behavioural change.

Backdoor is a means to access a computer system or encrypted data that bypasses the system’s security mechanisms. Attackers often use back-doors that they detect or install themselves as part of exploit. In some cases, a worm or virus is designed to take advantage of a backdoor created by an earlier attack. In simple words, the default password can function as back-doors if they are not changed by the user. Another example would be some debugging features that can also act as back-doors if they are not removed in the release version.

Exploit is a piece of software, data or a sequence of commands that takes advantage of a bug or vulnerabilities to cause unanticipated behaviour. In simple words “to use something for one’s own advantage”.

rkhunter is a popular tool to detect such vulnerabilities and malwares . It can scan the entire system for rootkits and help us to defend ourselves against potential threats.

rkhunter

rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. It checks the system against a database of known rootkits as well as perform the following security checks:

• Compare MD5 hash
• Look for default files used by rootkits
• Check wrong file permissions for binaries
• Look for suspected strings in LKM and KLD modules
• Look for hidden files
• Optional scan within plaintext and binary files

Install rkhunter on CentOS and RHEL

All files required for installation of rkhunter are contained in the EPEL repository.

$ yum install epel-release

Once the EPEL repository is installed, issue the following commands as root to start the installation routine and for updating the database.

$ sudo yum install rkhunter
$ rkhunter --update

Update system file properties

This is a necessary step to establish a foundation database file to compare scans. On a clean install, the first run of propupd creates a new database file. On later scans, running the propupd command updates the database file. Update the database file when you are satisfied you have only trusted source system file changes.

$ rkhunter --propupd

Automate Rootkit Hunter using CRON

rkhunter can be setup to run checks every day so that we always have up-to-date information about intrusions. This can be accomplished by creating a cronjob.

Create a cron file in the following location /etc/cron.daily/rkhunter.sh and place the following script in the file.

#!/bin/sh
(
/usr/local/bin/rkhunter --versioncheck
/usr/local/bin/rkhunter --update
/usr/local/bin/rkhunter --cronjob --report-warnings-only
) | /bin/mail -s 'rkhunter Daily Scan Report (PutYourServerNameHere)' your@email.here

Update the script with the email address to which the report should be sent.

• The --versioncheck checks if our threat definitions are up to date.
• The --update option updates our threat definitions, if required.
• The --cronjob option bypasses interactive key presses.

Set execute permission on the file you have just created

$ chmod 755 /etc/cron.daily/rkhunter.sh

The cron utility will run once daily and a report will be sent via email. Emails will be sent only if issues/problems surface. Otherwise, no emails will be sent.

Manual Scan

You can initiate a manual scan by issuing the following command:

rkhunter -c

which runs rkhunter in interactive mode. In other words, when it gets to the end of a particular scan, you need to press ‘enter’ to continue. If you want to “auto-skip” interactive mode, add the -sk option:

rkhunter -c -sk

Your scan results will look as follows:

System checks summary
=====================

File properties checks...
    Files checked: 130
    Suspect files: 0

Rootkit checks...
    Rootkits checked : 498
    Possible rootkits: 0

Applications checks...
    All checks skipped

The system checks took: 14 minutes and 34 seconds

All results have been written to the log file: /var/log/rkhunter/rkhunter.log

No warnings were found while checking the system.

Conclusion

If a rootkit has been installed in a compromised system, rebuilding the system is the best course of action. Because rootkits are so proficient in hiding themselves. Extremely strong monitoring and intrusion-detection/intrusion-prevention efforts need to be implemented. This article serves as an introductory tutorial to detection of rootkits.

References

• Knowledgebase
• What is backdoor (computing)? - Definition from WhatIs.com
• Exploits and exploit kits

A way to detect the rootkits and exploits in CentOS/RHEL was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

State-of-the-art distributed databases represent a distillation of years of research in distributed systems. The concepts underlying any distributed system can thus be overwhelming to comprehend. This is truer when you are dealing with databases without the strong consistency guarantee. Databases without strong consistency guarantees come in a range of flavours; but they are bunched under a category called NoSQL databases.

NoSQL databases do not represent a single kind of data model, nor do they offer uniform guarantees regarding consistency and availability. However, they are built on very similar principles and ideas.

From a historical perspective, the advent of NoSQL databases was precipitated by the publication of Dynamo by Amazon¹ & BigTable by Google, and the emergence of a number of open-source distributed data stores, which were (improved?) clones of either (or both) of these systems. Bigtable-inspired NoSQL stores are referred to as column-stores (e.g. HyperTable, HBase), whereas Dynamo influenced most of the key/value-stores. We will term these systems loosely as Dynamo-family databases, which include Riak, Aerospike, Project Voldemort, and Cassandra.

I would like to focus on systems design ideas in Dynamo-family NoSQL databases in this article, with a particular focus on Cassandra. The approach of this article is to compare and contrast Cassandra with Dynamo; and in this process, touch upon the underlying ideas. Expect a lot of homework & further readings; I will have copious amounts of references throughout the article.

Caveats

Cassandra, although heavily influenced by Dynamo, also borrows from BigTable.

A chunk of the differences between Cassandra & Dynamo stem from the fact that the data-model of Dynamo is a key-value store, while Cassandra is designed as a column-family data store (which is a concept from BigTable in which the primary abstraction is a sparsely populated wide table).

For what it’s worth, Cassandra is full-fledged database implementation while Dynamo is a set of ideas which when taken together can form a highly available distributed data-store (courtesy of it being a walled system with its implementation never released to public domain).

We will focus on a comparison of the implementation choices in Cassandra & Dynamo which address the challenges of a distributed system; and will try to steer clear of contrasts arising from differences in the data models.

Meat of the Matter

The essence of this article can be summarized as below:

The remainder of this article is little more than a collection of introductions to the above concepts.

High Availability for Writes

High-availability writes in a distributed database with leaderless replication (both Dynamo and Cassandra employ leaderless replication) requires a heuristic for conflict resolution between concurrent writes. This is essential because every replica of data is considered equal and concurrent writes on the same record at two different replicas are considered perfectly valid.

The common heuristics for conflict resolution are vector clocks or last-write-wins.

Vector Clocks

Vector Clocks are a mechanism to notify the actors about the occurrence of conflicts.

To illustrate the process, assume three actors with actor IDs Anu, Baba and Chandra respectively. Let an existing data-point in the data-store be {"street" : "Lavelle", "city" : "Bangalore"} with key ‘address’. We will call this version of ‘address’ V0. Anu updates the street such that the data now reads {"street" : "Cubbon", "city" : "Bangalore"}, which we will call V1. This is updated to a single replica. A concurrent update is performed by Baba, who changes the city such that the data now reads {"street" : "Lavelle", "city" : "Bombay"} (V2). This is updated to another replica.

In a data-store using vector clocks, the data-store holds onto both V1 and V2 because they do not descend from each other. When an actor reads the data at a future point of time (for example, Chandra is reading the ‘address’ data), the data-store will hand back both values. The client decides on the merge strategy of the sibling data returned to it. Once descendancy can be calculated, values stored with vector clocks that have been succeeded will be removed.

Vector Clocks were proposed as a conflict resolution mechanism in the original Dynamo paper; however, most Dynamo-family databases have last-write-wins as their conflict resolution mechanism.²

The maintainers of Riak have lucid explanations of how vector clocks work and the associated challenges in Why Vector Clocks Are Easy and Why Vector Clocks Are Hard.

Last Write Wins

In last-write-wins, the latest change to a data-point alone is retained. Thus, in the above example, V2 will be the final version of the data in a last-write-wins data-store. Thus, last-write-wins is a simplification over the vector clock approach. It could lead to potential data loss (such as the update to ‘street’ introduced in V1 is lost).

The maintainers of Cassandra believe this simplification is justified because of the data model is different from a key-value database. Each row is broken up into columns which are updated independently. This fine-grained implementation of last-write-wins is argued to work well for Cassandra. In case of time-ties for the same column, Cassandra has a rule-based, deterministic method to get a commutative result. You can read more on why Cassandra doesn’t need vector clocks here.

Temporary Failures

Temporary failures occur when a replica is unavailable for read and/or write operations for a small duration of time. This could arise because of GC stalls, network and hardware outages, or maintenance shutdowns.

Hinted Handoff is a common strategy in write paths to handle and repair temporary failures in systems with leaderless replication. In read paths, the approach for handling temporary failures could be either strict or sloppy quorum.

Hinted Handoff

In Hinted Handoff, when a write is performed and a replica node for the row is either known to be down ahead of time, or does not respond to the write request, the coordinator will store a hint locally. This hint is basically a wrapper around the mutation indicating that it needs to be replayed to the unavailable node(s).

Once a node discovers via gossip that a node for which it holds hints has recovered, it will send the data row corresponding to each hint to the target.

Hinted Handoff has two purposes:

• It allows the database to offer full write availability when consistency is not required.³
• It improves response consistency after temporary outages.⁴

The implications of hinted writes on read & write consistencies are covered in the next section on strict & sloppy quorum. Notes on Cassandra’s implementation of hinted handoff can be found here.

Strict & Sloppy Quorum

Quorum is the number of replicas which should acknowledge a particular read or write operation; it is closely associated with the replication factor. The use of hinted writes to meet consistency requirements in read paths decides whether a quorum is strict or sloppy.

In Dynamo, in scenarios where the available replicas is less than the total replicas, sloppy quorum is used to ensure read availability. The hinted writes stored in a node other than the replicas count towards read consistency requirements, and thus reads can be served even in cases where available replicas is less than the consistency requirements.

In Cassandra, strict quorum is used. This means that hinted writes do not count towards read or write consistency requirements (with the exception of ANY write consistency).

Consistent Hashing

This article started with saying that distributed databases represent a distillation of years of research in distributed systems. No other concept illustrates this better than consistent hashing (or ring hashing). Consistent hashing has been around since 1997 ⁵, and formed the basis of the formation of Akamai Technologies, and the subsequent birth of the Content Distribution Network industry.

In consistent hashing, the output range of a hash function is treated as a fixed circular space or “ring” (i.e. the largest hash value wraps around to the smallest hash value). Each node in the system is assigned multiple random value within this space. Each random value is called a vnode position; a single node is associated to multiple vnodes & consequently multiple positions on the ring.

Each data item identified by a key is assigned to a node by hashing the data item’s key to yield its position on the ring, and then walking the ring clockwise to find the first vnode with a position larger than the item’s position. The node associated with the vnode is the location of the data item.

The principle advantage of consistent hashing is incremental stability; the departure or arrival of a node into the cluster only affects its immediate neighbours and other nodes remain unaffected.

However, ring hashing is prone to uneven load distribution. The table below, taken from the paper on Jump Hash (an alternative algorithm to ring hash), shows the standard error in loads & the 99% confidence interval of bucket sizes as multiples of average loads.

This table can be read in the following way: in an implementation with 10 vnodes per node, the standard deviation of load is within (1– 0.3151810) ≈ 0.69. The 99% confidence interval for the bucket sizes lies between 0.37 to 1.98 times the average load. Such a high variance could make capacity planning tricky (although I am yet to see this in practice with Cassandra). In Cassandra, the number of vnodes is controlled by the parameter num_tokens.

Consistent hashing is also a part of the replication strategy in Dynamo-family databases. In Cassandra, two strategies exist. In SimpleStrategy, a node is anointed as the location of the first replica by using the ring hashing partitioner. Subsequent replicas are placed on the next nodes clockwise on the ring. In NetworkTopologyStrategy, for each datacenter, the same steps are performed with a difference when choosing subsequent replicas: subsequent replicas are placed on the next node clockwise on the ring which belongs to a different rack than the location of the previous replica.

The use of consistent hashing in Voldemort and Riak follow the same above-illustrated pattern. An excellent primer on alternatives to ring hashing & their respective trade-offs can be found here.

Anti-Entropy using Merkle trees

The distributed nature of data means data in a replica can become inconsistent with other replicas over time. Dynamo-family databases have a multi-pronged approach to deal with it. Hinted Hand-offs is a strategy in write-paths to handle and repair temporary failures. Read repair is a strategy to repair inconsistencies observed in the read-path.

The above two strategies work behind the scene to repair data; but due to their very nature of repairing data accessed during reads or writes alone, they cannot repair every data item in the replicas. They work best if the system membership churn is low and node failures are transient. Hence, these databases provide a manual way to trigger repairs of data. This is called an anti-entropy repair.⁶

Anti-entropy is a process of comparing the data of all replicas and updating each replica to the newest version. It relies on Merkle tree hash exchanges between nodes. The following extract, from Cassandra documentation, is a good overview of the use of Merkle trees in the anti-entropy process.

Merkle trees are binary hash trees whose leaves are hashes of the individual key values. The leaf of a Cassandra Merkle tree is the hash of a row value. Each Parent node higher in the tree is a hash of its respective children. Because higher nodes in the Merkle tree represent data further down the tree, Casandra can check each branch independently without requiring the coordinator node to download the entire data set.

After the initiating node receives the Merkle trees from the participating peer nodes, the initiating node compares every tree to every other tree. If a difference is detected, the differing nodes exchange data for the conflicting range(s), and the new data is written to SSTables. The comparison begins with the top node of the Merkle tree. If no difference is detected, then the data requires no repair. If any difference is detected, the process proceeds to the left child node and compares and then the right child node. When a node is found to differ, inconsistent data exists for the range that pertains to that node. All data that corresponds to the leaves below that Merkle tree node will be replaced with new data.

The key difference in Cassandra’s implementation of anti-entropy from Dynamo is that the Merkle trees are built per column family, and they are not maintained for longer than it takes to send them to neighboring nodes.⁷ Instead, the trees are generated as snapshots of the dataset during major compactions: this means that excess data might be sent across the network, but it saves local disk IO, and is preferable for very large datasets.

Notes on Cassandra’s implementation of anti-entropy can be found here. A more-detailed blog-entry on anti-entropy, with a similar take on comparing Riak, Cassandra & Dynamo as this article, can be found here.

Gossip Protocols

Gossip protocols are a class of peer-to-peer communication protocols inspired by information dissemination in real-life social networks. In Dynamo, decentralized failure detection protocols use a simple gossip-style protocol that enable each node in the system to learn about the arrival (or departure) of other nodes. Failure detection helps nodes in avoiding communication with unresponsive peers during read and write operations. Dynamo’s failure detection protocol is based on Gupta et al. (2001), although the exact implementation details are obscure.

The Cassandra implementation of gossip is very similar to Dynamo, and we have a lot more information on it via its documentation. The gossip process runs every second and exchanges state messages with up to three other nodes in the cluster. The nodes exchange information about themselves and about the other nodes that they have gossiped about, so all nodes quickly learn about all other nodes in the cluster. A gossip message has a version associated with it, so that during a gossip exchange, older information is overwritten with the most current state for a particular node.

The gossip process tracks state from other nodes both directly (nodes gossiping directly to it) and indirectly (nodes communicated about secondhand, third-hand, and so on). Rather than have a fixed threshold for marking failing nodes, Cassandra uses an accrual detection mechanism to calculate a per-node threshold that takes into account network performance, workload, and historical conditions. During gossip exchanges, every node maintains a sliding window of inter-arrival times of gossip messages from other nodes in the cluster. Configuring the phi_convict_threshold property adjusts the sensitivity of the failure detector.

That’s All, Folks!

The scope of topics covered in this article are vast; and I do not believe a single article does justice to them. However, the attempt here was to compile a compendium on the ideas behind NoSQL distributed systems. A future article will attempt a similar take on NewSQL databases, but that’s for another day. Hope you find this useful. Until next time!

Notes

1. A fair amount of review-articles exists for the Dynamo paper: a few interesting takes can be found here & here.
2. Riak has last-write-wins as its default mechanism for conflict resolution, inspite of a high chance of data-loss. As its maintainers put it,

Vector clocks are hard: even with perfect implementation you can’t have perfect information about causality in an open system without unbounded information growth.

3. The catch here is ‘when consistency is not required’. In Dynamo which uses sloppy quorum, hinted writes count towards write consistency requirements, and hence hinted handoff helps offer full write availability.

In Cassandra, for every write consistency other than ANY, hinted writes do not count towards write consistency requirements. Hence, full write availability may not be available.

4. In a similar vein to the note above, sloppy quorum in Dynamo means that hints stored in nodes other than the replicas count towards read consistency requirements; thus, improving read availability.

Cassandra does not use hinted writes in computing read consistency requirements; thus hinted handoffs do not improve read availability.

5. This may not be entirely true. A form of consistent hashing is claimed to have existed in Teradata since 1986. I could not find any source to back this claim. However, it is clear the Akamai paper was the first to name this technique as consistent hashing.

6. Anti-entropy processes need not always be manual. Riak periodically clears & regenerates the Merkle hash trees used in the anti-entropy process. They term it active anti-entropy.

7. Cassandra does not persist the hash trees generated during anti-entropy. On the other hand, Riak (which seems to have followed the original Dynamo approach on this) uses persistent, on-disk hash trees instead of in-memory hash trees. The advantages of this approach, as stated by its maintainers, are twofold: i) Riak can run anti-entropy operations with a minimal impact on memory usage. ii) Riak nodes can be restarted without needing to rebuild hash trees.

Originally published at sujithjay.com on October 2, 2018.

Dynamo vs Cassandra : Systems Design of NoSQL Databases was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

This article is an introductory reference to understanding Apache Spark on YARN. Since our data platform at Logistimo runs on this infrastructure, it is imperative you (my fellow engineer) have an understanding about it before you can contribute to it. This article assumes basic familiarity with Apache Spark concepts, and will not linger on discussing them.

Overview on YARN

YARN is a generic resource-management framework for distributed workloads; in other words, a cluster-level operating system. Although part of the Hadoop ecosystem, YARN can support a lot of varied compute-frameworks (such as Tez, and Spark) in addition to MapReduce.

The central theme of YARN is the division of resource-management functionalities into a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is the unit of scheduling on a YARN cluster; it is either a single job or a DAG of jobs (jobs here could mean a Spark job, an Hive query or any similar constructs).

The ResourceManager and the NodeManager form the data-computation framework. The ResourceManager is the ultimate authority that arbitrates resources among all the applications in the system. The NodeManager is the per-machine agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler [1].

The per-application ApplicationMaster is, in effect, a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks [1].

Glossary

The first hurdle in understanding a Spark workload on YARN is understanding the various terminology associated with YARN and Spark, and see how they connect with each other. I will introduce and define the vocabulary below:

Application

A Spark application is the highest-level unit of computation in Spark. A Spark application can be used for a single batch job, an interactive session with multiple jobs, or a long-lived server continually satisfying requests. A Spark job can consist of more than just a single map and reduce. On the other hand, a YARN application is the unit of scheduling and resource-allocation. There is a one-to-one mapping between these two terms in case of a Spark workload on YARN; i.e, a Spark application submitted to YARN translates into a YARN application.

Spark Driver

To understand the driver, let us divorce ourselves from YARN for a moment, since the notion of driver is universal across Spark deployments irrespective of the cluster manager used.

Spark applications are coordinated by the SparkContext (or SparkSession) object in the main program, which is called the Driver. In plain words, the code initialising SparkContext is your driver. The driver process manages the job flow and schedules tasks and is available the entire time the application is running (i.e, the driver program must listen for and accept incoming connections from its executors throughout its lifetime. As such, the driver program must be network addressable from the worker nodes) [4].

YARN Client

A program which submits an application to YARN is called a YARN client, as shown in the figure in the YARN section. Simple enough.

The notion of driver and how it relates to the concept of client is important to understanding Spark interactions with YARN. In particular, the location of the driver w.r.t the client & the ApplicationMaster defines the deployment mode in which a Spark application runs: YARN client mode or YARN cluster mode.

Client mode: The driver program, in this mode, runs on the YARN client. Thus, the driver is not managed as part of the YARN cluster.

Take note that, since the driver is part of the client and, as mentioned above in the Spark Driver section, the driver program must listen for and accept incoming connections from its executors throughout its lifetime, the client cannot exit till application completion.

Cluster mode: The driver program, in this mode, runs on the ApplicationMaster, which itself runs in a container on the YARN cluster. The YARN client just pulls status from the ApplicationMaster. In this case, the client could exit after application submission.

The first fact to understand is: each Spark executor runs as a YARN container [2]. This and the fact that Spark executors for an application are fixed, and so are the resources allotted to each executor, a Spark application takes up resources for its entire duration. This is in contrast with a MapReduce application which constantly returns resources at the end of each task, and is again allotted at the start of the next task.

Also, since each Spark executor runs in a YARN container, YARN & Spark configurations have a slight interference effect. I will illustrate this in the next segment.

Configuration and Resource Tuning

With our vocabulary and concepts set, let us shift focus to the knobs & dials we have to tune to get Spark running on YARN. We will be addressing only a few important configurations (both Spark and YARN), and the relations between them.

We will first focus on some YARN configurations, and understand their implications, independent of Spark.

• yarn.nodemanager.resource.memory-mb

It is the amount of physical memory, in MB, that can be allocated for containers in a node. This value has to be lower than the memory available on the node.

<name>yarn.nodemanager.resource.memory-mb</name>
<value>16384</value> <!-- 16 GB -->

• yarn.scheduler.minimum-allocation-mb

It is the minimum allocation for every container request at the ResourceManager, in MBs. In other words, the ResourceManager can allocate containers only in increments of this value. Thus, this provides guidance on how to split node resources into containers. Memory requests lower than this will throw a InvalidResourceRequestException.

<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value><!-- 2 GB -->

• yarn.scheduler.maximum-allocation-mb

The maximum allocation for every container request at the ResourceManager, in MBs. Memory requests higher than this will throw a InvalidResourceRequestException.

<name>yarn.scheduler.maximum-allocation-mb</name> <value>8192</value><!-- 8 GB -->

Thus, in summary, the above configurations mean that the ResourceManager can only allocate memory to containers in increments of yarn.scheduler.minimum-allocation-mb and not exceed yarn.scheduler.maximum-allocation-mb, and it should not be more than the total allocated memory of the node, as defined by yarn.nodemanager.resource.memory-mb.

We will refer to the above statement in further discussions as the Boxed Memory Axiom (just a fancy name to ease the discussions). A similar axiom can be stated for cores as well, although we will not venture forth with it in this article.

Let us now move on to certain Spark configurations. In particular, we will look at these configurations from the viewpoint of running a Spark job within YARN.

• spark.executor.memory

Since every executor runs as a YARN container, it is bound by the Boxed Memory Axiom. However, a source of confusion among developers is that the executors will use a memory allocation equal to spark.executor.memory. In essence, the memory request is equal to the sum of spark.executor.memory + spark.executor.memoryOverhead. Thus, it is this value which is bound by our axiom.

• spark.driver.memory

In cluster deployment mode, since the driver runs in the ApplicationMaster which in turn is managed by YARN, this property decides the memory available to the ApplicationMaster, and it is bound by the Boxed Memory Axiom. But as in the case of spark.executor.memory, the actual value which is bound is spark.driver.memory + spark.driver.memoryOverhead.

In case of client deployment mode, the driver memory is independent of YARN and the axiom is not applicable to it. In turn, it is the value spark.yarn.am.memory + spark.yarn.am.memoryOverhead which is bound by the Boxed Memory Axiom.

I hope this article serves as a concise compilation of common causes of confusions in using Apache Spark on YARN. More details can be found in the references below. Please leave a comment for suggestions, opinions, or just to say hello. Until next time!

References

[1] “Apache Hadoop 2.9.1 — Apache Hadoop YARN”. hadoop.apache.org, 2018, Available at: Link. Accessed 23 July 2018.

[2] Ryza, Sandy. “Apache Spark Resource Management And YARN App Models — Cloudera Engineering Blog”. Cloudera Engineering Blog, 2018, Available at: Link. Accessed 22 July 2018.

[3] “Configuration — Spark 2.3.0 Documentation”. spark.apache.org, 2018, Available at: Link. Accessed 22 July 2018.

[4] “Cluster Mode Overview — Spark 2.3.0 Documentation”. spark.apache.org, 2018, Available at: Link. Accessed 23 July 2018.

Originally published at sujithjay.com on July 24, 2018.

Understanding Apache Spark on YARN was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a microservice environment handling huge workloads, a performance dip of a single microservice will mean the performance degradation of an entire application. A single weak link in the chain can endanger our SLAs. The SCM product at Logistimo handles mission-critical public-health supply chains for host of countries, via an array of microservices. Since the vitality of these systems is important at many levels, we take monitoring performance very seriously.

Our microservices are managed in a Kubernetes (K8s) cluster. The K8s ecosystem provides an array of options to monitor health and performance of the services. However, we wanted a solution which could monitor the performance of the application at the micro-service level. This brought us to the use of Elastic APM for our Kubernetes cluster. APM stands for Application Performance Monitoring. Deploying Elastic APM in Kubernetes ecosystem is a topic on which not much literature is found on the web. Hence, We decided to share our experience.

Elastic APM system:

If you are using the ELK stack( Elasticsearch, Logstash and Kibana ) for log aggregation in your infrastructure, you might be recording all the application logs in the Elasticsearch. While this in itself is useful, It is necessary to have APM monitoring for microservices. Elastic APM streams all events to Elastic search much like the Logstash.

Since data is stored as raw documents in Elasticsearch, your service metrics can coexist with your infrastructure logs making it easy to explore all of your data in one place. These metrics can be visualised in Kibana with pre-configured dashboards.

To do this APM metrics collection, you need two components:

1. APM server
2. APM agent

APM server

APM Server is a central component in the Elastic APM system. It receives data from APM agents and transforms them into Elastic search documents

A single APM server can handle data from multiple microservices. APM Server works by exposing an HTTP server to which agents post the APM data they collect.

To deploy the basic APM Server in K8s infrastructure, you can use following Kube-apm-server.yaml file:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-server-config
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"
      frontend:
        enabled: false
    setup.template.settings:
      index:
        number_of_shards: 1
        codec: best_compression
    setup.dashboards.enabled: true
    setup.kibana:
      host: "< kibana endpoint >"
    output.elasticsearch:
      hosts: ["< elasticsearch endpoint >"]
      indices:
        - index: "apm-%{[beat.version]}-sourcemap"
          when.contains:
            processor.event: "sourcemap"

- index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "error"

- index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "transaction"

- index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "span"
---
apiVersion: v1
kind: Service
metadata:
  name: apm-server
  labels:
    app: apm-server
spec:
  ports:
  - port: 8200
    targetPort: 8200
    name: http
    nodePort: 31000
  selector:
    app: apm-server
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: apm-server
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 1
  template:
    metadata:
      labels:
        app: apm-server
    spec:
      containers:
      - name: apm-server
        image: docker.elastic.co/apm/apm-server:6.3.1
        ports:
        - containerPort: 8200
          name: apm-port
        volumeMounts:
        - name: apm-server-config
          mountPath: /usr/share/apm-server/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: apm-server-config
        configMap:
          name: apm-server-config

Replace the host: “< Kibana endpoint >” and hosts: [“< Elasticsearch endpoint >”] with end points.

APM Server comes with predefined Kibana dashboards and index templates for APM data.

Following is the list of default dashboards available. You can also configure custom dashboards according to your needs.

1. Transactions
2. Span Details
3. Services
4. Errors
5. Error Details

These dashboards give you an instant overview of application response times, requests per minutes, error occurrences and more.

APM Agent

The agents are libraries that run inside your application process. Agents automatically instrument your application in order to measure the duration of requests to your service. It also automatically measures things like database queries, cache calls, external http requests and errors. Ensure you add APM agent to all of your micro services to get the metrics.

APM agent for Dockerized Java application:

The first step in getting started with the Elastic APM Java agent is to download the latest release of the agent jar file from maven central.

Add the following code in your Dockerfile to download and set the javaagent environment variable at the build time of the microservice.

RUN wget -P < $TOMCAT_HOME/ > http://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/<latest-version>/elastic-apm-agent-<latest-version>.jar

ENV JAVA_OPTS=-javaagent:$TOMCAT_HOME/elastic-apm-agent-< elastic-apm.version >.jar

Elastic APM agent has lot of configuration options.The most important ones are service_name (required), server_url, and application_packages.

System properties:

-Delastic.apm.service_name=my-service
-Delastic.apm.application_packages=org.example
-Delastic.apm.server_url=http://localhost:8200

The complete Dockerfile entry for apm-agent will look like this

RUN cd $TOMCAT_HOME && wget http://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/<latest-version>/elastic-apm-agent-<latest-version>.jar
 

ENV JAVA_OPTS=-javaagent:$TOMCAT_HOME/elastic-apm-agent-<latest-version>.jar \
              -Delastic.apm.service_name=< Service name > \
              -Delastic.apm.application_packages=< application pakage name > \
              -Delastic.apm.server_url=< APM server url/endpoint >

The service name is used by Elastic APM to differentiate data coming from different services and to group data coming from the same services. When configuring an agent, you need to supply a service name.

The service name can only contain alphanumeric characters, spaces, underscores, and dashes (must match ^[a-zA-Z0–9 _-]+$).

You can set the Service Name and APM server url as the docker environmental variable as the value for these properties changes per environment you deploy.

APM agent for Dockerized Node.js application

In Dockerfile add the follwoing line

RUN npm install elastic-apm-node --save

Agent should be started before you mark other modules as dependency using require in your Node.js application . You will include the following code to enable agent in your application’s main file (usually index.js, server.js or app.js).

// Add this to the VERY top of the first file loaded in your app
  var apm = require('elastic-apm-node').start({
    
    // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
    serviceName: '< Service Name>',

// Use if APM Server requires a token
    secretToken: '< Secret token >',

// Set custom APM Server URL (default: http://localhost:8200 )
    serverUrl: '< APM Server URL >',
  })

var app = require('express')()

Provided the correct value for the serviceName and serverUrl for the metrics to be collected.

End Note:

Elastic APM allowed us to avoid developing additional metering and instrumentation inside the micro services. APM agent takes care of monitoring all the key metrics. The data has been immensely useful for us to detect problems early and understand the state of the systems in production.

Happy Monitoring !!

Using Elastic APM in Kubernetes was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: I wrote this article as part of a contribution to Scala Documentation. The original post can be found here.

Methods may define multiple parameter lists. When a method is called with a fewer number of parameter lists, then this will yield a function taking the missing parameter lists as its arguments. This is formally known as currying.

Here is an example, defined in Traversable trait from Scala collections:

def foldLeft[B](z: B)(op: (B, A) => B): B

foldLeft applies a binary operator op to an initial value z and all elements of this traversable, going left to right. Shown below is an example of its usage.

Starting with an initial value of 0, foldLeft here applies the function (m, n) => m + n to each element in the List and the previous accumulated value.

val numbers = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) 
val res = numbers.foldLeft(0)((m, n) => m + n) 
print(res) // 55

Multiple parameter lists have a more verbose invocation syntax; and hence should be used sparingly. Suggested use cases include:

Single functional parameter

In case of a single functional parameter, like op in the case of foldLeft above, multiple parameter lists allow a concise syntax to pass an anonymous function to the method. Without multiple parameter lists, the code would look like this:

numbers.foldLeft(0, {(m: Int, n: Int) => m + n})

Note that the use of multiple parameter lists here also allows us to take advantage of Scala type inference to make the code more concise as shown below; which would not be possible in a non-curried definition.

numbers.foldLeft(0)(_ + _)

Also, it allows us to fix the parameter z and pass around a partial function and reuse it as shown below:

val numbers = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) 
val numberFunc = numbers.foldLeft(List[Int]())_ 
val squares = numberFunc((xs, x) => xs:+ x*x) print(squares.toString()) // List(1, 4, 9, 16, 25, 36, 49, 64, 81, 100) 
val cubes = numberFunc((xs, x) => xs:+ x*x*x) print(cubes.toString()) // List(1, 8, 27, 64, 125, 216, 343, 512, 729, 1000)

Implicit parameters

To specify certain parameters in a parameter list as implicit, multiple parameter lists should be used. An example of this is:

def execute(arg: Int)(implicit ec: ExecutionContext) = ???

Originally published at sujithjay.com on January 27, 2018.

Multiple Parameter Lists in Scala was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.