The Open Source Routing Machine (OSRM) is an open-source router designed for use with data from the OpenStreetMap project. The following services are available via HTTP API, C++ library interface and NodeJs wrapper:

• Nearest — Snaps coordinates to the street network and returns the nearest matches
• Route — Finds the fastest route between coordinates
• Table — Computes the duration or distances of the fastest route between all pairs of supplied coordinates
• Match — Snaps noisy GPS traces to the road network in the most plausible way
• Trip — Solves the Traveling Salesman Problem using a greedy heuristic
• Tile — Generates Mapbox Vector Tiles with internal routing metadata

Why OpenStreetMap over Google Maps ?

Open Street Maps (OSM) is an open source project maintained by the OpenStreetMap Community that provides free editable maps of the whole world. The most obvious advantage in using OpenStreetMap-based software over a commercial solution is economical convenience, because OpenStreetMap comes as free software.

The downside is that it takes a little configuration in order to setup a working web service. The easiest and quickest way to setup your own routing engine is to use Docker images

There are two pre-processing pipelines available:

• Contraction Hierarchies (CH)
• Multi-Level Dijkstra (MLD)

We recommend using MLD by default except for special use-cases such as very large distance matrices where CH is still a better fit for the time being. In the following we explain the MLD pipeline. If you want to use the CH pipeline instead replace osrm-partition and osrm-customize with a single osrm-contract and change the algorithm option for osrm-routed to --algorithm ch.

Generate data using Docker:

Our web service will be based on a map export from OpenStreetMap. There are several possible ways to obtain a map export.

The first option is to point your browser to the OpenStreetMap’s export page, zoom on the interested area, and click on Export. (You may need to use one of the other sources if the native export does not work; Overpass API is a good choice.) This will allow you to finely choose which areas to include in your map and, in general, to cut down on the pre-processing times.

https://www.openstreetmap.org/export#map=6/16.225/73.059

However, there are limits on the size of the export you can get from the OSRM website, so you might want to download a ready-made export. A number of services, such as Geofabrik and Mapzen offer updated and ready-made map exports of countries and selected metropolitan areas, which are good for most use cases.

In this article we will see how the OSRM data is generated for India using MLD pipeline.

To generate the routing data you need to download the .pbf file for India from Geofabrik.

wget http://download.geofabrik.de/asia/india-latest.osm.pbf

OSM data comes in a variety of formats including XML and PBF, and contain a plethora of data. The data includes information which is irrelevant to routing, such as positions of public waste baskets. Also, the data does not conform to a hard standard and important information can be described in various ways. Thus it is necessary to extract the routing data into a normalized format. This is done by the OSRM tool named extractor.

Profiles are used during this process to determine what can be routed along, and what cannot (private roads, barriers etc.).

For the MLD pipeline we need to extract (osrm-extract) a graph out of the OpenStreetMap base map

docker run -t -v $(pwd):/data osrm/osrm-backend:latest osrm-extract -p /opt/car.lua /data/india-latest.osm.pbf

The flag -v $(pwd):/data creates the directory /data inside the docker container and makes the current working directory $(pwd) available there. The file /data/india-latest.osm.pbf inside the container is referring to $(pwd)/india-latest.osm.pbf on the host

Then partition (osrm-partition) this graph recursively into cells, by using the following command

docker run -t -v $(pwd):/data osrm/osrm-backend:latest osrm-partition /data/india-latest.osrm

Note that india-latest.osrm has a different file extension.

Once the graph is partitioned in to cells, customize the cells (osrm-customize) by calculating routing weights for all cells

docker run -t -v $(pwd):/data osrm/osrm-backend:latest osrm-customize /data/india-latest.osrm

For CH the partition and customize pipeline stage gets replaced by adding shortcuts from the Contraction Hierarchies algorithm (osrm-contract)

Once the graph is customized, then spawning up the development HTTP server (osrm-routed) responding to queries

OSRM Docker Deployment

Docker image can be run standalone without Kubernetes

docker run -t -i -p 5000:5000 -v $(pwd):/data -d osrm/osrm-backend:latest osrm-routed — algorithm mld /data/india-latest.osrm

Make requests against the HTTP server

Optionally start a user-friendly frontend on port 9966, and open it up in your browser

docker run -p 9966:9966 osrm/osrm-frontend

OSRM Kubernetes Deployment

To deploy the OSRM in Kubernetes cluster, we need the routing data inside the Kubernetes cluster. There are multiple ways to achieve this. The reliable way is to use persistent volume model in a dynamic Kubernetes cluster. On a standalone k8s cluster the simple and easiest way is to use the hostPath.

To deploy on standalone cluster once the data generation is completed, copy all the generated files to the following location /data/osrm/india on the server. Make sure this data is available on all the Kubernetes nodes.

Now use the below mentioned YAML file to deploy the osrm-backend application in k8s.

osrm-backend-india.yaml

apiVersion: v1
kind: Service
metadata:
  name: osrm-india
  labels:
    app: osrm-india
spec:
  ports:
  - port: 5000
    targetPort: 5000
    name: http
  selector:
    app: osrm-india
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: osrm-india
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: osrm-india
    spec:
      containers:
      - name: osrm-india
        image: osrm-backend:latest
        command: ["/bin/sh", "-c"]
        args: ["osrm-routed --algorithm mld /data/india-latest.osm.pbf"]
        ports:
        - containerPort: 5000
          name: osrm-port
        volumeMounts:
        - name: india
          readOnly:  true
          mountPath: /data
      volumes:
      - name: india
        hostPath:
          path: /data/osrm/india
        resources:
          requests:
            memory: "512Mi"
            cpu: "0.5"
          limits:
            memory: "512Mi"
            cpu: "0.5"

Below config make sure the data is mounted to the docker container while deploying

        volumeMounts:
        - name: india
          readOnly:  true
          mountPath: /data
      volumes:
      - name: india
        hostPath:
          path: /data/osrm/india

Command to run the osrm-backend image in Kubernetes

kubectl apply -f osrm-india.yaml

To deploy the user-facing frontend with map, apply the below file

osrm-frontend.yaml

kind: Service
metadata:
  name: osrm-frontend
  labels:
    app: osrm-frontend
spec:
  ports:
  - port: 9966
    targetPort: 9966
    name: http
  selector:
    app: osrm-frontend
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: osrm-frontend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: osrm-frontend
    spec:
      containers:
      - name: osrm-frontend
        image: osrm/osrm-frontend
        ports:
        - containerPort: 9966
          name: osrm-frontend
        env:
        - name: BACKEND
          value: 'http://osrm-india:5000'

Command to run the osrm-backend image in Kubernetes

kubectl apply -f osrm-frontend.yaml

Note:

OSRM data can be generated for any countries. The data generation consumes lot of memory on your server based on the country . To generate OSRM data for India, docker runs consumes almost 7 GB of RAM on the server.

Conclusion

This blog covers the configuration and maintenance of a web service which can answer questions such as:

• What is the closest street to a given pair of coordinates?
• What’s the best way to get from point A to point B?
• How long does it take to get from point A to point B with a car, or by foot?

References

• Geofabrik Download Server
• BBBike extracts OpenStreetMap

OSRM data generation and deployment was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Most people assume that Linux is already secure, and that’s a false assumption. It’s important to know that the Linux operating system has so many distributions and each one will differ from the command line perspective, but the logic is the same. This article will help you to secure and harden the Linux server.

Following are the important hardening tips for your linux servers:

• Remove unnecessary services
• Check for security on key files
• Set strong password policy
• Limit root access using SUDO
• Only allow root to access CRON
• Remote access and SSH basic settings
• Disable Xwindow
• Selinux (Security Enhanced Linux)
• Minimize Package Instllation
• Checking accounts for empty passwords
• Monitor user activities
• Install and configure fail2ban
• Rootkit detection
• Monitor system logs
• Enable 2-factor authentication

1. Remove unnecessary services

Hardening systems by eliminating unnecessary services can enhance security
and improve overall system performance. To begin, you first need to know
which services are running on your system. Since services run in various ways, there are several places to check.

$ ps –ax   (will list all currently running processes)

$ ls –l /etc/rc.d/rc3.d/S*  (will show all start-up scripts (if you boot into graphics mode, replace rc3.d with rc5.d))

$ netstat –a   (will list all open ports)

$ chkconfig –list   (will show the current startup status of all processes known by chkconfig)

Ideally, you should see only those ports that must be open to provide the
functionality required by the system.

2. Check for security on key files

• /etc/fstab: make sure the owner & group are set to root.root and the
  permissions are set to 0644 (-rw-r — r — )
• verify that /etc/passwd, /etc/shadow & /etc/group are all owned by ‘root’
• verify that permissions on /etc/passwd & /etc/group are rw-r — r — (644)
• verify that permissions on /etc/shadow are r — — — — (400)

3. Set strong password policy

Ensure the default system password policy matches your organization
password policy. These settings are stored in /etc/login.defs and should
minimally contain settings for the following.
PASS_MAX_DAYS 90
PASS_MIN_DAYS 6
PASS_MIN_LEN 14
PASS_WARN_AGE 7

4. Limit root access using SUDO

Sudo allows an administrator to provide certain users the ability to run some
commands as root, while logging all sudo activity. Sudo operates on a percommand basis. The sudoers file controls command access.

5. Only allow root to access CRON

The cron daemon is used to schedule processes. The crontab command is
used to create personal crontab entries for users or the root account. To
enhance security of the cron scheduler, you can establish the cron.deny and
cron.allow files to control use of the crontab. The following commands will
establish root as the only user with permission to add cron jobs.

$ cd /etc/
$ /bin/rm -f cron.deny at.deny
$ echo root >cron.allow
$ echo root >at.allow
$ /bin/chown root:root cron.allow at.allow
$ /bin/chmod 400 cron.allow at.allow

6. Remote access and SSH basic settings

Telnet is not recommended for remote access. Secure Shell (SSH) provides
encrypted telnet-like access and is considered a secure alternative to telnet.
However, older versions of SSH have vulnerabilities and should not be used.
To disable SSH version 1 and enhance the overall security of SSH, consider
making the following changes to your sshd_config file:

Protocol 2
PermitRootLogin no
PermitEmptyPasswords no
Banner /etc/issue
IgnoreRhosts yes
RhostsAuthentication no
RhostsRSAAuthentication no
HostbasedAuthentication no
LoginGraceTime 1m (or less — default is 2 minutes)
SyslogFacility AUTH (provides logging under syslog AUTH)
AllowUser [list of users allowed access]
DenyUser [list of system accounts and others not allowed]
MaxStartups 10 (or less — use 1/3 the total number of remote users)

Note: MaxStartups refers to the max number of simultaneous unauthenticated connections. This setting can be helpful against a bruteforce
script that performs forking.

Running ssh on an alternate port will help you avoid port scanners that are looking for open port. You can block such brute-force ssh attacks with a package like denyhosts, which utilizes tcpwrappers.

Alternatively, use your iptables firewall (see below) to limit access by IP
address or host/domain name.

7. Disable Xwindow

X window can be a large security risk considering the many exploits for the
product and since its data flows unencrypted across networks. There is no need to run X Window desktops like KDE or GNOME on your dedicated LAMP server. To disable X11, open the file ‘/etc/inittab‘ and set run level to 3. If you wish to remove it completely from the system use the below command.

$ yum groupremove “X Window System”

8. Selinux

The Security-enhanced Linux kernel enforces mandatory access control
policies that confine user programs and system servers to the minimum amount of privilege they require to do their jobs. . This confinement mechanism operates independently of the traditional Linux access control mechanisms. Implementing SE Linux can have unexpected effects on a system, and you may find standard daemons won’t run properly, or at all, or logfiles may not be writable, or other similar effects that require detailed configuration of SE Linux Currently.

9. Minimize package installation

It’s recommended to avoid installing useless packages to avoid vulnerabilities in packages. This may minimize risk that compromise of one service may lead to compromise of other services. Find and remove or disable unwanted services from the server to minimize vulnerability.

10. Checking accounts for empty passwords

Any account having an empty password means its opened for unauthorized access to anyone on the web and it’s a part of security within a Linux server. So, you must make sure all accounts have strong passwords and no one has any authorized access. Empty password accounts are security risks and that can be easily hackable. To check if there were any accounts with empty password, use the following command.

$ cat /etc/shadow | awk -F: ‘($2==””){print $1}’

11. Monitor user activities

Its important to collect the information of each user activities and processes consumed by them and analyse them at a later time or in case if any kind of performance, security issues. But how we can monitor and collect user activities information.

There are two useful tools called ‘psacct‘ and ‘acct‘ are used for monitoring user activities and processes on a system. These tools runs in a system background and continuously tracks each user activity on a system and resources consumed by services such as Apache, MySQL, SSH, FTP, etc.

12. Install and configure fail2ban

Any service that is exposed to the network is a potential target in this way. Most Linux servers offer an SSH login via Port 22 for remote administration purposes. This port is a well-known port, therefore, it is often attacked by brute force attacks. If you pay attention to application logs for these services, you will often see repeated, systematic login attempts that represent brute-force attacks by users and bots alike. Fail2ban can mitigate this problem by creating rules that automatically alter your iptables firewall configuration based on a predefined number of unsuccessful login attempts.

13. Rootkit detection

There are many ways that your server can be compromised by remote systems and malicious softwares. It’s important that once you discovered the attack, you try to find out how. Knowing how the attacker got in can help you reduce the risk of future compromises. Few such Malwares and vulnerabilities are Rootkit, Backdoor and Exploit. Following blog will guide you on the configuration of Rootkit hunter.

https://medium.com/logistimo-engineering-blog/a-way-to-detect-the-rootkits-and-exploits-in-centos-rhel-5b125a8d6a25

14. Monitor system logs

Logwatch is an application that helps with simple log management by daily analysing and reporting a short digest from activities taking place on your machine. This tool analyzes and generates daily reports on your system’s log activity.

15. Enable 2-factor authentication

Enabling 2-factor authentication for all the servers is not advisable. It can be configured on the jump host ( login box ) of the infrastructure. It acts as an additional security layer for your business, helping to address the vulnerabilities of a standard password-only approach

References

• A way to detect the rootkits and exploits in CentOS/RHEL
• Multi-Factor Authentication | JumpCloud
• How Fail2Ban Works to Protect Services on a Linux Server | DigitalOcean
• How To Install and Use Logwatch Log Analyzer and Reporter on a VPS | DigitalOcean

Monitoring metrics and runtime characteristics of an application server is essential to ensure the adequate functioning of the applications running on that server, as well as to prevent or resolve potential issues in a timely manner. As far as Java applications go, one of the most commonly used web application servers is Apache Tomcat, which will be the focus of this article.

Tomcat performance monitoring in Kubernetes cluster can be done either by relying on JMX beans or other tools available in the market. We have listed the prerequisite with which this monitoring can be achieved.

1. Prometheus Agent
2. Kubernetes YAML for application
3. Annotations
4. Scraping Policy
5. Prometheus Server

Prometheus Agent

Prometheus provides a collector that can scrape and expose the mBeans of a JMX target. It runs as a Java Agent, exposing an HTTP server and serving metrics of the local JVM. It can also be run as an independent HTTP server and scrape remote JMX targets, but this comes with various disadvantages, such as it being harder to configure and it being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.

The following Dockerfile adds support for the options we’ve listed.

FROM tomcat

RUN mkdir /data

ADD jmx_prometheus_javaagent-0.7-SNAPSHOT.jar /data/jmx_prometheus_javaagent-0.7-SNAPSHOT.jar

ADD prometheus-jmx-config.yaml /data/prometheus-jmx-config.yaml

CMD java -javaagent:/data/jmx_prometheus_javaagent-0.7.jar=8088:/data/prometheus-jmx-config.yaml -jar $TOMCAT_HOME/start.jar

• Connect to an exposed JMX port of the JVM (not recommended)
• Java agent version (recommended)

If you use the agent version, you’ll have to modify three configuration options:

• the Jar file location
• the port for the http(s) interface, where the metrics will be available for scraping, already in Prometheus friendly format
• additional configuration options

An example looks like this:

-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.7.jar=8088

The latest jar can be downloaded from the following link:

wget http://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.7/jmx_prometheus_javaagent-<latest-version>.jar

Create prometheus-jmx-config.yaml with following config in it

---
hostPort: 127.0.0.1:1234
lowercaseOutputName: false
lowercaseOutputLabelNames: false

The complete options for the prometheus-jmx-config.yaml is available here.

Kubernetes YAML for application

When you run the tomcat image in Kubernetes infrastructure, the JMX port needs to be exposed to scrape the data from the container. Following sample tomcat YAML file shows how to deploy the basic tomcat service in K8s infrastructure with prometheus JMX port exposed and annotated.

apiVersion: v1
kind: Service
metadata:
  name: <service name>
  labels:
    app: <service name>
spec:
  ports:
  - port: 8080
    targetPort: 8080
    name: http
  - port: 8088
    targetPort: 8088
    name: prom
  selector:
    app: <service name>
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: <deployment name>
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 1
  template:
    metadata:
      labels:
        app: <deployment name>
        type: java
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8088'
    spec:
      containers:
      - name: <container name>
        image: <docker image>
        ports:
        - containerPort: 8080
        env:
        - name: JAVA_AGENT_PORT
          value: "8088"
        resources:
          requests:
            memory: "1024Mi"
            cpu: "1"
          limits:
            memory: "1024Mi"
            cpu: "1"

Annotations

Annotations on pods allow a fine grained control of the scraping process:

• prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation will exclude the pod from the scraping process.
• prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
• prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports (default is a port-free target if none are declared).

These annotations need to be part of the pod metadata. They will have no effect if set on other objects such as Services or DaemonSets. The above Deployment manifest will instruct Prometheus to scrape all of its pods on port 8088.

Scraping Policy

To minimise the amount of configuration required from the user, the default setup will try to scrape all pods present in the system. This may have some undesirable side-effects such as:

• Prometheus issuing HTTP GET requests to TCP ports not even exposing an HTTP server.
• Prometheus issuing HTTP GET requests to fetch /metrics on services that may not export any metric, polluting logs.

While the annotations described in the previous section allow the user to opt-out of the scraping process, we also offer an alternative Prometheus configuration that only scrapes a minimal set of system metrics and requires the user to opt-in on a per-pod basis. In other words, in this mode the prometheus.io/scrape: true annotation is required for Prometheus to scrape the pod.

Now that we have all the required configs in place to scrape the JVM data from the tomcat container. Let see how it can be scraped via prometheus server.

Prometheus Server

Prometheus is a monitoring platform that collects metrics from monitored targets by scraping metrics HTTP endpoints on these targets. Please use the following docker-compose.yaml for downloading the Prometheus Image

---
version: '2'
services:
  prometheus:
    container_name: prometheus
    image: prom/prometheus:latest
    restart: always
    ports:
      - "9090:9090"
    volumes:
    - /data/prometheus/prometheus.yml:/prometheus/prometheus.yml:ro
    command: [ '--config.file=/prometheus/prometheus.yml', '--web.enable-lifecycle', '--web.enable-admin-api', '--storage.tsdb.retention=1y' ]

Configuring Prometheus

The Prometheus download comes with a sample configuration in a file called prometheus.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: 'prometheus
    scrape_interval: 30s
    kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
        - staging
    scheme: http
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

There are three blocks of configuration in the example configuration file: global, rule_files, and scrape_configs.

• The global block controls the Prometheus server's global configuration. We have two options present. The first, scrape_interval, controls how often Prometheus will scrape targets. The evaluation_interval option controls how often Prometheus will evaluate rules.
• The rule_files block specifies the location of any rules we want the Prometheus server to load. For now we've got no rules.
• The last block, scrape_configs, controls what resources Prometheus monitors. Since Prometheus also exposes data about itself as an HTTP endpoint it can scrape and monitor its own health. Prometheus expects metrics to be available on targets on a path of /metrics. So this default job is scraping via the URL: http://<hostname>:9090/metrics.

When you scrape data from the containers running inside the Kubernetes cluster, don’t forget to mentions the namespace from where the data should be scraped as mentioned in the above config file.

Once you have all prerequisites configured and deployed, the JVM metrics will be collected and stored in the prometheus. We use Grafana to create a graphical dashboard and measure the performance as shown below.

References

• JVM overview - Prometheus dashboard for Grafana
• GitHub - prometheus/jmx_exporter: A process for collecting metrics using JMX MBeans for Prometheus consumption
• Collect Docker metrics with Prometheus

Tomcat JVM metrics monitoring using Prometheus in Kubernetes was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are many ways that your server can be compromised by remote systems and malicious softwares. It’s important that once you discovered the attack, you try to find out how. Knowing how the attacker got in can help you reduce the risk of future compromises. Few such malwares and vulnerabilities are rootkit, backdoor and exploit.

Rootkits are softwares that can provide the administrative access to the servers. This malware can be planted into the targeted system by an attacker through following means:

1. Network port
2. An un-patched system
3. System with a weak administrator password etc.

Rootkits replace normal programs and system libraries that are part of the operating system on victim machines with versions that superficially appear to be normal, but that in reality subvert the security of the machine and cause malicious functions to be executed. This kind of attack does not usually trigger automated network security controls like intrusion detection systems. It can be discovered only by digging deep on the affected system for behavioural change.

Backdoor is a means to access a computer system or encrypted data that bypasses the system’s security mechanisms. Attackers often use back-doors that they detect or install themselves as part of exploit. In some cases, a worm or virus is designed to take advantage of a backdoor created by an earlier attack. In simple words, the default password can function as back-doors if they are not changed by the user. Another example would be some debugging features that can also act as back-doors if they are not removed in the release version.

Exploit is a piece of software, data or a sequence of commands that takes advantage of a bug or vulnerabilities to cause unanticipated behaviour. In simple words “to use something for one’s own advantage”.

rkhunter is a popular tool to detect such vulnerabilities and malwares . It can scan the entire system for rootkits and help us to defend ourselves against potential threats.

rkhunter

rkhunter (Rootkit Hunter) is a Unix-based tool that scans for rootkits, backdoors and possible local exploits. It checks the system against a database of known rootkits as well as perform the following security checks:

• Compare MD5 hash
• Look for default files used by rootkits
• Check wrong file permissions for binaries
• Look for suspected strings in LKM and KLD modules
• Look for hidden files
• Optional scan within plaintext and binary files

Install rkhunter on CentOS and RHEL

All files required for installation of rkhunter are contained in the EPEL repository.

$ yum install epel-release

Once the EPEL repository is installed, issue the following commands as root to start the installation routine and for updating the database.

$ sudo yum install rkhunter
$ rkhunter --update

Update system file properties

This is a necessary step to establish a foundation database file to compare scans. On a clean install, the first run of propupd creates a new database file. On later scans, running the propupd command updates the database file. Update the database file when you are satisfied you have only trusted source system file changes.

$ rkhunter --propupd

Automate Rootkit Hunter using CRON

rkhunter can be setup to run checks every day so that we always have up-to-date information about intrusions. This can be accomplished by creating a cronjob.

Create a cron file in the following location /etc/cron.daily/rkhunter.sh and place the following script in the file.

#!/bin/sh
(
/usr/local/bin/rkhunter --versioncheck
/usr/local/bin/rkhunter --update
/usr/local/bin/rkhunter --cronjob --report-warnings-only
) | /bin/mail -s 'rkhunter Daily Scan Report (PutYourServerNameHere)' your@email.here

Update the script with the email address to which the report should be sent.

• The --versioncheck checks if our threat definitions are up to date.
• The --update option updates our threat definitions, if required.
• The --cronjob option bypasses interactive key presses.

Set execute permission on the file you have just created

$ chmod 755 /etc/cron.daily/rkhunter.sh

The cron utility will run once daily and a report will be sent via email. Emails will be sent only if issues/problems surface. Otherwise, no emails will be sent.

Manual Scan

You can initiate a manual scan by issuing the following command:

rkhunter -c

which runs rkhunter in interactive mode. In other words, when it gets to the end of a particular scan, you need to press ‘enter’ to continue. If you want to “auto-skip” interactive mode, add the -sk option:

rkhunter -c -sk

Your scan results will look as follows:

System checks summary
=====================

File properties checks...
    Files checked: 130
    Suspect files: 0

Rootkit checks...
    Rootkits checked : 498
    Possible rootkits: 0

Applications checks...
    All checks skipped

The system checks took: 14 minutes and 34 seconds

All results have been written to the log file: /var/log/rkhunter/rkhunter.log

No warnings were found while checking the system.

Conclusion

If a rootkit has been installed in a compromised system, rebuilding the system is the best course of action. Because rootkits are so proficient in hiding themselves. Extremely strong monitoring and intrusion-detection/intrusion-prevention efforts need to be implemented. This article serves as an introductory tutorial to detection of rootkits.

References

• Knowledgebase
• What is backdoor (computing)? - Definition from WhatIs.com
• Exploits and exploit kits

A way to detect the rootkits and exploits in CentOS/RHEL was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

In a microservice environment handling huge workloads, a performance dip of a single microservice will mean the performance degradation of an entire application. A single weak link in the chain can endanger our SLAs. The SCM product at Logistimo handles mission-critical public-health supply chains for host of countries, via an array of microservices. Since the vitality of these systems is important at many levels, we take monitoring performance very seriously.

Our microservices are managed in a Kubernetes (K8s) cluster. The K8s ecosystem provides an array of options to monitor health and performance of the services. However, we wanted a solution which could monitor the performance of the application at the micro-service level. This brought us to the use of Elastic APM for our Kubernetes cluster. APM stands for Application Performance Monitoring. Deploying Elastic APM in Kubernetes ecosystem is a topic on which not much literature is found on the web. Hence, We decided to share our experience.

Elastic APM system:

If you are using the ELK stack( Elasticsearch, Logstash and Kibana ) for log aggregation in your infrastructure, you might be recording all the application logs in the Elasticsearch. While this in itself is useful, It is necessary to have APM monitoring for microservices. Elastic APM streams all events to Elastic search much like the Logstash.

Since data is stored as raw documents in Elasticsearch, your service metrics can coexist with your infrastructure logs making it easy to explore all of your data in one place. These metrics can be visualised in Kibana with pre-configured dashboards.

To do this APM metrics collection, you need two components:

1. APM server
2. APM agent

APM server

APM Server is a central component in the Elastic APM system. It receives data from APM agents and transforms them into Elastic search documents

A single APM server can handle data from multiple microservices. APM Server works by exposing an HTTP server to which agents post the APM data they collect.

To deploy the basic APM Server in K8s infrastructure, you can use following Kube-apm-server.yaml file:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-server-config
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"
      frontend:
        enabled: false
    setup.template.settings:
      index:
        number_of_shards: 1
        codec: best_compression
    setup.dashboards.enabled: true
    setup.kibana:
      host: "< kibana endpoint >"
    output.elasticsearch:
      hosts: ["< elasticsearch endpoint >"]
      indices:
        - index: "apm-%{[beat.version]}-sourcemap"
          when.contains:
            processor.event: "sourcemap"

- index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "error"

- index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "transaction"

- index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
          when.contains:
            processor.event: "span"
---
apiVersion: v1
kind: Service
metadata:
  name: apm-server
  labels:
    app: apm-server
spec:
  ports:
  - port: 8200
    targetPort: 8200
    name: http
    nodePort: 31000
  selector:
    app: apm-server
  type: NodePort
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: apm-server
spec:
  # this replicas value is default
  # modify it according to your case
  replicas: 1
  template:
    metadata:
      labels:
        app: apm-server
    spec:
      containers:
      - name: apm-server
        image: docker.elastic.co/apm/apm-server:6.3.1
        ports:
        - containerPort: 8200
          name: apm-port
        volumeMounts:
        - name: apm-server-config
          mountPath: /usr/share/apm-server/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: apm-server-config
        configMap:
          name: apm-server-config

Replace the host: “< Kibana endpoint >” and hosts: [“< Elasticsearch endpoint >”] with end points.

APM Server comes with predefined Kibana dashboards and index templates for APM data.

Following is the list of default dashboards available. You can also configure custom dashboards according to your needs.

1. Transactions
2. Span Details
3. Services
4. Errors
5. Error Details

These dashboards give you an instant overview of application response times, requests per minutes, error occurrences and more.

APM Agent

The agents are libraries that run inside your application process. Agents automatically instrument your application in order to measure the duration of requests to your service. It also automatically measures things like database queries, cache calls, external http requests and errors. Ensure you add APM agent to all of your micro services to get the metrics.

APM agent for Dockerized Java application:

The first step in getting started with the Elastic APM Java agent is to download the latest release of the agent jar file from maven central.

Add the following code in your Dockerfile to download and set the javaagent environment variable at the build time of the microservice.

RUN wget -P < $TOMCAT_HOME/ > http://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/<latest-version>/elastic-apm-agent-<latest-version>.jar

ENV JAVA_OPTS=-javaagent:$TOMCAT_HOME/elastic-apm-agent-< elastic-apm.version >.jar

Elastic APM agent has lot of configuration options.The most important ones are service_name (required), server_url, and application_packages.

System properties:

-Delastic.apm.service_name=my-service
-Delastic.apm.application_packages=org.example
-Delastic.apm.server_url=http://localhost:8200

The complete Dockerfile entry for apm-agent will look like this

RUN cd $TOMCAT_HOME && wget http://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/<latest-version>/elastic-apm-agent-<latest-version>.jar
 

ENV JAVA_OPTS=-javaagent:$TOMCAT_HOME/elastic-apm-agent-<latest-version>.jar \
              -Delastic.apm.service_name=< Service name > \
              -Delastic.apm.application_packages=< application pakage name > \
              -Delastic.apm.server_url=< APM server url/endpoint >

The service name is used by Elastic APM to differentiate data coming from different services and to group data coming from the same services. When configuring an agent, you need to supply a service name.

The service name can only contain alphanumeric characters, spaces, underscores, and dashes (must match ^[a-zA-Z0–9 _-]+$).

You can set the Service Name and APM server url as the docker environmental variable as the value for these properties changes per environment you deploy.

APM agent for Dockerized Node.js application

In Dockerfile add the follwoing line

RUN npm install elastic-apm-node --save

Agent should be started before you mark other modules as dependency using require in your Node.js application . You will include the following code to enable agent in your application’s main file (usually index.js, server.js or app.js).

// Add this to the VERY top of the first file loaded in your app
  var apm = require('elastic-apm-node').start({
    
    // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
    serviceName: '< Service Name>',

// Use if APM Server requires a token
    secretToken: '< Secret token >',

// Set custom APM Server URL (default: http://localhost:8200 )
    serverUrl: '< APM Server URL >',
  })

var app = require('express')()

Provided the correct value for the serviceName and serverUrl for the metrics to be collected.

End Note:

Elastic APM allowed us to avoid developing additional metering and instrumentation inside the micro services. APM agent takes care of monitoring all the key metrics. The data has been immensely useful for us to detect problems early and understand the state of the systems in production.

Happy Monitoring !!

Using Elastic APM in Kubernetes was originally published in Logistimo Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.