<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emir Mujic on Medium]]></title>
        <description><![CDATA[Stories by Emir Mujic on Medium]]></description>
        <link>https://medium.com/@emirmujic?source=rss-99664390f1d2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bAPIEduEzx3jjeIY_fwXDg.png</url>
            <title>Stories by Emir Mujic on Medium</title>
            <link>https://medium.com/@emirmujic?source=rss-99664390f1d2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 00:47:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@emirmujic/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Trigger AWS Lambda with CloudWatch Alarm: Terraform — Event Source Mapping]]></title>
            <link>https://medium.com/@emirmujic/trigger-aws-lambda-with-cloudwatch-alarm-terraform-event-source-mapping-1e6641f010c5?source=rss-99664390f1d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e6641f010c5</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[sns]]></category>
            <dc:creator><![CDATA[Emir Mujic]]></dc:creator>
            <pubDate>Tue, 26 Nov 2019 22:57:37 GMT</pubDate>
            <atom:updated>2019-11-26T22:57:37.380Z</atom:updated>
            <content:encoded><![CDATA[<h3>Trigger AWS Lambda with CloudWatch Alarm: Terraform — Event Source Mapping</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6KNQE145gfwjjus92fucA.jpeg" /></figure><h3>What is AWS Lambda?</h3><blockquote>AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume — there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service — all with zero administration.</blockquote><h3>AWS Lambda as part of your automation toolkit</h3><p>If your infrastructure is hosted on AWS, you may want to use AWS Lambda as part of your DevOps/SRE automation toolkit. In this article you will learn how to create simple automation using AWS services — CloudWatch Alarm, SNS, SQS and Lambda.</p><h3>Lambda — source of automation</h3><p>Let’s create automation that will remove EC2 from ELB, reboot it and after a timeout return it to ELB. And it’s all triggered by alert on CloudWatch. Simple, right?</p><pre><strong>#my_lambda.py</strong></pre><pre>import boto3<br>import os<br>import time<br>import logging<br>import json</pre><pre>## Prepare logger<br>logger = logging.getLogger()<br>logger.setLevel(logging.INFO)</pre><pre>## Initialize boto3 <br>elb = boto3.client(‘elb’)<br>ec2 = boto3.client(‘ec2’)</pre><pre>## Load environment <br>elbID = os.environ[‘elbID’]</pre><pre>def restart_service(event, context):<br>  logger.info(‘## SQS message arrived.’)<br>  message = event[‘Records’][0][‘body’]</pre><pre>  ## Parse message to retrieve instanceID<br>  message_json = json.loads(message)<br>  instanceID = message_json[“Trigger”][“Dimensions”][0][“value”]</pre><pre>  ## Get instance name<br>  instanceName = get_ec2_name(instanceID)<br>  logger.info(‘## Instance: ‘ + instanceName)</pre><pre>  ## Remove ec2 from load balancer<br>  logger.info(‘## Instance removed from LB — 5mins draining.’)<br>  remove_from_load_balancer(instanceID)</pre><pre>  ## Wait for elb draining process 5mins<br>  time.sleep(300)</pre><pre>  ## Proceed with restart<br>  logger.info(‘## Instance reboot started — 5mins timeout.’)<br>  ec2.reboot_instances(InstanceIds=[instanceID])</pre><pre>  ## Sleep 5mins after reboot<br>  time.sleep(300)</pre><pre>  ## Return ec2 to load balancer<br>  logger.info(‘## Instance added to LB.’)<br>  add_to_load_balancer(instanceID)</pre><pre>  ## Health check, retry every 20s<br>  while (check_instance_health(instanceID) != ‘InService’):<br>    time.sleep(20)<br>  logger.info(‘## All done, exiting.’)<br>  return</pre><pre>def remove_from_load_balancer(instanceID):<br>  elb.deregister_instances_from_load_balancer(<br>  LoadBalancerName=elbID, Instances=[{‘InstanceId’: instanceID}])<br>  return</pre><pre>def add_to_load_balancer(instanceID):<br>  elb.register_instances_with_load_balancer(<br>  LoadBalancerName=elbID, Instances=[{‘InstanceId’: instanceID}])<br>  return</pre><pre>def check_instance_health(instanceID):<br>  response = elb.describe_instance_health(<br>  LoadBalancerName=elbID, Instances=[{‘InstanceId’: instanceID}])<br>  return(response[“InstanceStates”][0][“State”])</pre><pre>def get_ec2_name(instanceID):<br>  response = ec2.describe_instances(InstanceIds=[instanceID])<br>  for tag in response[“Reservations”][0][“Instances”][0][“Tags”]:<br>    if tag[‘Key’] == ‘Name’:<br>      return tag[‘Value’]</pre><h3>Basic setup— Terraform</h3><p>The Golden Standard for <a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">IaC</a> is <a href="https://www.terraform.io/">Terraform</a>, so I won’t explain it deeply. Snippets are organized as a single module with scripts folder reserved for python script and dependencies.</p><p><strong>Create SQS Queue, SNS Topic and Policy</strong></p><p>In the first step let’s create SNS topic, SQS queue and attach a policy that allows topic to send messages to queue.</p><pre># Queue<br>resource &quot;aws_sqs_queue&quot; &quot;my_lambda_restart_queue&quot; {<br>  name = &quot;my_lambda_restart_queue&quot;<br>  max_message_size = 262144<br>  message_retention_seconds = 600<br>  receive_wait_time_seconds = 0<br>  visibility_timeout_seconds = 2400<br>}</pre><pre># Topic and Subscription<br>resource &quot;aws_sns_topic&quot; &quot;sns_my_lambda_restart_topic&quot; {<br>  name = &quot;my_lambda_restart&quot;<br>}</pre><pre>resource &quot;aws_sns_topic_subscription&quot; &quot;sns_my_lambda_restart_sub&quot; {<br>  topic_arn = &quot;${aws_sns_topic.sns_my_lambda_restart_topic.arn}&quot;<br>  protocol  = &quot;sqs&quot;<br>  endpoint  = &quot;${aws_sqs_queue.my_lambda_restart_queue.arn}&quot;<br>  raw_message_delivery = true<br>}</pre><pre># Policy<br>data &quot;template_file&quot; &quot;sqs-policy&quot; {<br>  template = &quot;${file(&quot;${path.module}/sqs_policy.json&quot;)}&quot;<br>  vars = {<br>    sqs_arn = &quot;${aws_sqs_queue.my_lambda_restart_queue.arn}&quot;<br>    sns_arn = &quot;${aws_sns_topic.sns_my_lambda_restart_topic.arn}&quot;<br>  }<br>}</pre><pre>resource &quot;aws_sqs_queue_policy&quot; &quot;my-lambda-restart-sqs-policy&quot; {<br>  queue_url = &quot;${aws_sqs_queue.my_lambda_restart_queue.id}&quot;<br>  policy = &quot;${data.template_file.sqs-policy.rendered}&quot;<br>}</pre><pre><strong>#sqs_polic.json</strong></pre><pre>{<br> “Version”: “2012–10–17”,<br> “Statement”: [<br>   {<br>     “Principal”: {<br>        “AWS”: “*”<br>      },<br>     “Effect”: “Allow”,<br>     “Action”: [<br>       “SQS:SendMessage”<br>      ],<br>      “Resource”: “${sqs_arn}”,<br>      “Condition”: {<br>        “ArnEquals”: {<br>             “aws:SourceArn”: “${sns_arn}”<br>        }<br>      }<br>    }<br>  ]<br>}</pre><p><strong>Create CloudWatch Alert</strong></p><p>Now, create CloudWatch Alert for each instance added to the load balancer. Note that alarm_action needs to be SNS topic created in chapter above.</p><pre># Required data<br>data &quot;aws_instances&quot; &quot;my_instances&quot; {<br>  filter {<br>    name   = &quot;tag:group&quot;<br>    values = [&quot;instances-tag&quot;]<br>  }<br>}</pre><pre># Cloudwatch Alert<br>resource &quot;aws_cloudwatch_metric_alarm&quot; &quot;high_network_transmit&quot; {<br>  count = &quot;${length(data.aws_instances.my_instances.ids)}&quot;<br>  alarm_name = &quot;${count.index} - Lambda Trigger&quot;<br>  comparison_operator = &quot;GreaterThanThreshold&quot;<br>  evaluation_periods  = 6<br>  metric_name = &quot;NetworkOut&quot;<br>  namespace = &quot;AWS/EC2&quot;<br>  statistic = &quot;Average&quot;<br>  period = 300<br>  threshold = 100000000<br>  treat_missing_data  = &quot;notBreaching&quot;<br>  alarm_actions=[&quot;${aws_sns_topic.sns_my_lambda_restart_topic.arn}&quot;]<br>  dimensions {<br>   InstanceId = &quot;${element(data.aws_instances.my_instances.ids, count.index)}&quot;<br>  }<br>}</pre><p><strong>Create AWS Role for Lambda</strong></p><p>Lambda needs access to SQS, to be able to change the state of EC2’s and manipulate ELB. So let’s create IAM role for lambda and attach the right policies.</p><pre>resource “aws_iam_role” “my-lambda-role” {<br>  assume_role_policy = &lt;&lt;EOF<br>  {<br>    “Version”: “2012–10–17”,<br>    “Statement”: [<br>      {<br>        “Effect”: “Allow”,<br>        “Principal”: {<br>          “Service”: “lambda.amazonaws.com”<br>        },<br>        “Action”: “sts:AssumeRole”<br>      }<br>    ]<br>  }<br>  EOF<br>}</pre><pre>resource “aws_iam_policy” “my-lambda-sqs-policy” {<br>  name = “my-lambda-sqs-policy”<br>  description = “Allow lambda function to read sqs message.”<br>  policy = “${data.template_file.sqs-receive-policy.rendered}”<br>}</pre><pre>resource “aws_iam_policy” “my-lambda-ec2-policy” {<br>  name = “my-lambda-ec2-policy”<br>  description = “Allow lambda function to manipulate ec2 instances.”<br>  policy = “${data.template_file.ec2-policy.rendered}”<br>}</pre><pre>resource “aws_iam_policy” “my-lambda-elb-policy” {<br>  name = “my-lambda-elb-policy”<br>  description = “Allow lambda function to manipulate elb.”<br>  policy = “${data.template_file.elb-logs-policy.rendered}”<br>}</pre><pre>data “template_file” “sqs-receive-policy” {<br>  template = “${file(“${path.module}/sqs-receive-policy.json”)}”<br>  vars = {<br>    sqs_arn = “${aws_sqs_queue.my_lambda_restart_queue.arn}”<br>  }<br>}</pre><pre>data “template_file” “ec2-policy” {<br>  template = “${file(“${path.module}/ec2-policy.json”)}”<br>}</pre><pre>data “template_file” “elb-policy” {<br>  template = “${file(“${path.module}/elb-policy.json”)}”<br>}</pre><pre>resource “aws_iam_role_policy_attachment” “role-policy-attachment-lambda-receive-sqs” {<br>  role = “${aws_iam_role.my-lambda-role.name}”<br>  policy_arn = “${aws_iam_policy.my-lambda-sqs-policy.arn}”<br>}</pre><pre>resource “aws_iam_role_policy_attachment” “role-policy-attachment-lambda-ec2” {<br>  role = “${aws_iam_role.my-lambda-role.name}”<br>  policy_arn = “${aws_iam_policy.my-lambda-ec2-policy.arn}”<br>}</pre><pre>resource “aws_iam_role_policy_attachment” “role-policy-attachment-lambda-elb” {<br>  role = “${aws_iam_role.my-lambda-role.name}”<br>  policy_arn = “${aws_iam_policy.my-lambda-elb-policy.arn}”<br>}</pre><pre>data &quot;aws_elb&quot; &quot;my-elb&quot; {<br>  name = &quot;my-elb&quot;}&quot;<br>}</pre><pre><strong>#ec2-policy.json<br></strong>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;ec2:DescribeInstances&quot;,<br>                &quot;ec2:DescribeInstanceStatus&quot;,<br>                &quot;ec2:StopInstances&quot;,<br>                &quot;ec2:StartInstances&quot;,<br>                &quot;ec2:RebootInstances&quot;<br>              ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><pre><strong>#sqs-receive-policy.json</strong></pre><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;sqs:ReceiveMessage&quot;,<br>                &quot;sqs:ChangeMessageVisibility&quot;,<br>                &quot;sqs:DeleteMessage&quot;,<br>                &quot;sqs:GetQueueAttributes&quot;<br>              ],<br>            &quot;Resource&quot;: &quot;${sqs_arn}&quot;<br>        }<br>    ]<br>}</pre><pre><strong>#elb-logs-policy.json</strong></pre><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Action&quot;: [<br>                &quot;elasticloadbalancing:DescribeInstanceHealth&quot;,                 &quot;elasticloadbalancing:RegisterInstancesWithLoadBalancer&quot;,<br>&quot;elasticloadbalancing:DeregisterInstancesFromLoadBalancer&quot;,<br>                &quot;logs:CreateLogGroup&quot;,<br>                &quot;logs:CreateLogStream&quot;,<br>                &quot;logs:PutLogEvents&quot;<br>              ],<br>            &quot;Resource&quot;: &quot;*&quot;<br>        }<br>    ]<br>}</pre><p><strong>Create Lambda</strong></p><p>To ensure we have at least one instance in ELB we need to set reserved_concurrent_executions to value = 1<strong> </strong>to prevent running concurrent lambdas.<strong> </strong>Also have in mind that max allowed execution timeout for AWS lambda is <strong>15 minutes</strong>.</p><pre>data &quot;archive_file&quot; &quot;source&quot; {<br>  type        = &quot;zip&quot;<br>  source_dir  = &quot;${path.module}/scripts/&quot;<br>  output_path = &quot;${path.module}/lambda_function.zip&quot;<br>}</pre><pre>resource &quot;aws_lambda_function&quot; &quot;my-lambda-restart&quot; {<br>  filename = &quot;${substr(data.archive_file.source.output_path, length(path.cwd) + 1, -1)}&quot;<br>  function_name = &quot;my_lambda&quot;<br>  description = &quot;Restarts service when notified by CloudWatch Alert&quot;<br>  role = &quot;${aws_iam_role.my-lambda-role.arn}&quot;<br>  handler = &quot;lambda_function.restart_service&quot;<br>  source_code_hash = &quot;${data.archive_file.source.output_base64sha256}&quot;<br>  runtime = &quot;python3.6&quot;<br>  reserved_concurrent_executions = 1<br>  timeout = 900</pre><pre>environment {<br>    variables = {<br>      elbID = &quot;${data.aws_elb.my-elb.name}&quot;<br>    }<br>  }<br>}</pre><p><strong>Create Lambda event — trigger</strong></p><p>Last step is to create aws_lambda_event_source_mapping resource that will create event which will invoke lambda on every SQS message in the queue.</p><pre>resource &quot;aws_lambda_event_source_mapping&quot; &quot;my_lambda_restart_mapping&quot; {<br>  batch_size = 1<br>  event_source_arn = &quot;${aws_sqs_queue.my_lambda_restart_queue.arn}&quot;<br>  enabled = true<br>  function_name = &quot;${aws_lambda_function.my-lambda-restart.arn}&quot;<br>}</pre><p>Now we just need to apply new infrastructure to state and automation is ready to go. Also you can run terraform fmt to format *.tf files if you plan to use snippets above.</p><p>In this article I used <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">boto3</a> AWS SDK for python for EC2 and ELB manipulation. Boto provides an easy to use, object-oriented API, as well as low-level access to AWS services and it’s a great tool for automation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BWBGu-TQ63NR7GgpEdNhg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e6641f010c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collect cluster  and Istio metrics with Prometheus Operator]]></title>
            <link>https://medium.com/@emirmujic/collect-cluster-and-istio-metrics-with-prometheus-operator-42d4499d621a?source=rss-99664390f1d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/42d4499d621a</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Emir Mujic]]></dc:creator>
            <pubDate>Fri, 14 Dec 2018 17:33:06 GMT</pubDate>
            <atom:updated>2018-12-14T17:33:06.867Z</atom:updated>
            <content:encoded><![CDATA[<h3>Collect cluster and Istio metrics with Prometheus Operator</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*njh1UGa1q5a82hhjhfwjcg.png" /></figure><p>To have a Kubernetes cluster up and running is pretty easy, but as K8S being a distributed system is not easy to troubleshoot. You need a good monitoring solution and the <a href="https://prometheus.io/">Prometheus</a> is probably the best fit.</p><h4>Istio and metrics</h4><p>As you probably know, Istio comes with Prometheus and Grafana out-of-box, but in this article I’ll show you how to get Istio metrics with Prometheus Operator. More about Istio and how to set it up you can read in my other article : <a href="https://medium.com/@emirmujic/istio-and-metallb-on-minikube-242281b1134b"><strong>Istio and MetalLB on Minikube </strong></a>.</p><h4>Prometheus Operator</h4><blockquote>The Prometheus Operator is simple to install with a single command line, and enables users to configure and manage instances of Prometheus using simple declarative configuration that will, in response, create, configure, and manage Prometheus monitoring instances. <a href="https://coreos.com/blog/the-prometheus-operator.html">CoreOS Blog</a></blockquote><p>In simple words, you can deploy it and create a ServiceMonitor resource which will scrape the Prometheus metrics from the defined set of pods. Basically, it instructs Prometheus to watch on a new target.</p><h4><strong>Let’s deploy</strong></h4><p>For Prometheus installation use the official Helm chart <a href="https://github.com/helm/charts/tree/master/stable/prometheus-operator">prometheus-operator</a>. This chart has a lot of options, so I encourage you to take a look at <a href="https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml">default values file</a> for further customisation. Among other services, this chart installs <a href="https://grafana.com/">Grafana</a> and <a href="https://prometheus.io/docs/instrumenting/exporters/">exporters</a> ready to monitor your cluster.</p><p>To enable scraping from Istio you need to create additional scraping jobs. Also to enable persistent storage for Prometheus and Grafana you need to pass some additional config to Helm during installation.</p><p>Before running helm install lets create customConfig.yaml with additional configuration for Istio metrics scraping and persistent storage for Prometheus and Grafana.</p><pre>prometheus:<br>  prometheusSpec:<br>    storageSpec:<br>      volumeClaimTemplate:<br>        spec:<br>          storageClassName: standard<br>          accessModes: [&quot;ReadWriteOnce&quot;]<br>          resources:<br>            requests:<br>            storage: 10Gi<br>    additionalScrapeConfigs:<br>    - job_name: &#39;istio-mesh&#39;<br>      scrape_interval: 5s<br>      kubernetes_sd_configs:<br>      - role: endpoints<br>        namespaces:<br>          names:<br>          - istio-system<br>      relabel_configs:<br>      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]<br>        action: keep<br>        regex: istio-telemetry;prometheus<br>    - job_name: &#39;envy&#39;<br>      scrape_interval: 5s<br>      kubernetes_sd_configs:<br>      - role: endpoints<br>        namespaces:<br>          names:<br>          - istio-system<br>      relabel_configs:<br>      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]<br>        action: keep<br>        regex: istio-statsd-prom-bridge;statsd-prom<br>    - job_name: &#39;istio-policy&#39;<br>      scrape_interval: 5s<br>      kubernetes_sd_configs:<br>      - role: endpoints<br>        namespaces:<br>          names:<br>          - istio-system<br>      relabel_configs:<br>      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]<br>        action: keep<br>        regex: istio-policy;http-monitoring<br>    - job_name: &#39;istio-telemetry&#39;<br>      scrape_interval: 5s<br>      kubernetes_sd_configs:<br>      - role: endpoints<br>        namespaces:<br>          names:<br>          - istio-system<br>      relabel_configs:<br>      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]<br>        action: keep<br>        regex: istio-telemetry;http-monitoring<br>    - job_name: &#39;pilot&#39;<br>      scrape_interval: 5s<br>      kubernetes_sd_configs:<br>      - role: endpoints<br>        namespaces:<br>          names:<br>          - istio-system<br>      relabel_configs:<br>      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]<br>        action: keep<br>        regex: istio-pilot;http-monitoring<br>    - job_name: &#39;galley&#39;<br>      scrape_interval: 5s<br>      kubernetes_sd_configs:<br>      - role: endpoints<br>        namespaces:<br>          names:<br>          - istio-system<br>      relabel_configs:<br>      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]<br>        action: keep<br>        regex: istio-galley;http-monitoring<br>grafana:<br>  adminPassword: &quot;YourSuperPass1234=&quot;<br>  persistence:<br>    enabled: true<br>    accessModes: [&quot;ReadWriteOnce&quot;]<br>    size: 10Gi</pre><p>Now, install Prometheus Operator and Prometheus with all dependencies:</p><pre>$ helm install \<br>    --name prom \<br>    --namespace monitoring \     <br>    -f customConfig.yaml \     <br>    stable/prometheus-operator</pre><p>Wait a few minutes and the whole stack should be up-and-running. Check for Helm release status with$ helm status prom and check for all pods in monitoring namespace with $ kubectl get pods -n monitoring. Final step is to create serviceMonitor for Istio and it looks like this:</p><pre>apiVersion: monitoring.coreos.com/v1<br>kind: ServiceMonitor<br>metadata:<br>  name: istio-monitor<br>  namespace: monitoring<br>  labels:<br>    monitoring: services<br>spec:<br>  selector:<br>    matchExpressions:<br>      - {key: istio, operator: In, values: [pilot, mixer]}<br>  namespaceSelector:<br>    matchNames:<br>      - istio-system<br>  jobLabel: istio<br>  endpoints:<br>  - port: prometheus <br>    interval: 5s   <br>  - port: http-monitoring <br>    interval: 5s    <br>  - port: statsd-prom <br>    interval: 5s</pre><p>Save it to serviceMonitor.yaml and just apply it with kubectl:</p><pre>$ kubectl apply -f serviceMonitor.yaml</pre><p>After everything is up-and-running, do port forwarding to enable access to your Grafana:</p><pre>$ kubectl port-forward -n monitoring prom-grafana-some_awesome_id 3000</pre><p>Visit <a href="http://localhost:3000/"><strong>http://localhost:3000</strong></a> and login to your Grafana with username admin and password YourSuperPass1234= . When you login to Grafana default dashboards will be present. To see Istio metrics we need to import Istio dashboards.</p><h4>Import Istio dashboards on Grafana</h4><p>To get Istio metrics inside Grafana, you need to import suitable dashboards. Simplest way is to get Istio dashboards from <a href="https://github.com/istio/istio/tree/master/install/kubernetes/helm/subcharts/grafana/dashboards">official Istio Helm repo</a>, and manually import them into Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-sgEZOJp_S6jHEy1EEDDvQ.png" /></figure><p>Another approach is to create ConfigMap from every single Istio dashboard json file, and Prometheus Operator will import those configurations into Grafana for you.</p><pre>$ kubectl create configmap -n monitoring grafana-dashboard-service --from-file=./dashboards/istio-service-dashboard.json --dry-run -o yaml &gt; dashboards-confmaps/istio-service-dashboard.yaml</pre><p>Before you apply any of those ConfigMaps, check that all have name starting with grafana-dashboard and label grafana_dashboard with value of 1.</p><pre>metadata:<br>  name: grafana-dashboard-service<br>  labels:<br>    grafana_dashboard: &quot;1&quot;</pre><p>If you try to import them in one single ConfigMap containing all dashboards, you will get next error The ConfigMap “grafana-dashboard” is invalid: metadata.annotations: Too long: must have at most 262144 characters. File is just too big to handle.</p><p>Alternative for all this manual work is to use awesome tool named wizzy to import those dashboards. wizzy is a rich user-friendly command line tool written in node.js to manage Grafana entities. More about, and how to use it, you can read <a href="https://utkarshcmu.github.io/wizzy-site/home/">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42d4499d621a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Istio and MetalLB on Minikube]]></title>
            <link>https://medium.com/@emirmujic/istio-and-metallb-on-minikube-242281b1134b?source=rss-99664390f1d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/242281b1134b</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Emir Mujic]]></dc:creator>
            <pubDate>Wed, 24 Oct 2018 14:31:12 GMT</pubDate>
            <atom:updated>2018-10-24T14:31:12.412Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzNP8GpOmURbr3aFlLCcIA.jpeg" /></figure><p><strong>Minikube</strong> is a tool that makes it easy to run K8S locally, but that is only part of the puzzle. We want that our local dev environment looks like production env as much as possible. As <strong>Istio</strong> becomes more popular and widely used I’ll try to explain how to set it up with <strong>MetalLB</strong> — bare metal load-balancer for K8S.</p><h3>Istio</h3><p>Service mesh is a configurable infrastructure layer for microservice oriented applications. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application. In practice, the service mesh is typically implemented as an array of lightweight network proxies that are deployed alongside application code. The mesh provides service discovery, load balancing, encryption, authentication and authorisation, support for the circuit breaker pattern, and other capabilities.</p><p>The service mesh is usually implemented by providing a proxy instance, called a sidecar, for each service instance. Sidecars handle inter‑service communications, monitoring, security‑related concerns — anything that can be abstracted away from the individual services. This way, developers can handle development, support, and maintenance for the application code in the services; operations can maintain the service mesh and run the app.</p><p><strong>Istio</strong>, backed by Google, IBM, and Lyft, is currently the best‑known service mesh architecture. <strong>Kubernetes</strong>, which was originally designed by Google, is currently the only container orchestration framework supported by Istio.</p><h4>Installing Istio</h4><p>Istio will be installed in it’s own directory (current version is 1.0.2)</p><pre>$ curl -L <a href="https://git.io/getLatestIstio">https://git.io/getLatestIstio</a> | sh -<br>$ cd ~/istio-1.0.2<br>$ sudo mv bin/istioctl /usr/local/bin</pre><p>After</p><pre>$ helm init</pre><p>is completed, Istio can be installed in <strong>istio-system</strong> namespace via helm charts:</p><pre>$ cd ~/istio-1.0.2<br>$ helm install install/kubernetes/helm/istio --name istio --namespace istio-system</pre><p>Don’t forget to label the namespace for injection:</p><pre>$ kubectl label namespace default istio-injection=enabled</pre><p>Afterwards, namespace pods can be checked with:</p><pre>$ kubectl get pods --namespace=istio-system</pre><p>Now check <strong>istio-ingressgateway </strong>service,</p><pre>kubectl get service -n istio-system istio-ingressgateway</pre><p>similar output should appear:</p><pre>NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP                                                                                                             <br>istio-ingressgateway   LoadBalancer   10.104.226.154   &lt;pending&gt;</pre><p>Notice that <strong>EXTERNAL-IP</strong> is in <strong>&lt;pending&gt;</strong> state, so that’s the part when MetalLB comes in.</p><h3>MetalLB</h3><p>If a Kubernetes cluster is installed on <em>bare metal</em> or on virtual machines, it’s missing an external load balancer. It’s a cloud network load balancer that comes with platforms such as Azure or GCM, that provides an externally-accessible IP address that sends traffic to the correct port on your cluster nodes. MetalLB is a<strong> load balancer</strong> designed to run on and to work with Kubernetes and it will allow you to use the type LoadBalancer when you declare a service.</p><p><em>A LoadBalancer service is the standard way to expose a service to the internet. It will give you a single IP address that will forward all traffic to your service.</em></p><h4>Install MetalLB on Minikube</h4><p>MetalLB runs in two parts: a cluster-wide <strong>controller</strong>, and a per-machine protocol <strong>speaker</strong>. Install MetalLB by applying the manifest:</p><pre>$ kubectl apply -f <a href="https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml">https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml</a></pre><p>This manifest creates a bunch of resources. Most of them are related to access control, so that MetalLB can read and write the Kubernetes objects it needs to do its job.</p><p>The two pieces of interest are the “<strong>controller</strong>” deployment, and the “<strong>speaker</strong>” DaemonSet. By monitoring:</p><pre>$ kubectl get pods -n metallb-system</pre><p>similar output should appear:</p><pre>NAME                          READY     STATUS    RESTARTS   AGE<br>controller-7fbd769fcc-58fm8   1/1       Running   0          50m<br>speaker-rctgd                 1/1       Running   0          50m</pre><p>In real environment one controller pod and one speaker pod for each node should appear in a cluster, but using Minikube, only one controller and one speaker show up.</p><h4>Apply ConfigMap</h4><p>MetalLB’s configuration is a standard Kubernetes ConfigMap, config under the <strong>metallb-system</strong> namespace. It contains two pieces of information: what IP addresses it’s allowed to hand out and which protocol to use for such task.<br>In this configuration MetalLB is instructed to hand out address from the 192.168.99.100/28 range, using <strong>layer 2 mode</strong> (protocol: <strong>layer2</strong>). <br>In local development, Minikube IP address is used as the start of the range. To get Minikube IP address use:</p><pre>$ minikube ip</pre><p>Result example:</p><pre>192.168.99.100</pre><p>Apply this configuration:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  namespace: metallb-system<br>  name: config<br>data:<br>  config: |<br>    address-pools:<br>    - name: custom-ip-space<br>      protocol: layer2<br>      addresses:<br>      - 192.168.99.100/28</pre><p>Now check <strong>istio-ingressgateway </strong>service again,</p><pre>kubectl get service -n istio-system istio-ingressgateway</pre><p>similar output should appear:</p><pre>NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP                                                                                                             <br>istio-ingressgateway   LoadBalancer   10.104.226.154   <strong>192.168.99.96</strong></pre><p>MetalLB is used here in order to provide an IP address for the Istio IngressGateway service of LoadBalancer type thus emulating a cloud deployment. This is used to avoid NodePort implementations with Istio as they are not to be used due to Istio design.</p><h3>MetalLB and Istio example: Using an Istio Gateway</h3><p>In a Kubernetes environment, the Kubernetes Ingress Resource is used to specify services that should be exposed outside the cluster. In an Istio service mesh, a better approach (which also works in both Kubernetes and other environments) is to use a different configuration model, namely Istio Gateway. A Gateway allows Istio features such as monitoring and routing rules to be applied to traffic entering the cluster.</p><h4>Before you begin</h4><ul><li>Setup <strong>Istio</strong> and <strong>MetalLB</strong> by following the instructions in sections above.</li><li>Make sure your current directory is the <strong>istio directory</strong>.</li></ul><p>Start the <strong>httpbin sample</strong>, which will be used as the destination service to be exposed externally.</p><pre>$ kubectl apply -f &lt;(istioctl kube-inject -f samples/httpbin/httpbin.yaml)</pre><p>Execute the following command to determine if your Kubernetes cluster is running in an environment that supports external load balancers:</p><pre>$ kubectl get svc istio-ingressgateway -n istio-system</pre><p>Check EXTERNAL-IP column for address of gateway:</p><pre>NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP   <br>istio-ingressgateway   LoadBalancer   10.104.226.154  192.168.99.96</pre><p>If the EXTERNAL-IP value is set, your environment has an external load balancer that you can use for the ingress gateway, in this case MetalLB. If the <strong>EXTERNAL-IP</strong> value is <strong>&lt;none&gt;</strong> (or perpetually <strong>&lt;pending&gt;</strong>), your environment does not provide an external load balancer for the ingress gateway, in other words, <strong>check MetalLB installation</strong>.</p><h4>Configure ingress using an Istio Gateway</h4><p>An ingress Gateway describes a load balancer operating at the edge of the mesh that receives incoming HTTP/TCP connections. It configures exposed ports, protocols, etc. but, unlike Kubernetes Ingress Resources, does not include any traffic routing configuration. Traffic routing for ingress traffic is configured using Istio routing rules instead, exactly in the same way as for internal service requests.</p><ul><li>Create an Istio Gateway:</li></ul><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: Gateway<br>metadata:<br>  name: httpbin-gateway<br>spec:<br>  selector:<br>    istio: ingressgateway # use Istio default gateway implementation<br>  servers:<br>  - port:<br>      number: 80<br>      name: http<br>      protocol: HTTP<br>    hosts:<br>    - &quot;*&quot;</pre><ul><li>Configure routes for traffic entering via the Gateway:</li></ul><pre>apiVersion: networking.istio.io/v1alpha3<br>kind: VirtualService<br>metadata:<br>  name: httpbin<br>spec:<br>  hosts:<br>  - &quot;*&quot;<br>  gateways:<br>  - httpbin-gateway<br>  http:<br>  - match:<br>    - uri:<br>        prefix: /<br>    route:<br>    - destination:<br>        port:<br>          number: 8000<br>        host: httpbin</pre><p>The Gateway configuration resources allow external traffic to enter the Istio service mesh and make the traffic management and policy features of Istio available for edge services.</p><p>In the preceding steps, service is created inside the service mesh and exposed an HTTP endpoint of the service to external traffic.</p><p>Visit <a href="http://192.168.99.100/headers"><em>http://192.168.99.96:80/</em></a> and see <strong>httpbin</strong> magic.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=242281b1134b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>