<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohamed Saeed on Medium]]></title>
        <description><![CDATA[Stories by Mohamed Saeed on Medium]]></description>
        <link>https://medium.com/@mohamedmsaeed?source=rss-1748c840df1f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*H-KcSCDPNg30IYPF</url>
            <title>Stories by Mohamed Saeed on Medium</title>
            <link>https://medium.com/@mohamedmsaeed?source=rss-1748c840df1f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:50:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohamedmsaeed/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Memory_working_set vs Memory_rss in Kubernetes, which one you should monitor?]]></title>
            <link>https://mohamedmsaeed.medium.com/memory-working-set-vs-memory-rss-in-kubernetes-which-one-you-should-monitor-8ef77bf0acee?source=rss-1748c840df1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ef77bf0acee</guid>
            <category><![CDATA[memories]]></category>
            <category><![CDATA[oomkill]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Mohamed Saeed]]></dc:creator>
            <pubDate>Wed, 29 Jul 2020 15:56:53 GMT</pubDate>
            <atom:updated>2020-07-29T15:56:53.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1BccTr64au0WmP7eemZ5ZA.png" /></figure><p>In this article, we will know how cAdvisor collects memory_working_set and memory_rss metrics. Which one we should monitor, and which one causes OOMkill if we applied memory limits to your pods.</p><h3>Before we get to the difference:</h3><p>If you exec inside your container and navigate to the following directory, you can find all the container’s memory information you may need, usage, limits, cache, how many time your container got oomkilled, ..etc:</p><pre>cd /sys/fs/cgroups/memory/</pre><p>List the content of this directory and you will get the following control files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FSKjvu-UKo3C0FRzbwJqvQ.png" /></figure><p>Each file contains a piece of information about the memory. You can read more about them <a href="https://lwn.net/Articles/432224/">here</a>.</p><h4>What we are really interested in now is memory.status file:</h4><pre>root@test:/sys/fs/cgroup/memory# cat memory.stat<br>cache 32575488<br>rss 33964032<br>rss_huge 0<br>shmem 1757184<br>mapped_file 16625664<br>dirty 0<br>writeback 0<br>pgpgin 19272<br>pgpgout 2871<br>pgfault 16137<br>pgmajfault 66<br>inactive_anon 1757184<br>active_anon 33927168<br>inactive_file 27709440<br>active_file 2433024<br>unevictable 0<br>hierarchical_memory_limit 4089446400<br>total_cache 32575488<br>total_rss 33964032<br>total_rss_huge 0<br>total_shmem 1757184<br>total_mapped_file 16625664<br>total_dirty 0<br>total_writeback 0<br>total_pgpgin 19272<br>total_pgpgout 2871<br>total_pgfault 16137<br>total_pgmajfault 66<br>total_inactive_anon 1757184<br>total_active_anon 33927168<br>total_inactive_file 27709440<br>total_active_file 2433024<br>total_unevictable 0</pre><p>cAdvisor gathers those numbers and uses them to calculate memory metrics which are collected by Prometheus.</p><h3>The difference between “container_memory_working_set_bytes” and “container_memory_rss”:</h3><h4>container_memory_rss:</h4><p>From <a href="https://github.com/google/cadvisor/blob/50b23f4ed9bc53cf068316b67bee04c4145f1e73/info/v1/container.go#L379">cAdvisor code</a>, the memory RSS is:</p><blockquote>The amount of anonymous and swap cache memory (includes transparent hugepages).</blockquote><p>and it equals to the value of <strong>total_rss</strong> from <strong>memory.status</strong> file, find the code <a href="https://github.com/google/cadvisor/blob/50b23f4ed9bc53cf068316b67bee04c4145f1e73/container/libcontainer/handler.go#L839">here</a>:</p><pre>ret.Memory.RSS = s.MemoryStats.Stats[&quot;total_rss&quot;]</pre><p>Don’t confuse <strong>RSS</strong> with ‘<strong>resident set size</strong>’. From kernel <a href="https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v1/memory.html">documentation</a>:</p><blockquote>Note:<br>Only anonymous and swap cache memory is listed as part of ‘rss’ stat.<br> This should not be confused with the true ‘resident set size’ or the<br> amount of physical memory used by the cgroup.<br> ‘rss + file_mapped” will give you resident set size of cgroup.<br> (Note: file and shmem may be shared among other cgroups. In that case,<br> file_mapped is accounted only when the memory cgroup is owner of page<br> cache.)</blockquote><h4>container_memory_working_set_bytes:</h4><p>From <a href="https://github.com/google/cadvisor/blob/50b23f4ed9bc53cf068316b67bee04c4145f1e73/info/v1/container.go#L391">cAdvisor code</a>, they define the working set memory as:</p><blockquote>The amount of working set memory, this includes recently accessed memory,<a href="https://superuser.com/questions/1317468/what-is-dirty-memory-and-how-to-deal-with-it#:~:text=&#39;Dirty&#39;%20memory%20is%20memory%20representing,written%20out%20to%20disk%20yet.">dirty memory</a>, and kernel memory. Working set is &lt;= &quot;usage&quot;.</blockquote><p>and it equals to Usage minus total_inactive_file, find the code <a href="https://github.com/google/cadvisor/blob/50b23f4ed9bc53cf068316b67bee04c4145f1e73/container/libcontainer/handler.go#L862">here</a>:</p><pre>inactiveFileKeyName := &quot;total_inactive_file&quot;<br> if cgroups.IsCgroup2UnifiedMode() {<br>  inactiveFileKeyName = &quot;inactive_file&quot;<br> }</pre><pre>workingSet := ret.Memory.Usage<br> if v, ok := s.MemoryStats.Stats[inactiveFileKeyName]; ok {<br>  if workingSet &lt; v {<br>   workingSet = 0<br>  } else {<br>   workingSet -= v<br>  }<br> }<br>ret.Memory.WorkingSet = workingSet</pre><h4><strong>Note:</strong></h4><p>The value from kubectl top pods the command is from container_memory_working_set_bytesmetric.</p><h3>Which one is important to monitor?</h3><p>Now we know how cAdviser gets ‘<strong>RSS</strong>’ and ‘<strong>workin_set_bytes</strong>’ values, which one to monitor?</p><p>It is worth mentioning that if you are using resource limits on your pods, then you need to monitor <strong>both</strong> of them to prevent your pods from being <strong>oom-killed</strong>.</p><p>If ‘container_memory_rss’ increased to the limits -&gt; oomkill. <br>And if ‘container_memory_working_set_bytes’ increased to the limits -&gt; oomkill.</p><h3>Useful Articles:</h3><ul><li>Know more about dirty memory <a href="https://superuser.com/questions/1317468/what-is-dirty-memory-and-how-to-deal-with-it#:~:text=&#39;Dirty&#39;%20memory%20is%20memory%20representing,written%20out%20to%20disk%20yet.">here</a>.</li><li>Anonymous memory <a href="https://stackoverflow.com/a/22456025/9283447">here</a>.</li><li><a href="https://medium.com/faun/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d">https://medium.com/faun/how-much-is-too-much-the-linux-oomkiller-and-used-memory-d32186f29c9d</a></li><li><a href="https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-b190cc97f0f6">https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-b190cc97f0f6</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ef77bf0acee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Zipkin to Trace Requests in Flask Application]]></title>
            <link>https://mohamedmsaeed.medium.com/use-zipkin-to-trace-requests-in-flask-application-68886f02e46?source=rss-1748c840df1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/68886f02e46</guid>
            <category><![CDATA[distributed-tracing]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[zipkin]]></category>
            <category><![CDATA[request-tracing]]></category>
            <dc:creator><![CDATA[Mohamed Saeed]]></dc:creator>
            <pubDate>Tue, 10 Mar 2020 20:25:18 GMT</pubDate>
            <atom:updated>2020-03-10T20:25:18.647Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I will show you how to trace requests from Flask APIs using Zipkin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d6BuViewoyHRVUTtJECPTg.png" /><figcaption>The request’s trace.</figcaption></figure><p>All services are containerized and a docker-compose file is used to bring them all up.</p><p>You can find the code in this <a href="https://github.com/MohamedMSaeed/tracing_flask_zipkin">GitHub repository</a>.</p><h3>Used Tools</h3><h4>Zipkin</h4><p>Zipkin is a distributed tracing system. Using Zipkin will make you able to see all the services that a request goes through, determine the duration time needed for each service.</p><p><strong>What is a distributed tracing system:</strong></p><p>From <a href="https://opentracing.io/docs/overview/what-is-tracing/">OpenTracing website</a>:<br>Distributed tracing, also called distributed request tracing, is a method used to profile and monitor applications, especially those built using a microservices architecture. Distributed tracing helps pinpoint where failures occur and what causes poor performance.</p><h4>Flask</h4><p>Flask is a lightweight WSGI framework. It is one of the most popular Python frameworks.</p><p><em>Here is their </em><a href="https://flask.palletsprojects.com/en/1.1.x/quickstart/"><em>documentation</em></a><em>.</em></p><h4>py_zipkin</h4><p>py_zipkin provides a context manager/decorator along with some utilities to facilitate the usage of Zipkin in Python applications.</p><p><em>You can read more about py_zipkin </em><a href="https://github.com/Yelp/py_zipkin"><em>here</em></a><em>.</em></p><blockquote>We use py_zipkin here as the application is a Python app. If you are interested in other languages, you can find a list of supported tracers by Zipkin <a href="https://zipkin.io/pages/tracers_instrumentation.html">here</a>.</blockquote><h4>Docker Compose</h4><p>Docker Compose is a tool that you can use to define and run multi docker containers. You use a YAML file to define all services and with just one command you can start them all.</p><p><em>To learn more about Compose, see their </em><a href="https://docs.docker.com/compose/"><em>documentation</em></a><em>.</em></p><h3>The code</h3><p>As shown in the following image, <strong>api_01</strong> will send two requests, one to <strong>api_02</strong> and the other to <strong>api_03</strong>. <strong>api_02</strong> will send a request to <strong>api_03</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJBmBYPBrCVwbbfDucCmtw.png" /></figure><h4>Dockerfile</h4><p>Each API has its own Dockerfile. We build and ship the app environment using containers without the need to install or prepare the app env on the machine itself. So you don’t really need to install Python on your laptop.</p><pre>From python:3.7<br><br>COPY ./requirements.txt /app/requirements.txt<br><br>WORKDIR /app<br><br>RUN pip install -r requirements.txt<br><br>ENTRYPOINT [ &quot;python&quot; ]<br><br>CMD [ &quot;app.py&quot; ]</pre><p>In the requirements.txt file add</p><pre>flask==1.1.1<br>requests<br>py_zipkin</pre><h4>docker-compose file</h4><p>It is a very basic docker-compose.yaml file. Contests of four services, a Zipkin server, and 3 flask APIs.</p><pre>version: &quot;3.4&quot;<br><br>services:<br>  zipkin:<br>    image: openzipkin/zipkin:latest<br>    container_name: zipkin<br>    ports:<br>      - &quot;9411:9411&quot;<br><br><br>  api_01:<br>    build:<br>      context: api_01<br>    container_name: api_01<br>    ports:<br>      - &quot;5001:5000&quot;<br>    environment:<br>      - &#39;ZIPKIN_DSN=http://zipkin:9411/api/v1/spans&#39;<br>    volumes:<br>      - &#39;./api_01:/app&#39;<br><br>...</pre><p>All APIs’ code is mounted inside the containers. If you changed the code it will affect directly. I made it this way for development porpoises.</p><h4>api_01</h4><p>Import some libraries.</p><pre>from flask import Flask, request<br>import requests<br>from py_zipkin.zipkin import zipkin_span, create_http_headers_for_new_span, ZipkinAttrs, Kind, zipkin_client_span<br>from py_zipkin.request_helpers import create_http_headers<br>from py_zipkin.encoding import Encoding</pre><p>To start a Flask server:</p><pre>app = Flask(__name__)</pre><pre># Start the app server<br>if __name__ == &#39;__main__&#39;:<br>    app.run(debug=True, host=&#39;0.0.0.0&#39;, threaded=True)</pre><p>Start a Zipkin trace when a request is made to / route:</p><pre>@app.route(&#39;/&#39;)<br>def index():<br>    with zipkin_span(<br>        service_name=&#39;api_01&#39;,<br>        span_name=&#39;index_api_01&#39;,<br>        transport_handler=default_handler,<br>        port=5000,<br>        sample_rate=100,<br>        encoding=Encoding.V2_JSON<br>    ):<br>        call_api_02()<br>        call_api_03()<br>    return &#39;OK&#39;, 200</pre><p>zipkin_span will start a trace and creates the first span that is called “index_api_01”, then we will call the 2 functions call_api_02 and call_api_03. Each function will make a GET request to an API.</p><pre>@zipkin_client_span(service_name=&#39;api_01&#39;, span_name=&#39;call_api_02&#39;)<br>def call_api_02():<br>    headers = create_http_headers()<br>    requests.get(&#39;http://api_02:5000/&#39;, headers=headers)<br>    return &#39;OK&#39;<br><br><br>@zipkin_client_span(service_name=&#39;api_01&#39;, span_name=&#39;call_api_03_FROM_01&#39;)<br>def call_api_03():<br>    headers = create_http_headers()<br>    requests.get(&#39;http://api_03:5000/&#39;, headers=headers)<br>    return &#39;OK&#39;</pre><p>Notice that before calling the function a client span is created:</p><pre>@zipkin_client_span(service_name=&#39;api_01&#39;, span_name=&#39;call_api_02&#39;)</pre><p>Now we need to pass traceID and parent spanID to the next service to keep tracking the request. This can be done by passing headers that carry this information. In addition, it creates a new spanID for the next span.</p><pre>headers = create_http_headers()</pre><h4>api_02</h4><p>In api_02 we are expecting that this API can only be called using api_01. Also, we know that api_01 will send a request with headers that contain tracer information. So we can start our Zipkin trace as the following</p><pre>@app.route(&#39;/&#39;)<br>def index():<br>    with zipkin_span(<br>        service_name=&#39;api_02&#39;,<br>        zipkin_attrs=ZipkinAttrs(<br>            trace_id=request.headers[&#39;X-B3-TraceID&#39;],<br>            span_id=request.headers[&#39;X-B3-SpanID&#39;],<br>            parent_span_id=request.headers[&#39;X-B3-ParentSpanID&#39;],<br>            flags=1,<br>            is_sampled=request.headers[&#39;X-B3-Sampled&#39;],<br>        ),<br>        span_name=&#39;index_api_02&#39;,<br>        transport_handler=default_handler,<br>        port=5000,<br>        sample_rate=100,<br>        encoding=Encoding.V2_JSON<br>    ):<br>        call_api_03()<br>    return &#39;OK&#39;, 200</pre><p>It is as the same as api_01 except for <strong>zipkin_attrs. </strong>Here we are telling Zipkin tracer not to start a new trace but use the one which information is passed in the headers.</p><p>Then we call api_03 passing the new span information in the headers.</p><pre>@zipkin_client_span(service_name=&#39;api_02&#39;, span_name=&#39;call_api_03&#39;)<br>def call_api_03():<br>    headers = create_http_headers()<br>    requests.get(&#39;http://api_03:5000/&#39;, headers=headers)<br>    return &#39;OK&#39;</pre><h4>api_03</h4><p>In api_03 we will start Zipkin trace as we did in api_02. But after that, we will call a function that sleeps for 2 seconds</p><pre>@zipkin_client_span(service_name=&#39;api_03&#39;, span_name=&#39;sleep_api_03&#39;)<br>def sleep():<br>    time.sleep(2)<br>    return &#39;OK&#39;</pre><p>And that is it.</p><h3>Run the demo</h3><h4>Clone the code</h4><p>You can find the code in this <a href="https://github.com/MohamedMSaeed/tracing_flask_zipkin">GitHub repository</a>.</p><h4>Start The Containers:</h4><p>To start the containers run the following command:</p><pre>docker-compose up --build</pre><p>Then visit the api_01’s URL [ <a href="http://localhost:5001/">http://localhost:5001</a> ]</p><p>Remember the request flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJBmBYPBrCVwbbfDucCmtw.png" /></figure><p>To see this on Zipkin, visit Zipkin’s URL [ <a href="http://localhost:9411/zipkin">http://localhost:9411/zipkin</a> ] then click on the search button to get traces saved.</p><p>Select the trace that is saved last.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RBITDZQ2viOvXBrWtEiUCQ.png" /><figcaption>Select the trace.</figcaption></figure><p>And here we go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yx3b5Ply07pB811YLpNP6g.png" /></figure><p>As you can see from the previous image, api_01 made 2 requests ‘to api_02 and api_03’. Also, api_02 made a request to ‘api_03’</p><p>Each row here represents a <strong>span. </strong>You can create spans whenever it is needed.</p><p>The previous run used a basic installation, there are more advanced options and ways to create spans and to implement the whole solution. This demo wasn’t meant to be complicated as it is meant to be a POC.</p><p>For more information, please read the following documentation:</p><ul><li><a href="https://zipkin.io/">Zipkin official page</a></li><li><a href="https://github.com/Yelp/py_zipkin">py_zipkin github repository</a></li><li>If you have any question, you can ask the Zipkin dev team on gitter, <a href="https://gitter.im/openzipkin/zipkin#">here</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68886f02e46" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convert a VM to AMI in AWS]]></title>
            <link>https://mohamedmsaeed.medium.com/convert-a-vm-to-ami-in-aws-b101a1c52e0e?source=rss-1748c840df1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b101a1c52e0e</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[virtual-machine]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[ami]]></category>
            <category><![CDATA[ec2]]></category>
            <dc:creator><![CDATA[Mohamed Saeed]]></dc:creator>
            <pubDate>Mon, 23 Dec 2019 15:10:52 GMT</pubDate>
            <atom:updated>2019-12-23T15:10:52.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*KziWiVtYZMeegIm7hQLl_A.jpeg" /></figure><p>In some cases, you may need to commit your VM as an image and use it in a cloud.</p><p><strong>AWS</strong>; one of the biggest cloud providers in the market; gives you the power to use your own modified VM on the cloud. Simply, you will export your VM as an image then create an AMI from it and finally, run an EC2 using that new AMI you have just created! Simple hah!</p><p>So How can we do that!?</p><h3>Things you need to get this done</h3><ol><li>VM image.</li><li>S3 bucket where we will upload the image</li><li>IAM role called ‘<strong>vmimport</strong>’.</li><li>AWS CLI installed on your machine.</li></ol><blockquote>Be careful: AWS only supports certain Operating Systems. You can find a list <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/vmie_prereqs.html#prerequisites">here</a>.</blockquote><h4>1. Export your VM as an image:</h4><p>You can prepare a VM, install whatever you want to install on it and then when you are done, export it.</p><p>From <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#export-vm-image">AWS official documents</a> here are some of the formats that you can use:</p><ul><li>Open Virtualization Archive (OVA)</li><li>Virtual Machine Disk (VMDK)</li><li>Virtual Hard Disk (VHD/VHDX)</li><li>Open Virtualization Format (OVF)</li></ul><h4>2. Create an S3 bucket:</h4><p>Create an S3 bucket in the region where you will create your EC2s. We will use this bucket to store the VM image.</p><h4>3. Create an IAM role called ‘<strong>vmimport</strong>’:</h4><p>To allow <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/what-is-vmimport.html">VM import/export</a> service to perform some operation you should create a role called ‘vmimport’.</p><ul><li>The role that you will create needs some permission to work properly. Simply those permissions to enable VM import/export from accessing your S3 bucket and register your VM image as AMI.</li><li>To do that click on open IAM service &gt;&gt; <strong>policies</strong> &gt;&gt; create policy. <br>Choose JSON and paste the following code. <br>NOTE: change BUCKET with your bucket name.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*locBf7u-SBfbr0gO7lQl7g.png" /></figure><pre>{<br>   &quot;Version&quot;:&quot;2012-10-17&quot;,<br>   &quot;Statement&quot;:[<br>      {<br>         &quot;Effect&quot;:&quot;Allow&quot;,<br>         &quot;Action&quot;:[<br>            &quot;s3:GetBucketLocation&quot;,<br>            &quot;s3:GetObject&quot;,<br>            &quot;s3:ListBucket&quot; <br>         ],<br>         &quot;Resource&quot;:[<br>            &quot;arn:aws:s3:::BUCKET&quot;,<br>            &quot;arn:aws:s3:::BUCKET/*&quot;<br>         ]<br>      },<br>      {<br>         &quot;Effect&quot;:&quot;Allow&quot;,<br>         &quot;Action&quot;:[<br>            &quot;s3:GetBucketLocation&quot;,<br>            &quot;s3:GetObject&quot;,<br>            &quot;s3:ListBucket&quot;,<br>            &quot;s3:PutObject&quot;,<br>            &quot;s3:GetBucketAcl&quot;<br>         ],<br>         &quot;Resource&quot;:[<br>            &quot;arn:aws:s3:::BUCKET&quot;,<br>            &quot;arn:aws:s3:::BUCKET/*&quot;<br>         ]<br>      },<br>      {<br>         &quot;Effect&quot;:&quot;Allow&quot;,<br>         &quot;Action&quot;:[<br>            &quot;ec2:ModifySnapshotAttribute&quot;,<br>            &quot;ec2:CopySnapshot&quot;,<br>            &quot;ec2:RegisterImage&quot;,<br>            &quot;ec2:Describe*&quot;<br>         ],<br>         &quot;Resource&quot;:&quot;*&quot;<br>      }<br>   ]<br>}</pre><ul><li>Review the policy &gt;&gt; choose a name then &gt;&gt; save.</li><li>Open IAM service &gt;&gt; Roles &gt;&gt; Create Role</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n0G-_OeKElJbUOzFefu9qQ.png" /></figure><ul><li>For now, choose EC2 &gt;&gt; Next.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rjJ9tEW8Ks1L24plhrTKLg.png" /></figure><ul><li>Select the policy you have just created. Add a tag. Then save the role with the name “vmimport”.</li><li>Now, open vmimport role &gt;&gt; Trust relationships &gt;&gt; Edit trust relationships.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lwTGcP8GTkiX3OzrYY1j6g.png" /></figure><ul><li>Paste the following code then click update</li></ul><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;vmie.amazonaws.com&quot;<br>      },<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>      &quot;Condition&quot;: {<br>        &quot;StringEquals&quot;: {<br>          &quot;sts:Externalid&quot;: &quot;vmimport&quot;<br>        }<br>      }<br>    }<br>  ]<br>}</pre><h4>4. Install AWS CLI:</h4><p>To install and configure AWS CLI please follow the following documentation.</p><p><a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html">https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html</a></p><h3><strong>Do the magic and convert VM to AMI:</strong></h3><p>With AWC CLI installed and configured run the following command</p><pre>aws ec2 import-image --disk-containers Format=vmdk,UserBucket=&quot;{S3Bucket=BUCKET,S3Key=VM-disk.vmdk}&quot;</pre><p>The output will be something like this:</p><pre>{<br>   &quot;ImportImageTasks&quot;: [<br>      {<br>         &quot;ImportTaskId&quot;: &quot;import-ami-05464789141f433fb&quot;,<br>            &quot;Progress&quot;: &quot;19&quot;,<br>            &quot;SnapshotDetails&quot;: [<br>                {<br>                    &quot;DiskImageSize&quot;: 2782431232.0,<br>                    &quot;Format&quot;: &quot;VMDK&quot;,<br>                    &quot;Status&quot;: &quot;active&quot;,<br>                    &quot;UserBucket&quot;: {<br>                        &quot;S3Bucket&quot;: &quot;BUCKET&quot;,<br>                        &quot;S3Key&quot;: &quot;VM-disk.vmdk&quot;<br>                    }<br>                }<br>            ],<br>            &quot;Status&quot;: &quot;active&quot;,<br>            &quot;StatusMessage&quot;: &quot;converting&quot;<br>        }<br>    ]<br>}</pre><p>You can check the converting status using the <strong>ImportTaskId</strong> from the previous output:</p><pre>aws ec2 describe-import-image-tasks --import-task-ids import-ami-05464789141f433fb</pre><p>Once it is done, the status will be updated to ‘<strong>completed’.</strong></p><p>Find more about status messages <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html#check-import-task-status">here</a>.</p><h4>The only way?</h4><p>It is worth mentioning that this is not the only way to do the task. You can, for example, use <a href="https://www.packer.io/"><strong>Packer</strong></a><strong> </strong>or <a href="https://docs.aws.amazon.com/server-migration-service/latest/userguide/server-migration.html"><strong>AWS Server Migration Service</strong></a><strong>.</strong></p><p>For more information, see the <a href="https://docs.aws.amazon.com/vm-import/latest/userguide/vmimport-image-import.html">official documents</a> on AWS.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b101a1c52e0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Some Mac Shortcuts for Developers]]></title>
            <link>https://mohamedmsaeed.medium.com/some-mac-shortcuts-for-developers-cb26a419183e?source=rss-1748c840df1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb26a419183e</guid>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[shortcuts]]></category>
            <dc:creator><![CDATA[Mohamed Saeed]]></dc:creator>
            <pubDate>Sun, 15 Dec 2019 07:47:08 GMT</pubDate>
            <atom:updated>2019-12-23T08:06:55.975Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f_HDV-okVCQYwGk-5D2qGw.jpeg" /></figure><p>I have been using a mac for a while and I faced some troubles using the same shortcuts I used with Linux. So here are some helpful shortcuts.</p><h4>1. Terminal:</h4><ul><li>Copy and paste:</li></ul><pre>command + c || command + v</pre><ul><li>Break a process:</li></ul><pre>control + c</pre><ul><li>Go to the beginning of a line inside the terminal:</li></ul><pre>fn + shift + left arrow</pre><ul><li>Go to the end of a line inside the terminal:</li></ul><pre>fn + shift + right arrow</pre><ul><li>New window:</li></ul><pre>command + N</pre><ul><li>New tab:</li></ul><pre>command + T</pre><ul><li>Close a tab:</li></ul><pre>command + W</pre><ul><li>Go to tab number 1 || 2 .. etc:</li></ul><pre>command 1 || command 2 ... etc</pre><ul><li>Search in history commands:</li></ul><pre>control + R</pre><ul><li>Exit current shell:</li></ul><pre>control + D</pre><ul><li>Delete text where the cursor is:</li></ul><pre>fn + delete</pre><h4>2. Chrome:</h4><ul><li>Open the Developer Tools Chrome:</li></ul><pre>command + option + I || command + option + J || command + Shift + C.</pre><ul><li>New incognito window:</li></ul><pre>shift + command + N</pre><ul><li>open last closed tab:</li></ul><pre>command + shift + T</pre><ul><li>Open downloads:</li></ul><pre>option + command + L</pre><h4>3. General:</h4><ul><li>Navigate through more than one window for the same application:</li></ul><pre>command + `</pre><ul><li>Open new tab:</li></ul><pre>command + tab</pre><ul><li>Close current tab:</li></ul><pre>command + w</pre><ul><li>Home button (go to the beginning of a page):</li></ul><pre>fn + left arrow</pre><ul><li>End button (go to the end of a page)</li></ul><pre>fn + right arrow</pre><ul><li>Go to the beginning of a line:</li></ul><pre>command + left arrow</pre><ul><li>Go to the end of a line:</li></ul><pre>command + right arrow</pre><ul><li>Get next opened tab:</li></ul><pre>control + tab</pre><ul><li>Get the previously opened tab:</li></ul><pre>control + shift + tab</pre><ul><li>Open new tab:</li></ul><pre>command + t</pre><ul><li>Delete text in front of the cursor:</li></ul><pre>fn + delete</pre><p><strong><em>And of course, there are more other shortcuts that are helpful. If have any, please let us know it in the comments.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb26a419183e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Show all credentials’ value in Jenkins using script console.]]></title>
            <link>https://mohamedmsaeed.medium.com/show-all-credentials-value-in-jenkins-using-script-console-83784e95b857?source=rss-1748c840df1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/83784e95b857</guid>
            <category><![CDATA[decrypt]]></category>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[script-console]]></category>
            <category><![CDATA[groovy]]></category>
            <dc:creator><![CDATA[Mohamed Saeed]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 16:20:24 GMT</pubDate>
            <atom:updated>2019-02-25T16:20:24.894Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*q1ZrSz1R6mvXlT2xKxqKDA.png" /></figure><p>There is no doubt that Jenkins has become one of the most important CI/CD tools that is being used around the globe.</p><p>It has a lot of incredible features that make life easier. One of the most powerful feature, is Script Console. Jenkins offers a <a href="https://wiki.jenkins.io/display/JENKINS/Jenkins+Script+Console">console </a>that can execute Groovy scripts to do anything within the Jenkins master run-time or in the run-time on agents.</p><p>This console can be used in configuring Jenkins and debugging Jenkins run-time issues, misusing this console with leak of security may cause a lot of harm to you. You may lose the server or even get your infrastructure hacked.</p><h4>Access Script Console:</h4><p>Go to “Manage Jenkins” then click “Script Console”. Or you can easily go to this URL, “Jenkins_URL/script”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*teNBnozdhbzsmpqMsKeMtA.png" /></figure><p>Type any groovy code in the box then click Run. It will be executed on the server.</p><h3>Examples of what can be done:</h3><p>You can disable all jobs, delete the work-space for all disabled jobs to save space, decrypt credentials configured within Jenkins and decrypt any password stored in Jenkins for instance, “user password” if you have its hash code.</p><h4>To disable all jobs:</h4><p>To disable all jobs in your Jenkins at once, go to the magic console and execute the next peace of code.</p><pre>import hudson.model.*<br> <br> <br>disableChildren(Hudson.instance.items)<br> <br>def disableChildren(items) {<br>  for (item in items) {<br>    if (item.class.canonicalName == &#39;com.cloudbees.hudson.plugins.folder.Folder&#39;) {<br>        disableChildren(((com.cloudbees.hudson.plugins.folder.Folder) item).getItems())<br>    } else {<br>      item.disabled=true<br>      item.save()<br>      println(item.name)<br>    }<br>  }<br>}</pre><p>You can even delete their work-space after being disabled:</p><pre>//michaeldkfowler<br>import jenkins.model.*<br>Jenkins.instance.getAllItems(AbstractProject.class)<br>.findAll {it.disabled}<br>    .each {<br>      println(&quot;Wiping workspace for &quot;+it.fullName)<br>      it.doDoWipeOutWorkspace()<br>    }</pre><h4>Decrypt credentials defined in Jenkins and list values.</h4><p>Credentials are very critical and it’s important to save them in some place that no one can get them easily. Saving them in Jenkins is not the best way to do so.</p><p>The next set of code can easily print out all credentials stored in Jenkins server of type Private-Key then of type Username and Password with their VALUES!.</p><pre>def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(<br>    com.cloudbees.plugins.credentials.common.StandardUsernameCredentials.class,<br>    Jenkins.instance,<br>    null,<br>    null<br>);</pre><pre>for (c in creds) {<br>     println( ( c.properties.privateKeySource ? &quot;ID: &quot; + c.id + &quot;, UserName: &quot; + c.username + &quot;, Private Key: &quot; + c.getPrivateKey() : &quot;&quot;))<br>}</pre><pre>for (c in creds) {<br>     println( ( c.properties.password ? &quot;ID: &quot; + c.id + &quot;, UserName: &quot; + c.username + &quot;, Password: &quot; + c.password : &quot;&quot;))<br>}</pre><h4>Decrypt the password if you got its HASH.</h4><p>Accounts’ password and proxy user password are hashed and saved some where on the server. Assuming that you got that hash, then you can decrypt it using the next code.</p><pre>println(hudson.util.Secret.decrypt(&quot;{HASHxxxxx=}&quot;))</pre><pre># or /////</pre><pre>println(hudson.util.Secret.fromString(&quot;{HASHxxxxx=}&quot;).getPlainText())</pre><p>Of course, getting the hash is difficult but here is an example of how easy it might be.</p><p>On a server, a proxy is defined and username/password are saved to use this proxy. If you inspected the password field you will get the hashed value of the password from HTML!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BZhOCF92IFh-E1UvDzJJdA.png" /><figcaption>Jenkins saves passwords value and return the hash to the browser to use it later.</figcaption></figure><h4>Running shell commands:</h4><p>As a debugging tool, the console gives you the ability to run any shell command. i.e. to execute the commend “ls” on the server:</p><pre>println new ProcessBuilder(&#39;sh&#39;,&#39;-c&#39;,&#39;ls&#39;).redirectErrorStream(true).start().text</pre><p>You can run any shell commands or scripts on the server by creating a job and execute you commands. But you can use the console as a hidden place to do what ever you want to do without letting others know about it.</p><blockquote>WARNING: In general, if Jenkins user has sudo permission, anyone authorized to run shell on the server will be sudo as will.</blockquote><h3>Secure Your Jenkins</h3><p>You have to spend enough time to decide how to secure your Jenkins. Regarding accessing the console you can do the following:</p><h4>First: Limit users who can reach the console.</h4><p>To do this you may use “ Matrix-based security”. Go to “Configure Global Security” then in “ Authorization” part choose “ Matrix-based security”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVsVxxXavjmZaOWSlRdi6g.png" /><figcaption>Non admins should have no Administer privileges.</figcaption></figure><p>In overall section, give the “Administer” option to admins only. Those with Administer can access “Manage Jenkins” and “Script Console”.</p><h4>Second: Don’t give jenkins user any sudo permissions.</h4><p>Don’t add jenkins user in sodoers. It will give root permissions to all of your Jenkins admins. They can easily run a cmd as a root and change what ever they want.</p><h4>Be Safe…</h4><blockquote>It can configure any Jenkins setting. It can disable security, reconfigure security, even open a backdoor on the host operating system completely outside of the Jenkins process. Due to the mission critical importance many organizations place on Jenkins in their infrastructure this point is especially important because it would allow an attacker to move laterally within infrastructure with little effort.</blockquote><blockquote>From the console page.</blockquote><h4>More?</h4><p>You can find more examples of what can be done using the console on <a href="https://wiki.jenkins.io/display/JENKINS/Jenkins+Script+Console">the plugin website</a>. Also, you can have a look on this <a href="https://github.com/jenkinsci/jenkins-scripts">github repo</a> and <a href="https://github.com/samrocketman/jenkins-script-console-scripts">this one</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/971/1*DHJKgTtYNuWc7-bjXiOBtA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83784e95b857" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Jenkins with Grafana and Prometheus]]></title>
            <link>https://mohamedmsaeed.medium.com/monitoring-jenkins-with-grafana-and-prometheus-a7e037cbb376?source=rss-1748c840df1f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7e037cbb376</guid>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Mohamed Saeed]]></dc:creator>
            <pubDate>Mon, 24 Dec 2018 22:29:06 GMT</pubDate>
            <atom:updated>2018-12-27T08:59:28.776Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I will show you how to configure <a href="https://prometheus.io/">Prometheus</a> and <a href="https://grafana.com/">Grafana </a>to monitor your Jenkins server and create a such wonderful dashboard. A dashboard that contains lots of information and statistics about your Jenkins server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eA8QtRrg-xy7_IhCYJBSvw.png" /><figcaption>Dashboard can contain statistics about failed and success jobs numbers, JVM memory, CPU utilization and more!!</figcaption></figure><blockquote>We will use <a href="https://docs.docker.com/">docker</a> containers to get our servers up and running. We will create Jenkins, Prometheus and Gravana containers. Then export data from Jenkins passing it to Prometheus then to Gravana.</blockquote><h4>Getting started:</h4><p>In this step we will create the three docker containers. If you have your servers up and running you can skip this step.</p><pre>docker run -d --name jenkins -p 8080:8080 -p 50000:50000 jenkins<br>docker run -d --name prometheus -p 9090:9090 prom/prometheus<br>docker run -d --name grafana -p 3000:3000 grafana/grafana</pre><h4>Install necessary plugin in Jenkins:</h4><ul><li>If this your time using Jenkins, you can have a look at their documentations <a href="https://jenkins.io/doc/">here</a>.</li><li>Go to “manage Jenkins” then “manage plugins” and install “ <a href="https://wiki.jenkins-ci.org/display/JENKINS/Prometheus+Plugin">Prometheus metrics plugin</a>”.</li><li>This plugin will expose an endpoint (default /prometheus) with metrics where a Prometheus Server can scrape data.</li><li>You can see the output of this plugin visiting this URL<br>“http://Jenkins_HOST:PORT/prometheus”</li></ul><h4>Configure Prometheus to scrape metrics from Jenkins:</h4><ul><li>To configure Prometheus server you will need to edit in the configuration file “prometheus.yml” inside the container.</li><li>In the scrape_configs part you need to add a job for Jenkins server:</li></ul><pre>- job_name: ‘jenkins’<br>  metrics_path: /prometheus<br>  static_configs:<br>    - targets: [‘Jenkins_Host:PORT’]<br></pre><ul><li>Replace “Jenkins_Host” and “PORT” with the URL of your Jenkins server then restart the server.</li></ul><blockquote>If you are using docker container, you can only restart the container.</blockquote><ul><li>This is how Prometheus looks like:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*392uY6J7Z4Pqw71OqN_TCg.png" /><figcaption>You can visit prometheus server on “localhost:8086”</figcaption></figure><ul><li>Beside “Execute” button there is a drop down menu that contains all metrics. You will find some metrics start with <strong>“Prometheus_*”</strong> which Prometheus uses to monitor itself. You will also find some starting with <strong>“Jenkins_*”</strong> and<strong> “vm_*”</strong> and those are representing data related to Jenkins server. You can spend sometime exploring these metrics.</li><li>We will use those metrics later in Grafana server.</li></ul><h4>Connect Prometheus with Grafana:</h4><ul><li>Go to Grafana server &lt;localhost:3000&gt; and login. The default user and password are “admin/admin”</li><li>From “ Home Dashboard” click on “add data source” then click on “Prometheus”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIVJUuLL10or8r3iOKEAuQ.png" /></figure><ul><li>In the URL field enter Prometheus URL in our case &lt;prometheus_container_id:9090&gt;. you can leave all other fields to default. Then click save and test.</li><li>Now you are ready to create your dashboard.</li></ul><h4>Create a dashboard in Grafana to display data:</h4><ul><li>From “ Home Dashboard” click on “new dashboard”. Choose the panel type that you want. We will use “<a href="http://docs.grafana.org/features/panels/singlestat/">Singlestat</a>”.</li><li>Click on “Panel Title” then “edit”.</li><li>In the query section you can provide the metrics from Prometheus server. Let’s take “jenkins_plugins_active” as an example.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIRYUReI2VMbVV36PGYqrQ.png" /></figure><ul><li>Save and go back to your dashboard.</li><li>Now you have a dashboard with only one panel that represent one piece of information about Jenkins. You can add as many panels as you want to create your own dashboard.</li><li>You can visit Grafana’s website for more info about how to create more panels.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7e037cbb376" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>