<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Clement Demonchy on Medium]]></title>
        <description><![CDATA[Stories by Clement Demonchy on Medium]]></description>
        <link>https://medium.com/@cols-knil?source=rss-a0844e3fa871------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*U4RZ7wc_rNAsmH6R7UutfA.jpeg</url>
            <title>Stories by Clement Demonchy on Medium</title>
            <link>https://medium.com/@cols-knil?source=rss-a0844e3fa871------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 00:41:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cols-knil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AWS organization on personal account part 3 : budgets and cost reports]]></title>
            <link>https://awstip.com/aws-organization-on-personal-account-part-3-budgets-and-cost-reports-388ce0b8acad?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/388ce0b8acad</guid>
            <category><![CDATA[billing]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[sso]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Tue, 14 Feb 2023 18:04:25 GMT</pubDate>
            <atom:updated>2023-02-21T16:05:16.406Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS organization on personal account part 3 : <strong>budgets </strong>and cost reports</h3><p>Welcome to the third part in this series, this time we will setup some alerting for your AWS bills, understand how tagging done in part 2 will help get an better picture of expenses and get a proper cost dashboard.</p><h3>Budget and Budget Report</h3><p>Even through understanding your AWS bill itself is a complicated issue, setting a budget in order to get alert based on threshold, forecast and other metrics is relatively straight forward.</p><p>Under Billing Console you have a budget menu were you can create an budget in this screenshot we go with the simplest option, got an alert if any free tier for the whole account is reached and send a email.</p><figure><img alt="In the create budget page we got an section &quot;Budget Setup&quot; with Use template and Customize option present, template is checked and next section is Templates were option Zero spend budget is checked other options are Monthly cost budget and Daily Saving Plan coverage budget. After that final section is the Zero spend template with a name for the budget and a field to insert email that will be notified by the alert" src="https://cdn-images-1.medium.com/max/703/1*mJat_U65D-wX73lpC0_vEQ.png" /></figure><p>Once budget are created you can ask for Budget report to be send directly by email</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uc5f5ScbG_dlP0B0xllD-w.png" /></figure><p>You will feel safer after setting some alert keep an eye on cost activity but this will not answer our question of why cost are going up, to gain some insight head to the cost explorer</p><h3>Cost Explorer</h3><p>This a great tools provided by AWS were you can filter and agreggate your data on a lot of different parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Uor7w4dD3ctd1nxVFO8jA.png" /></figure><p>On this you can do is the ability to go back to some previously exploration by saving you current parameters using the &quot;Save to report library&quot; button and name it to use it again later.</p><p>There is also an option to export result as csv to feed into other system</p><h3>Cost &amp; Usage report</h3><p>If you want to drill deeper seeing rate, cost not limited to a service, or automate what you can do in Cost Explorer you have to create an CUR (Cost and Usage Reports).</p><p>This will upload detailed billing information in a s3 bucket to be displayed using a dashboard tool, in our example we are going to use Athena for this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*iiefgm8_-NXG8Z4yu7Do8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lb1ZVbhvsET_H_Hp42Q5Bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I_s2hp2ElbTLsX2nvxtGug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7RiqpqKt2voU-tW-ArInQ.png" /></figure><p>Need to wait for the first result to be export to s3, we can then find an SQL query to run on Athena in order to create the table</p><p>If you look at the end of the CUR you will find the Cost allocation Tag we set in part 2, simple tag will not be shown here contrary to Cost explorer</p><p>Here some of the previously seen examples as terraform</p><pre>resource &quot;aws_budgets_budget&quot; &quot;total_organization&quot; {<br>  name         = &quot;total-organization-budget&quot;<br>  budget_type  = &quot;COST&quot;<br>  limit_amount = &quot;5&quot; # I usually manage to stay under free tier ;)<br>  limit_unit   = &quot;USD&quot;<br>  time_unit    = &quot;DAILY&quot;</pre><pre>notification {<br>    comparison_operator        = &quot;GREATER_THAN&quot;<br>    threshold                  = 100<br>    threshold_type             = &quot;PERCENTAGE&quot;<br>    notification_type          = &quot;ACTUAL&quot;<br>    subscriber_email_addresses = [&quot;youremail@dot.com&quot;]<br>  }<br>}</pre><pre>resource &quot;aws_s3_bucket&quot; &quot;billing&quot; {<br>  bucket = &quot;billing-eu-west-3&quot;<br>}</pre><pre>data &quot;aws_iam_policy_document&quot; &quot;billing_write_access&quot; {<br>  # READ/WRITE access for all s3 bucket <br>  statement {<br>    actions = [<br>      &quot;s3:GetBucketAcl&quot;,<br>      &quot;s3:GetBucketPolicy&quot;<br>    ]<br>    resources = [<br>      aws_s3_bucket.billing.arn,<br>    ]<br>    principals {<br>      type        = &quot;AWS&quot;<br>      identifiers = [&lt;root account id&gt;]<br>    }<br>  }<br>  statement {<br>    actions = [<br>      &quot;s3:PutObject&quot;,<br>    ]<br>    resources = [<br>      &quot;${aws_s3_bucket.billing.arn}/*&quot;,<br>    ]<br>    principals {<br>      type        = &quot;AWS&quot;<br>      identifiers = [&lt;root account id&gt;]<br>    }<br>  }<br>}</pre><pre>resource &quot;aws_s3_bucket_policy&quot; &quot;allow_access_from_billing_account&quot; {<br>  bucket = aws_s3_bucket.billing.id<br>  policy = data.aws_iam_policy_document.billing_write_access.json<br>}</pre><pre>resource &quot;aws_cur_report_definition&quot; &quot;cost_report&quot; {<br>  report_name                = &quot;cost_report_&quot;<br>  time_unit                  = &quot;HOURLY&quot;<br>  format                     = &quot;Parquet&quot;<br>  compression                = &quot;Parquet&quot;<br>  additional_schema_elements = [&quot;RESOURCES&quot;]<br>  s3_bucket                  = aws_s3_bucket.billing.id<br>  s3_prefix                  = &quot;cost-report&quot;<br>  s3_region                  = &quot;eu-west-3&quot;<br>  additional_artifacts       = [&quot;ATHENA&quot;]<br>  refresh_closed_reports     = &quot;true&quot;<br>  report_versioning          = &quot;OVERWRITE_REPORT&quot;<br></pre><p>You can use your budget alert to also check consumption of resource like RAM or Reserved Instance usage see <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/budgets_budget">example</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=388ce0b8acad" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/aws-organization-on-personal-account-part-3-budgets-and-cost-reports-388ce0b8acad">AWS organization on personal account part 3 : budgets and cost reports</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clean AWS organization on your personal account part 2: enforce tag policy]]></title>
            <link>https://cols-knil.medium.com/clean-aws-organization-on-your-personal-account-part-2-enforce-tag-policy-4022d451d24?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/4022d451d24</guid>
            <category><![CDATA[tag]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[billing]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Mon, 13 Feb 2023 08:41:18 GMT</pubDate>
            <atom:updated>2023-02-13T08:41:18.900Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the second part in this series, this time we will setup on <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies.html">tag policy</a> for your AWS resources, that will allow us to define efficient billing in next post, in order to proceed you need to have setup properly your AWS organization.</p><h3>Tag Policy</h3><p>An tag policy is a list of tag key (think project, company, private, …) with some more rules that can enforce: capitalization, restrict value to hardcoded content or regex, specific resources to enforce non-compliant one (this could have unintended effect so I would recommend to read <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_tag-policies-enforcement.html">this</a> before using it).</p><p>So this step is another clickops operation, under AWS Organizations service you can activate different policies the one we want right now is &quot;Tag policies&quot;.</p><figure><img alt="Under AWS Organizations -&gt; Policies -&gt; Tag policies, we have a big button Enable Tag Policies" src="https://cdn-images-1.medium.com/max/1007/1*8q1gfzqf92nNdbY1IaIm3A.png" /></figure><p>Once this is done you need to create a new policies.</p><figure><img alt="On Tag policies interface we have a menu to add new policy with a button &quot;create policy&quot;" src="https://cdn-images-1.medium.com/max/965/1*Ch4CsBkQ5qR6PoJitxfoEg.png" /></figure><p>This new policy will be defined by it name, description and a list of &quot;Tag Key&quot;.</p><figure><img alt="Under visual editor of the tag policy create there is a menu were can add new Tag key were we specify the name and can check other option" src="https://cdn-images-1.medium.com/max/952/1*MFI1rwQdD9omaMEtkLkGLQ.png" /></figure><p>Also you can tag your tag policy if you want to feel meta.</p><p>After creating a tag policy you have to attach it on you account or root user.</p><figure><img alt="In tag policy service un Target we have a list of attached policy and a Attach Detach button" src="https://cdn-images-1.medium.com/max/1001/1*MbY2u_Nctjwug3OXTprdTQ.png" /></figure><p>This allow you to check which resources you created that are not compliant your tag using Tag Policies AWS service</p><figure><img alt="Under dashboard for tag policy we can see existing one and have the possibility to search non compliant resources in our AWS organization and filter by region" src="https://cdn-images-1.medium.com/max/1024/1*RCz4vPsRao1JcS7-AnDdgw.png" /></figure><p>You can also start activating them in Billing as Cost allocation tags now that they are created and applied, it will matter in the next part of this series.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rh2lBMuyMH8hHlHFK-xUtA.png" /></figure><p>Here the same configuration done using terraform (you still need to click on enable Tag policies in AWS Organization first)</p><pre>resource &quot;aws_organizations_policy&quot; &quot;global_default_billing&quot; {<br>  content  = jsonencode(<br>    {<br>      tags = {<br>        application = {<br>          tag_key   = {<br>            &quot;@@assign&quot; = &quot;application&quot;<br>          }<br>          tag_value = {<br>            &quot;@@assign&quot; = [<br>              &quot;frontend&quot;,<br>              &quot;backend&quot;,<br>              &quot;storage&quot;,<br>              &quot;database&quot;,<br>              &quot;crawler&quot;,<br>              &quot;api&quot;,<br>              &quot;billing&quot;<br>            ]<br>          }<br>        }<br>        project     = {}<br>      }<br>    }<br>  )<br>  name     = &quot;global-default-billing&quot;<br>  tags     = {&quot;application&quot;: &quot;billing&quot;}<br>  type     = &quot;TAG_POLICY&quot;<br>}</pre><pre>resource &quot;aws_organizations_policy_attachment&quot; &quot;organization&quot; {<br>  policy_id = aws_organizations_policy.global_default_billing.id<br>  target_id = &quot;r-f8ph&quot;<br>}</pre><pre>resource &quot;aws_ce_cost_allocation_tag&quot; &quot;application&quot; {<br>  tag_key = &quot;application&quot;<br>  status  = &quot;Active&quot;<br>}<br>resource &quot;aws_ce_cost_allocation_tag&quot; &quot;project&quot; {<br>  tag_key = &quot;project&quot;<br>  status  = &quot;Active&quot;<br>}</pre><p>We can go on step further by adding SCPs that can enforce at creation constraint like requiring tag on not allowing some role to change them but it can conflict with the way you allocate resource like cloudformation or SAM for example</p><p><a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_tagging.html#example-require-tag-on-create">Example SCPs for tagging resources</a></p><p>Hope you like it and stay tuned for the next parts in this series Clean AWS organization on your personal account part 3: budgets and billing reports</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4022d451d24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Clean AWS organization on your personal account part 1: SSO access]]></title>
            <link>https://awstip.com/clean-aws-organization-on-your-personal-account-part-1-sso-access-1745971c2465?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/1745971c2465</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[sso]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Fri, 10 Feb 2023 17:13:38 GMT</pubDate>
            <atom:updated>2023-02-21T16:05:46.378Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello here, this post will relate my journey to upgrade my own AWS account to &quot;production like&quot; standard in term of security, organization and billing, each article will try to have one clear goal.</p><p>This article will start first by explaining fundamental concepts <br>*ROOT user<br>*organization<br>*AWS IAM Identity Center (previously AWS SSO)<br>*IAM roles</p><p>Then we show how to access your AWS account using tools like <a href="https://github.com/benkehoe/aws-sso-util">aws-sso-utils </a>instead of default user.</p><h3>Concepts</h3><h4>ROOT user</h4><p>This will be your default user created by AWS when opening your account that will be the admin and have all the right <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html">official documentation</a>.</p><h4>Organization</h4><p>This is the way AWS allow you to associates multiple account, setup global policy and a single point of billing <a href="https://aws.amazon.com/organizations/">official documentation</a>.</p><h4><strong>AWS IAM Identity Center</strong></h4><p>Previously know as AWS SSO this is the way to create user and role that have SSO access, you can also have groups setup to avoid repeating setup for new user <a href="https://aws.amazon.com/iam/identity-center/">official documentation</a>.</p><h4>AWS roles</h4><p>IAM roles regroup policy (what you can do with which services) that will be applied to a user or a group, those policies can be created by you or you can use the one already setup by AWS.</p><h4>IAM roles</h4><p>IAM roles are a new way to give temporary access, the result is almost the same from an user standpoint but the from the account administrator you have a IAM role were you allow people to log into <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html">official documentation</a>.</p><h3>Setting SSO access</h3><p>One of the most fundamental concept in security <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">The Principle of Least Privilege</a>, states that <strong>a subject should be given only those privileges needed for it to complete its task</strong>, so as we saw before root user is able to do anything on your AWS account, so we need to setup other ways to access your AWS account in a more restricted way.</p><p>At the beginning people used to created user, give them roles and create credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) to store in their computer but, the risk was to have this credential compromised and the switch between credential could mean you deployed on prod by accident.</p><p>The new recommended way for user is to create user on AWS IAM Identity Center that can have <strong>temporary credential</strong> generated using SSO.</p><p>Assuming you just created the root account, I would recommend setting <a href="https://docs.aws.amazon.com/accounts/latest/reference/root-user-mfa.html">Multi-factor authentication (MFA)</a></p><p>We now need to activate organization lucky for us this don’ t cost <a href="https://aws.amazon.com/organizations/">anything</a>, and will unlock lot of services we will discuss in more detail later, one this advantage and the ability to aggregate more account under this single organization umbrella, that something you will want for production ready env (one account by env, department, team, …).</p><p>We can link federated user from other tools like Okta for example but we will keep it out of this article as this is too much for a personal account with only pet project.</p><p>You need to go to service AWS SSO and select you account user under root.</p><figure><img alt="A display of my AWS account under AWS sso, at the root we have the Root user his id, below my default account with my name, his account id and my mail striked out" src="https://cdn-images-1.medium.com/max/800/1*AAOuiJQzeTs2KrrIcW351g.png" /></figure><p>Here we will create an new admin user.</p><figure><img alt="AWS SSO interface were we see a newly created admin user" src="https://cdn-images-1.medium.com/max/1024/1*BUcWhjO1FIKga5_pE54oyQ.png" /></figure><p>Then navigate under permission set to give it one that is already provided by AWS <strong>AdministratorAccess.</strong></p><figure><img alt="AWS SSO interface, permission set menu were we see AdministratorAccess provisonned" src="https://cdn-images-1.medium.com/max/1024/1*JZgrJy6YkUqAEwiUjtbjPQ.png" /></figure><p>We now have to set an local profile that can use you new</p><pre>[profile admin]<br> sso_start_url = &lt;FETCH FROM YOUR AWS IAM IDENTITY CENTER CONSOLE&gt;<br> sso_region = us-east-1<br> sso_account_id = &lt;YOUR ACCOUNT ID&gt;<br> sso_role_name = AdministratorAccess<br> region = us-east-1<br> output = json</pre><p>I lost some time making this work because the sso_role_name didn&#39;t work with the role name but with the permission set name in this case.</p><blockquote>aws sso login — profile admin; export AWS_PROFILE=admin</blockquote><p>You can also use this great tools to this setup, specify your role with each command and even start AWS console directly <a href="https://github.com/benkehoe/aws-sso-util">aws-sso-utils</a></p><p>Next step would be to create new users with more appropriate permission set, like ML related account, read only access for customer and monitoring, etc …</p><p>If you want to go deep into the security parts I recommend you look at how to enforce it at the organization level with <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html">AWS SCPs</a> and dig deeper into monitoring of the accounts <a href="https://aws.amazon.com/detective/">AWS detective</a>, <a href="https://aws.amazon.com/guardduty/">AWS GuardDuty</a>.</p><p>Hope you like it and stay tuned for the next parts in this series Clean AWS organization on your personal account part 2: enforce tag policy</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1745971c2465" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/clean-aws-organization-on-your-personal-account-part-1-sso-access-1745971c2465">Clean AWS organization on your personal account part 1: SSO access</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Airflow 2.0 on docker-compose]]></title>
            <link>https://cols-knil.medium.com/airflow-2-0-on-docker-compose-8a3d98c1a6f8?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a3d98c1a6f8</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Mon, 25 Jan 2021 07:56:21 GMT</pubDate>
            <atom:updated>2021-08-18T15:50:36.989Z</atom:updated>
            <content:encoded><![CDATA[<p>So you heard of the brand new Airflow 2.0 and want to run it on your own machine to give it a go.</p><p>Sadly, the<a href="http://airflow.apache.org/docs/apache-airflow/stable/start.html"> getting started</a> isn’t nearly ready for an integration environment as it neglects most of the interesting components and features like multiple workers to distribute tasks.</p><p>Don’t worry I have you covered.</p><p><strong>In this article I will show you a dummy project with:</strong></p><ul><li>Everything running under docker compose using official Airflow docker image</li><li>Separate container for Airflow scheduler and webserver</li><li>An Postgresql backend for Airflow operation that allow to run task in parallel</li><li>Multiple workers run with Celery</li><li>Flower as Celery’s GUI</li><li>Home-made dags instead of Airflow examples</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*FfaKwSfDzNNSc4NJ" /></figure><p>Here you can see the full infra we want to create from our project repository <a href="https://github.com/knil-sama/itm">https://github.com/knil-sama/itm</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9de703fe888b91d4c079282cb3d429ea/href">https://medium.com/media/9de703fe888b91d4c079282cb3d429ea/href</a></iframe><p>You can then acces the <a href="http://0.0.0.0:8080/">Airflow GUI</a></p><p>First thing you want to do once everything is up and running, is to login into airflow webserver with user “admin” and password “test”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TwyGy5AnCoav-600" /></figure><p>Then we can activate our dag “main_dag” that will then be picked up by our scheduler every 5 min (or you can click on the play icon to trigger it by hand).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x3g25o8sRkoQH92p" /></figure><p>Here for more details regarding DAG run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-NYTkMJ68rT5sg9E" /></figure><p>Another interface you can interact with is the <a href="http://0.0.0.0:5555/">Flower</a> one that provides a monitoring of your celery workers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hKirosPaJfpTknXY" /></figure><p>If we take a look at the <a href="https://github.com/knil-sama/itm/blob/main/docker-compose.yml">docker-compose</a> we can see the dependencies between each machines and starting order</p><p>webserver -&gt; scheduler -&gt; woker then flower</p><p>Each machine started with a dedicated airflow command, except the webserver where we do two operations before starting it:</p><ol><li>Initializing postgresql backend db (to simplify the demo here we don’t keep persistent storage and reinitialize each run in order to avoid conflict when doing init on an already existing backend)</li><li>Create an user admin because airflow don’t provide one by default only <a href="https://airflow.apache.org/docs/apache-airflow/stable/security/access-control.html">roles</a>, you can also allow user to self register</li></ol><p>Another thing to consider is that we share DAGs among the following machines so we rely on <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts.html?highlight=xcom">xcom</a> to rely simple variable and store image in mongodb, because we don’t have a shared filesystem</p><p>To avoid duplicated configuration we create single env file <a href="https://github.com/knil-sama/itm/blob/main/env_airflow.env">env_airflow.env</a></p><p>Here some details of the configuration that are needed</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/729cb9880f0b34c68d52bf9b48ffd2e9/href">https://medium.com/media/729cb9880f0b34c68d52bf9b48ffd2e9/href</a></iframe><p>After that you are all set to experiment yourself with Airflow</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a3d98c1a6f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cronjob 101 part 2]]></title>
            <link>https://medium.com/jobteaser-dev-team/cronjob-101-part-2-78503d3967b3?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/78503d3967b3</guid>
            <category><![CDATA[production]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[cronjob]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 07:35:14 GMT</pubDate>
            <atom:updated>2019-10-10T07:54:50.828Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubernetes Cronjob 101 part 2</h3><p><em>Hello there for the second part of </em><a href="https://medium.com/jobteaser-dev-team/kubernetes-cronjob-101-56f0a8ea7ca2"><em>cronjob 101</em></a><em>.</em></p><p>In the first part we focused on how to set up your first cronjob with the proper configuration and manipulate them with the Kubernetes cli. In this second part, we will give you a more detailed explanation of our workflow to launch and monitor our cronjob in production at <a href="https://medium.com/u/108deca96f18">JobTeaser Dev Team</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_HISJmlPTO2yXBPDx93omg.jpeg" /><figcaption>Take a chair and relax</figcaption></figure><p>One issue we faced with our initial workflow was that not all of our jobs could be run concurrently, so when we were working on separate branches, we would create a lock on stateful resource such as a database.</p><p>The solution we came up with, relies on using the <strong>--from</strong> option when creating a job that expects a deployed cronjob</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74620177f2226ad16cce2268499f4eb6/href">https://medium.com/media/74620177f2226ad16cce2268499f4eb6/href</a></iframe><p>So when we start working on a new cronjob, our workflow is as follows:</p><ol><li>Create a <strong>suspended</strong> cronjob</li><li>Check cronjob deployment</li><li>Create a job from this cronjob with <strong>--from</strong> and check out the result</li><li>Goto 1 until it work as expected</li><li>Deploy it in production</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*Lx_DKZQMFCdLBNos-10Slg.png" /><figcaption>cycle deployment cronjob</figcaption></figure><p>This help avoids the issue of manually changing schedules after each deployment to start the jobs, or letting bugged job stack up, possibly locking down resources.</p><p>The limitation when creating job this way is that you can’t alter it.</p><p>Therefore, what will you do if you have to handle occasional operations such as catching up for ETL job or rerun failed job with increased limits (timeout, CPU, …) ?</p><p>For example, let’s say you have a cronjob that runs every day and uses “yesterday” as starting date by default. What we do is rely on command line parameters for the script and let it set a default value for which there is no input. This way we can manually edit the cronjob:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66550f85560f268593f27766a7f1ec1a/href">https://medium.com/media/66550f85560f268593f27766a7f1ec1a/href</a></iframe><p>Now that you have this code in production if you want to rerun it using previous dates, what you can do is create a duplicate of the cronjob.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ec3b9d00e6093a231155fb0e78e2b39/href">https://medium.com/media/8ec3b9d00e6093a231155fb0e78e2b39/href</a></iframe><p>Edit its content to reflect your new limitations (CPU, RAM, deadline) and then create a job from your duplicated cronjob (I recommend setting “suspend” to initially true to avoid concurrent runs ;) ) without impacting the one handling the normal run.</p><p>As we said, you need to rely on a script to handle the setup of a default value because you can’t call Linux command in you <strong>args</strong>, like <strong>date</strong> for example.</p><p>But did you know that you can use variable sets in <strong>env</strong> and <strong>envFrom</strong> in your script? You just need to call them using <strong>$()</strong> instead of<strong> ${}</strong>. It can be useful if you don’t want your script to rely on env variables</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c40501a7aed2e6e8d3ec0b7601548b47/href">https://medium.com/media/c40501a7aed2e6e8d3ec0b7601548b47/href</a></iframe><p>It’s really convenient to be able to raise an alert when a job fails, sadly as we saw in the previous article this is not always easy due to the job itself, so what we do now is to generate a log at the start and the end on the pod runtime using the lifecycle option in our cronjob definition</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c24d77ba30930fe6838331263b72af7e/href">https://medium.com/media/c24d77ba30930fe6838331263b72af7e/href</a></iframe><p>But keep in mind that if you use a docker container with an entrypoint you can run into some issue where the lifecycle will be executed <a href="https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks">AFTER the entrypoint</a>.</p><p>Another great feature of Kubernetes is the <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/">init containers</a> option, it consists of a list of ordered containers that will be run sequentially.</p><p>Each container waiting for the previous one to finish before launching, this is mainly used with the service API to sync them when deploying.</p><p>But this also works with a cronjob, so we can use them to do some sync or clean up before starting a job.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b280f7dd32635d8a32c16c9e415fc08/href">https://medium.com/media/7b280f7dd32635d8a32c16c9e415fc08/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0kTnWOywbF5350gwdpDfzA.jpeg" /><figcaption>Time to wrap things up and go</figcaption></figure><p>So, in conclusion, even if the cronjob API itself didn’t evolve much since part 1, there are many ways to improve your current use by looking at other tools in Kubernetes whether it is kubectl, the API options or even by integrating it to your own CI.</p><p>EDIT:</p><p>Just found this great article on how to raise an alert if you use prometheus/grafana to analyze your Kubernetes cluster, <a href="https://thinklumo.com/alert-failed-long-running-cronjobs-kubernetes-prometheus/">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78503d3967b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/cronjob-101-part-2-78503d3967b3">Cronjob 101 part 2</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Packaging : How do we manage it at JobTeaser]]></title>
            <link>https://medium.com/jobteaser-dev-team/python-packaging-how-do-we-manage-it-at-jobteaser-2bc8d1e9d206?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bc8d1e9d206</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pypi]]></category>
            <category><![CDATA[package-manager]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Thu, 07 Feb 2019 15:22:02 GMT</pubDate>
            <atom:updated>2019-02-07T15:25:39.484Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python Packaging : How we manage it at JobTeaser</h3><h3>Problem</h3><p>Here at JobTeaser, we found ourselves duplicating a great amount of code between text cleaning operations for machine learning projects, AWS wrapper, and Kafka helper.</p><p>The solution we found to this trouble was to extract our reusable code into proper packages (correctly tested and documented), before pushing it to our own package server.</p><h3>Previous solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*EUOCRxTBTc3Z4Hb0AAPfPQ.png" /><figcaption>The previous solution with own pypi</figcaption></figure><p>The current solution relies on an EC2 machine that contains a pypi server. The CI (here CircleCi) will build→test →push a new version of the package to the pypi server.</p><p>Once published, other projects can use the package in their CI process by requesting it from pypi server.</p><h3>Previous solution limitations</h3><ul><li>Can’t have default read-only for everyone and an authenticated user at the same time.</li><li>Needs fine access control at user level because CircleCI doesn’t have fixed IPs.</li><li>Sometimes the pypi server hangs for no reason and we have to restart it.</li><li>All the data is stored on the server without any backup!</li></ul><h3>Looking for alternative(s)</h3><p>Because our infra already runs on AWS, this search was limited to solutions that used S3 as a backend, after some time we found two solutions:</p><ul><li><a href="http://novemberfiveco/s3pypi">S3pypi</a></li><li><a href="https://pypicloud.readthedocs.io/en/latest/">Pypicloud</a></li></ul><h4>S3pypi</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*ke3DDc9Woaj2JZx5W-kuAw.png" /><figcaption>How we could have implemented S3pypi</figcaption></figure><p>As you can see the strongest point of the S3pypi solution is that it is straightforward.</p><p>It effectively uses a Cloudfront to expose s3 content in the same way that a <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html">static website</a>.</p><p>But at Jobteaser, we manage all our AWS resource using Terraform. So we would have to take their templating and adapt it, not to mention that the security relies only on the Cloudfront setup.</p><p><strong>Pros:</strong></p><ul><li>Full AWS</li><li>Low costs</li></ul><p><strong>Cons:</strong></p><ul><li>Setup doesn’t integrate easily with our tooling (Terraform)</li><li>Doesn’t rely on user level access control</li></ul><h4>Pypicloud</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*v-jH4fEw3xsvTXZZ4cGOfg.png" /><figcaption>How we could have implemented Pypicloud</figcaption></figure><p>On the other hand we have Pypicloud that goes in the opposite direction with a more complex and modular solution.</p><p>Similarly to the current pypi solution, it requires a dedicated server but also has the added need for external storage and a caching service. But on the bright side, you can actually use any option you want for these components.</p><p><strong>Pros:</strong></p><ul><li>Modular solution</li><li>Has users and more configuration than pypi</li><li>Nice GUI for admin</li></ul><p><strong>Cons:</strong></p><ul><li>Needs a more complex setup</li><li>Costs are slightly higher</li><li>Users need to be created from the admin interface</li></ul><p>In the end the winner was Pypicloud, because, for the deployment, Cloudfront’s hidden complexity and the need to rely only on AWS IAM were unacceptable.</p><h3>Implementing the new solution</h3><p>First, we need to create an s3 bucket, with access rights granted to a unique AWS user, in our case, the Pypicloud server.</p><p>One major change from the recommended AWS configuration is the use of a PostgreSQL RDS as a cache instead of Dynamodb.</p><p>We made this choice because for internal use the added complexity and cost were not worth it.</p><p>Here’s the result:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43c75242621ce0dc0a4cef36634e9292/href">https://medium.com/media/43c75242621ce0dc0a4cef36634e9292/href</a></iframe><h3>Integration in the current pipeline</h3><p>In the meantime, we’ve actually migrated on Kubernetes, with a CI on Jenkins.</p><p>So here is the package publishing pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*lyn1pAHSw7PYMObHaaCOEg.png" /><figcaption>Pypicloud package pipeline</figcaption></figure><p>For the package publication, we have a Github repository that contains Pypicloud configuration, a Dockerfile and Kubernetes resources for the server container that will be deployed.</p><p>This repository also contains other packages projects that will be tested in the CI before being built and published using a dedicated user, as we saw before this published package will be sent to Pypicloud to be stored on s3.</p><p>The Pypicloud URL is templated in the Dockerfile in order to be accessed during the container build phase and set by Jenkins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*hR2hj50QfT6VcrxmQRBQTQ.png" /><figcaption>Pypicloud projects pipeline</figcaption></figure><p>For the python projects deployment we have one repository per project, those repositories contain:</p><ul><li>Dockerfile</li><li>Python source code</li><li>Kubernetes configuration</li><li>Tests code</li><li>Jenkins configuration</li></ul><p>This pipeline doesn’t require a Pypicloud user, because the Ingress configuration of the server will simply reject access from outside the Kubernetes cluster.</p><h3>TL;DR</h3><p>We needed a remote package service in python and we looked at 3 solutions:</p><ul><li><a href="https://github.com/pypiserver/pypiserver">pypiserver</a></li><li><a href="http://novemberfiveco/s3pypi">S3pypi</a></li><li><a href="https://pypicloud.readthedocs.io/en/latest/">Pypicloud</a></li></ul><p>In the JobTeaser context:</p><ul><li>Infrastructure Kubernetes on AWS</li><li>Internal use only</li><li>Backup of package outside server</li><li>User management at the service level</li></ul><p>The winner was Pypicloud.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bc8d1e9d206" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/python-packaging-how-do-we-manage-it-at-jobteaser-2bc8d1e9d206">Python Packaging : How do we manage it at JobTeaser</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes Cronjob 101 part 1]]></title>
            <link>https://medium.com/jobteaser-dev-team/kubernetes-cronjob-101-56f0a8ea7ca2?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/56f0a8ea7ca2</guid>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[workflow-automation]]></category>
            <category><![CDATA[cronjob]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Tue, 28 Aug 2018 09:18:25 GMT</pubDate>
            <atom:updated>2019-09-04T14:50:39.376Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I will try to explain with some specific examples how to run Cronjob on Kubernetes.</p><p>Cronjob are basically Kubernetes jobs with a scheduling and some specific parameters to handle failure. Each Kubernetes job contains 1 to n pod.</p><p>Those pods runs a program defined for the task and are killed when the task is over.</p><h3>The need</h3><p>At JobTeaser we were migrating away from a machine learning recommendation service running on a (ec2 + Airflow + Flask) platform on which tasks were running each night, to Kubernetes. The reasoning behind this change was to improve the monitoring and scaling capacities of the whole system.</p><p>Since our new project didn’t require any complex workflow dependencies, someone suggested that, instead of our usual Airflow tasks where relationships between each operation are explicitly described by graphs, we should try using Kubernetes Cronjobs for extract load transform (ETL) batch scheduling.</p><p>Most teams would have said: “wait a minute, it’s totally insane to learn a new tech during a single cycle”, but we just said YOLO (Yahoo!!! an Opportunity to Learn the Obvious). So, here’s what we’ve discovered on the way.</p><h3>Setup</h3><p>So first you need to setup Kubernetes client kubectl following <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">official documentation</a>.</p><p>Then either you have a remote Kubernetes cluster <a href="https://www.techrepublic.com/article/how-to-quickly-install-kubernetes-on-ubuntu/">setup already</a> or you have a local <a href="https://kubernetes.io/docs/setup/minikube/">minikube installed</a>, both solutions work for this case.</p><h3>Deploy a job on Kubernetes in 5 min</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7rd92PzVFdiQScrHLrVIdQ.jpeg" /><figcaption>This is my cat Walter. He has no idea of what Cronjob is but he has it in him. In this picture you can see him in the middle of his sleeping task.</figcaption></figure><p>So… We’ll start with the most basic example we can use with the following cronjob configuration.</p><p>This cronjob will run one job every 5 minutes (using a<a href="https://crontab.guru/"> cron format</a> for scheduling) and display “Hello world”</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f45819e6955d703aaacd552e43695c0a/href">https://medium.com/media/f45819e6955d703aaacd552e43695c0a/href</a></iframe><p>To launch this job you just need to run the following command (‘create’ also works but don’t <a href="https://stackoverflow.com/questions/47369351/kubectl-apply-vs-kubectl-create">mix them together</a>)</p><pre>kubectl apply -f most_basic_cronjob.yaml</pre><p>To check that the cronjob was created just use</p><pre>kubectl get cronjob</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*vA-vs-SY-ZyM7lvU_ca8Mg.png" /><figcaption>before first schedule</figcaption></figure><p>So cronjob was named “test” based on the <em>metadata.name</em> value, the planning is the same as the one in <em>spec.schedule. </em>As a default function the job will run, so suspend value is False. There is currently no task running it (we didn’t plan it yet), Last Schedule and Age values are therefore void.</p><p>If we check the status of our scheduled jobs after 5 minutes, here’s what we’ll see.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*NjYIaBaDUPnI_DH55a3ZlA.png" /><figcaption>Active first schedule</figcaption></figure><p>We can see a job has been created and is currently running, Last schedule and Age are filled in. To check the job in detail we need to go down a notch.</p><pre>kubectl get job</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*l56giFEwBKLSk9gCE2sK2g.png" /></figure><p>We can see here that a job was a created with a unique name and has not been completed yet, you can find the related pod by using the command.</p><pre>kubectl describe job <em>name_of_job</em></pre><p>That will show the number of times the job has been restarted, full config of the job and the pod created. You can also use it on the cronjob itself.</p><pre>kubectl describe cronjob test</pre><p>Also if you want you can just run a job directly using the following (work with a kubectl version greater than 1.10.1)</p><p>kubectl create -f ./cronjob.yaml</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eQdJ-dW7-h3bPY71Q11Qyg.jpeg" /><figcaption>See, the sleep task has been completed and the play task is being activated.</figcaption></figure><h3>Live editing of the cronjob</h3><p>But even better than that, you can edit the cronjob on the fly by using the following command.</p><pre>kubectl edit cronjob test</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c6c73aa8a5601bf4fde39fd42a61750/href">https://medium.com/media/9c6c73aa8a5601bf4fde39fd42a61750/href</a></iframe><p>Wow! That’s a lot of information. Let’s remove some noise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e07b0331bb789cf718b2478f0d2d6c7/href">https://medium.com/media/8e07b0331bb789cf718b2478f0d2d6c7/href</a></iframe><p>Even better, now you may want to understand the parameters available when you’re editing. Some are mandatory, others optional — here’s the breakdown.</p><h3>Mandatory parameters with default values</h3><p><strong>concurrencyPolicy:</strong> This parameter is responsible for running your job in parallel with other jobs. The available values are either “Allow” or “Forbid”, but remember “Forbid” CANNOT GUARANTEE THAT ONLY ONE JOB WILL BE RUN AT ONCE.</p><p><strong>failedJobHistoryLimit </strong>and<strong> successfulJobsHistoryLimit </strong>are just here to help trim the output of `kubectl get jobs` by deleting older entries.</p><p><strong>suspend:</strong> If <em>false</em>, the cronjob is actively scheduling job, if <em>true</em> existing job will not be forced to terminate but new ones cannot be scheduled; this is useful for debugging when you want to stop your job from being launch or on the contrary set it to <em>true </em>by default and to launch it manually.</p><h3>Optional parameters</h3><p><strong>activeDeadlineSeconds: </strong>Doesn’t kill or stop the job itself but deletes the pod on error (the job will be replaced by a new one at the scheduled time).</p><p><strong>startingDeadlineSeconds: </strong>If for technical reasons (e.g. a cluster is down, …) a job cannot be started in the interval between<em> schedule time </em>and<em> startingDeadlineSeconds</em> it will not start at all. This is useful for long jobs when you should wait for the next one or short jobs with high frequency that can stack up if they aren’t stopped.</p><p><strong>backoffLimit: </strong>Number of retries for pods launched by the job. If you want your pods to never restart, you need to set it at 0. However due to some <a href="https://github.com/kubernetes/kubernetes/issues/54870">issue</a> where pod can’t be restarted beyond backoffLimit it’s better, if you use “<strong>restartPolicy</strong> : Never”.</p><p>Here is a more detailed cronjob example with most of the parameters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a985c408b4c759cb52cc88f0a79b1ff6/href">https://medium.com/media/a985c408b4c759cb52cc88f0a79b1ff6/href</a></iframe><p>What we have here is a job that will run a script called job_offers.py that is expected to run for 45 minutes each day at 16h30, 17h30, 18h30 and 19h30 UTC.</p><p>We don’t want multiple runs to be launched so we use <strong>currencyPolicy</strong> : “Forbid”. We don’t want it to runs for more than the expected time and overlap with the next execution time slot either, so we use <strong>startDeadlineSeconds</strong>: 600. This way a new job can only start when it has enough time to complete, like this:</p><p>16h30–16h40, 17h30-17h40, 18h30–18h40, and 19h30-19h40.</p><p>To complicate the exercise you can sometimes have jobs that will hang on a resource and go beyond 45min. In this case we want to kill this job and expect the next one to be completed. To do so, we specify an <strong>activeDeadlineSeconds</strong>: 3300 of 55minutes so the job will necessarily be killed before the next one.</p><p>We also want to keep better track of failure so we increase <strong>failedJobsHistoryLimit</strong> up to 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OY2nmvdAbJ10oBwriUNzfQ.jpeg" /><figcaption>Petting job is about to get started for my cat.</figcaption></figure><h3>Nota Bene</h3><p>One of the most confusing concepts of Cronjobs is that jobs can never fail. How come? There are 3 options of what can happen to them:</p><ul><li>Waiting for completion</li><li>Completed</li><li>Deleted</li></ul><p>If the pod that executes the job fails for any reason (Out of memory, code failure, etc) each job will clutter the listing of jobs waiting for completion but won’t prevent the next job from running because they don’t have an active pod. That’s why it’s important to remember to also check that the pods are still running when managing cronjob and if not to delete jobs that can no longer complete.</p><pre>kubectl delete job job_id</pre><p>Also, <strong>activeDeadlineSeconds </strong>will not kill the job on the spot. It will start by deleting the pod and you will have a trace of this event in the job info acquired by the command <strong>kubectl describe job <em>job_id</em>, </strong>but after a new job is scheduled this job will be deleted for good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*T2C2Op9dMN8rw8qupH9DGg.png" /><figcaption>What you will see</figcaption></figure><p>So far I would say that cronjobs aren’t a suitable solution that work for lot of data workflows mainly because you can’t have strong guarantees between jobs. However, it could be a good idea to use them if your jobs match the following criteria:</p><ul><li>Can be run concurrently (no lock on database)</li><li>Have predictable memory usage (In case you use Limit on the pod)</li><li>Job can handle dependent job failure</li></ul><p>To give you an example, here what we have currently:</p><ul><li>One job that imports data in one shot</li><li>3 incremental views, the incremental part is important because that help us to handle import failures for both job that rely on it.</li><li>A job doing the machine learning based recommendation, we only keep the last valid run.</li><li>All jobs run at least 3 time a day to catch potential failures early on and act quickly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*QAesvmr2U57BbekZ42XSsg.png" /></figure><p>Also it’s critical that you don’t rely too much on kubernetes commands to monitor your cronjob, at JobTeaser we use Datadog and ELK.</p><p>If you liked this article stay tuned for an upcoming part where we will explain how to run a Machine Learning pipeline from data import to the REST API exposition with Kubernetes.</p><p><em>Go here for the second part of </em><a href="https://medium.com/@cols.knil/cronjob-101-part-2-78503d3967b3"><em>cronjob 101</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56f0a8ea7ca2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/jobteaser-dev-team/kubernetes-cronjob-101-56f0a8ea7ca2">Kubernetes Cronjob 101 part 1</a> was originally published in <a href="https://medium.com/jobteaser-dev-team">JobTeaser Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Autoscaling in Dynamodb with boto3]]></title>
            <link>https://cols-knil.medium.com/autoscaling-in-dynamodb-with-boto3-bf5bbeb99b10?source=rss-a0844e3fa871------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf5bbeb99b10</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[boto3]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Clement Demonchy]]></dc:creator>
            <pubDate>Sun, 08 Apr 2018 11:16:07 GMT</pubDate>
            <atom:updated>2018-04-08T11:16:07.752Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post I will explain how to create table of dynamodb, that will scale in response to increasing requests, for complete details about what happens, read <a href="http://boto3.readthedocs.io/en/latest/reference/services/application-autoscaling.html">boto3 documentation</a></p><p>tldr : <a href="https://gist.github.com/knil-sama/e9b9c16bac6a3f6a6e1c80ce18361ab1">public gist with full code</a></p><p>The problem we were facing at <a href="https://www.jobteaser.com/">JobTeaser</a> was something you often have to deal when working on Cloud development, which is to properly manage resources on both financial and performance aspects.</p><p>We, at JobTeaser use Dynamodb to store ML computed results (as key-value {user_id: [id_offer_1, id_offer_2]}). Our workflow is as follows</p><ul><li>Load <strong>huge </strong>amount of ML results in dynamodb every night</li><li>Read each of them during the day in order to set up a newsletter</li></ul><p>By following the usual method to create tables in Dynamodb, in other words, by setting a low value for Read/Write capacity to save cost would slow down all associated operations. On the other hand, if we set a high value, we waste capacity 90% of the uptime.</p><p>The solution we came to was to optimize the performance/cost factor was to use <em>ApplicationAutoScaling</em> client from boto3.</p><p>Let’s create a table in dynamodb. To keep things simple, I will just set one key in my example.</p><p>We will have to set up values for <em>ReadCapacityUnits </em>and <em>WriteCapacityUnits </em>when creating the table that will be use as hard limit for the corresponding capacity, but this will soon become irrelevant once we start using autoscaling.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/534e057c90dc4f3b7dddb54577cb2989/href">https://medium.com/media/534e057c90dc4f3b7dddb54577cb2989/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*czeHkgLafoWcPYUkn7BHtA.png" /><figcaption>table created in AWS console</figcaption></figure><p>Now it’s time to begin the autoscaling by setting a lower and upper limit inside which the current capacity can fluctuate.</p><p>We do so using</p><pre>boto3.client(‘application-autoscaling’)</pre><p>with the following parameters.</p><ul><li><em>ServiceNamepace </em>as dynamodb¹</li><li><em>RessourceId</em> is in format “/table/&lt;table_name&gt;”</li><li>the <em>ScalableDimension</em> in this case<em> </em>are<em> ReadCapacityUnits </em>and <em>WriteCapacityUnits</em> but you can only put down one value and keep the other set.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5442af152c7b8c7e2ca74b4181f9d5b/href">https://medium.com/media/a5442af152c7b8c7e2ca74b4181f9d5b/href</a></iframe><p>We can now look at the interface and saw that both are marked as autoscaling</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JdMvCEIJrw4wqA8VUByE-g.png" /></figure><p>And that, is how it should work … But AWS has other plan coming</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*PbFzAG-3S0KKbKb9nskRxQ.png" /><figcaption>NOPE !</figcaption></figure><p>We still need to do one more thing i.e. havz AWS know <strong>when</strong> to scale our dynamodb resource and <strong>how</strong> we want it to do so.</p><p>What we need is to define a policy; we have choice between <em>TargetTrackingScaling</em> and <em>StepScaling</em>.</p><h3>TargetTrackingScaling</h3><p>This is the most straight forward method and the one I recommend; you will need to specify how much you want to use one resource in normal cases.</p><ul><li><em>ScalableDimension </em>here <em>dynamodb:table:ReadCapacityUnits </em>or <em>dynamodb:table:WriteCapacityUnits</em></li><li><em>TargetValue </em>percent of resource we expect to use in a typical situation</li><li><em>PredefinedMetricType</em> any existing CloudWatch metric that will be used to indicate if we need to scale up or down</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/178371e5197bfbe7ced821e1b86089f3/href">https://medium.com/media/178371e5197bfbe7ced821e1b86089f3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/794/1*rnUvJM82D5IYfBYiP3ztrQ.png" /></figure><p>We can now see the alarms created by our policy in CloudWatch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nxjQG-fujpvO-Y_nIi5BQ.png" /></figure><h3>StepScaling</h3><p>This is almost the same as the previous except we go into details on how the CloudWatch alarms will be created and the step value for every time we scale up or down.</p><ul><li><em>ChangeInCapacit</em>y: Add or substract to the current capacity (specified at the table creation time) the <em>ScalingAdjustment.</em></li><li><em>ExactCapacity: </em>Change the current capacity to the <em>ScalingAdjustment</em> value.</li><li><em>PercentChangeInCapacity: </em>Multiplies current capacity by the percent in <em>ScalingAdjustment.</em></li><li><em>ScalableDimension: </em>Here <em>dynamodb:table:ReadCapacityUnits </em>or <em>dynamodb:table:WriteCapacityUnits.</em></li><li><em>TargetValue: </em>Percent of resource we expect to use in normal situations.</li><li><em>PredefinedMetricType:</em> Any existing CloudWatch metric that will be used to indicate if we need to scale up or down</li></ul><p>There is actually a lot more options available like the <em>MetricAggregationType</em> but to keep this article short and simple, we will not go into further detail; you can found more informations in AWS <a href="https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-simple-step.html">documentation</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d26e14fdd38c707ba3358685b7f23b8/href">https://medium.com/media/8d26e14fdd38c707ba3358685b7f23b8/href</a></iframe><h3>Conclusion</h3><p>This helped to reduce greatly the time of our operations and I a second part we will take a look at our metrics and how the two approch differs in terms of result.</p><ol><li>We need to fill this info because this client is not restricted to dynamodb</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf5bbeb99b10" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>