<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anthony Potappel on Medium]]></title>
        <description><![CDATA[Stories by Anthony Potappel on Medium]]></description>
        <link>https://medium.com/@anthony.potappel?source=rss-99c3ef88ca96------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UqRpgEtWTGW-ofyV.</url>
            <title>Stories by Anthony Potappel on Medium</title>
            <link>https://medium.com/@anthony.potappel?source=rss-99c3ef88ca96------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:36:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anthony.potappel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Generating data 100x faster with Rust — Part II]]></title>
            <link>https://itnext.io/generating-data-100x-faster-with-rust-part-ii-49eaa0f388af?source=rss-99c3ef88ca96------2</link>
            <guid isPermaLink="false">https://medium.com/p/49eaa0f388af</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Anthony Potappel]]></dc:creator>
            <pubDate>Mon, 05 Feb 2024 08:39:15 GMT</pubDate>
            <atom:updated>2024-03-19T12:18:53.621Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGF_c-OXT3hNo7u22ft4uA.jpeg" /><figcaption>Picture of scaling up data generation</figcaption></figure><h3>Generating data 100x faster with Rust — Part II</h3><p><strong>Welcome to Part II on faster Data Generation with Rust</strong></p><p>In <a href="https://medium.com/itnext/generating-data-100x-faster-with-rust-part-i-713182122d53">part I</a> we showed Rust’s potential to generate data over 100x faster compared to the widely used Python <a href="https://faker.readthedocs.io/en/master/">Faker</a> library.</p><p>Now, in Part II, we shift our focus to transforming our code into a practical Command Line Interface (CLI) tool. Our objectives in this second part include unlocking the potential of multi-threading, efficiently handling data as a DataFrame using the <a href="https://pola.rs/">Polars</a> library, and storing the data on disk as a Parquet file.</p><p>All referenced code is available in this <a href="https://github.com/aprxi/faster-data-generation">repository</a>.</p><p><strong>Command Line Interface (CLI) parser</strong></p><p>Our users would probably like to run different configurations without modifying the code. Additionally, the tool must fit in automated workflows and CI pipelines, with the capability to be configured via environment variables. In Rust, these objectives are achieved by importing the Clap library and writing a parser function.</p><pre>use std::env;<br>use clap::{Arg, Command};<br><br>const PROGRAM_NAME: &amp;str = &quot;rsfake&quot;;<br>const DEFAULT_SCHEMA_FILE: &amp;str = &quot;schema.json&quot;;<br>const DEFAULT_NO_ROWS: &amp;str = &quot;10000&quot;;<br>const RAYON_NUM_THREADS: &amp;str = &quot;1&quot;;<br><br><br>fn parse_cli_arguments() -&gt; Command {<br>    Command::new(PROGRAM_NAME)<br>        .version(env!(&quot;CARGO_PKG_VERSION&quot;)) // set version from Cargo.toml<br>        .about(&quot;Generates fake data based on the provided schema file.&quot;)<br>        .long_about(format!(<br>            &quot;This program generates fake data based on a JSON schema file. \<br>            You can specify the number of rows, the number of threads for \<br>            parallel processing, and the schema file to be used.\n\n\<br>            Example usage:\n    {} -s schema.json -r {} -t {}&quot;,<br>            PROGRAM_NAME, DEFAULT_NO_ROWS, RAYON_NUM_THREADS<br>        ))<br>        .arg(<br>            Arg::new(&quot;schema&quot;)<br>                .short(&#39;s&#39;)<br>                .long(&quot;schema&quot;)<br>                .env(&quot;FAKER_SCHEMA_FILE&quot;)<br>                .value_name(&quot;SCHEMA_FILE&quot;)<br>                .help(&quot;JSON file to describe column names and types&quot;)<br>                .default_value(DEFAULT_SCHEMA_FILE),<br>        )<br>        .arg(<br>            Arg::new(&quot;rows&quot;)<br>                .short(&#39;r&#39;)<br>                .long(&quot;rows&quot;)<br>                .env(&quot;FAKER_NUM_ROWS&quot;)<br>                .value_name(&quot;NUM_ROWS&quot;)<br>                .help(&quot;Number of rows to generate&quot;)<br>                .default_value(DEFAULT_NO_ROWS),<br>        )<br>        .arg(<br>            Arg::new(&quot;threads&quot;)<br>                .short(&#39;t&#39;)<br>                .long(&quot;threads&quot;)<br>                .env(&quot;RAYON_NUM_THREADS&quot;)<br>                .value_name(&quot;NO_THREADS&quot;)<br>                .help(&quot;Number of threads to use&quot;)<br>                .default_value(RAYON_NUM_THREADS),<br>        )<br>        .arg(<br>            Arg::new(&quot;output&quot;)<br>                .short(&#39;o&#39;)<br>                .long(&quot;output&quot;)<br>                .env(&quot;FAKER_OUTPUT_PATH&quot;)<br>                .value_name(&quot;OUTPUT_PATH&quot;)<br>                .help(&quot;Output path to write to&quot;),<br>        )<br>        .arg(<br>            Arg::new(&quot;input&quot;)<br>                .short(&#39;i&#39;)<br>                .long(&quot;input&quot;)<br>                .env(&quot;FAKER_INPUT_PATH&quot;)<br>                .value_name(&quot;INPUT_PATH&quot;)<br>                .help(&quot;Input path to read from&quot;),<br>        )<br>}</pre><p>In this parser function, all arguments are optional, with some assigned default values that are utilized if not specified by the user. The last two arguments — output to Parquet and input from Parquet — can remain undefined; the associated code executes only when values for these arguments are provided.</p><p>Furthermore, these arguments can be set via environment variables, this improves the tool’s usability in containerized or automated contexts.</p><p>Then, in the main function, we process the input arguments through this parser and use the resulting values in our application.</p><pre>fn main() {<br>    let args: Vec&lt;String&gt; = env::args().collect();<br>    let app = parse_cli_arguments();<br>    let matches = app.try_get_matches_from(args).unwrap_or_else(|e| {<br>        e.exit();<br>    });<br><br>    let schema_file = matches<br>        .get_one::&lt;String&gt;(&quot;schema&quot;)<br>        .expect(&quot;Failed to parse schema file&quot;);<br><br>    // additional check to see if chema file exists<br>    if !std::path::Path::new(&amp;schema_file).exists() {<br>        println!(&quot;Schema file \&quot;{}\&quot; does not exist&quot;, schema_file);<br>        parse_cli_arguments().print_help().unwrap();<br>        std::process::exit(1);<br>    }<br><br>    let no_threads = matches<br>        .get_one::&lt;String&gt;(&quot;threads&quot;)<br>        .map(|s| s.parse::&lt;usize&gt;().expect(&quot;Failed to parse thread count&quot;))<br>        .expect(&quot;Failed to parse default thread count&quot;);<br><br>    let no_rows = matches<br>        .get_one::&lt;String&gt;(&quot;rows&quot;)<br>        .map(|s| s.parse::&lt;usize&gt;().expect(&quot;Failed to parse row count&quot;))<br>        .expect(&quot;Failed to parse default row count&quot;);<br><br>    // Optional - only run when either is defined<br>    let output_path = matches.get_one::&lt;String&gt;(&quot;output&quot;);<br>    let input_path = matches.get_one::&lt;String&gt;(&quot;input&quot;);<br><br>    // rest of the code<br>}<br></pre><p><strong>Loading a schema dynamically</strong></p><p>In <a href="https://medium.com/itnext/generating-data-100x-faster-with-rust-part-i-713182122d53">part I</a> data generation is based on a predefined struct. While functional, this method lacks flexibility, because the code must be recompiled if the schema changes. To overcome this inconvenience, we move the schema definition to a JSON file. This adjustment enables users to modify the schema without having to recompile the code.</p><p>The JSON file contents, detailed below, describes the data for each column.</p><pre>{<br>  &quot;columns&quot;: [<br>    {<br>      &quot;name&quot;: &quot;id&quot;,<br>      &quot;type&quot;: &quot;u64&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;first_name&quot;,<br>      &quot;type&quot;: &quot;FirstName&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;last_name&quot;,<br>      &quot;type&quot;: &quot;LastName&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;email&quot;,<br>      &quot;type&quot;: &quot;FreeEmail&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;company&quot;,<br>      &quot;type&quot;: &quot;CompanyName&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;phone&quot;,<br>      &quot;type&quot;: &quot;PhoneNumber&quot;<br>    }<br>  ]<br>}</pre><p>To integrate the schema into our application, we utilize <a href="https://crates.io/crates/serde">Serde</a>, a popular data (de-)serialization library in the Rust ecosystem. Serde handles export/ import operations to and from file formats like JSON or YAML.</p><pre>pub fn load_json(json_file: &amp;str) -&gt; Result&lt;Value, Box&lt;dyn std::error::Error&gt;&gt; {<br>    let json_str = fs::read_to_string(json_file)?;<br>    let json: Value = serde_json::from_str(&amp;json_str)?;<br>    Ok(json)<br>}</pre><p><strong>Data Generation into a DataFrame</strong></p><p>The types from JSON schema are mapped to corresponding functions in the Faker library, generating a Series (column) for each type. These Series are then aggregated into a DataFrame (table).</p><p>To build these Series and the resultant DataFrame, we use <a href="https://pola.rs/">Polars</a>, a high-performance DataFrame library written in Rust. Polars is not only fast and efficient, but also offers a <a href="https://pypi.org/project/polars/">Python API</a>, providing a user experience similar to that of <a href="https://pandas.pydata.org/">Pandas </a>— the well-known library that popularized DataFrames in Python.</p><pre>fn create_series_from_type&lt;L&gt;(<br>    type_name: &amp;str,<br>    col_name: &amp;str,<br>    no_rows: usize,<br>    locale: L,<br>) -&gt; Series<br>where<br>    L: Data + Sync + Send + Copy,<br>{<br>    match type_name {<br>        &quot;FirstName&quot; =&gt; {<br>            let data: Vec&lt;String&gt; = (0..no_rows)<br>                .map(|_| FirstName(locale).fake::&lt;String&gt;())<br>                .collect();<br>            Series::new(col_name, data)<br>        }<br>        // other type_names similar<br>        ... <br>    }<br>}</pre><p>In conclusion, the core function of our data generation process involves loading the JSON schema, interpreting each column definition, generating a Series for each type, and finally aggregating these Series into a unified DataFrame. This completes the data generation workflow.</p><pre>pub fn generate_from_json(<br>    json_file: &amp;str,<br>    no_rows: usize,<br>) -&gt; Result&lt;DataFrame, Box&lt;dyn std::error::Error&gt;&gt; {<br>    let json = load_json(json_file)?;<br><br>    let mut columns = Vec::new();<br><br>    if let Some(columns_def) = json.get(&quot;columns&quot;).and_then(|c| c.as_array()) {<br>        for col_def in columns_def {<br>            let col_name = col_def<br>                .get(&quot;name&quot;)<br>                .and_then(|n| n.as_str())<br>                .unwrap_or_default();<br>            let col_type = col_def<br>                .get(&quot;type&quot;)<br>                .and_then(|t| t.as_str())<br>                .unwrap_or_default();<br><br>            let series_en = create_series_from_type(col_type, col_name, no_rows, EN);<br>            columns.push(series_en);<br>        }<br>    }<br>    Ok(DataFrame::new(columns)?)<br>}</pre><p>Now we can run the code using a separately managed JSON file.</p><pre># compile and test (fast build, slow execution)<br>cargo run -- -s schema.json<br><br># compile for release (slow build, fast execution)<br>cargo build --release<br>target/release/rsfake -s schema.json</pre><p>This setup allows to customize which columns to include, as well as their names and order, simply by modifying the JSON file. For an additional exercise, consider expanding the code to incorporate more types. The <a href="https://crates.io/crates/fake">Fake.rs</a> library offers a variety of available types that you could explore.</p><p><strong>Implement Multi-Threading</strong></p><p>To further boost performance, we use multi-threading. This technique divides the workload into smaller tasks. By processing these tasks concurrently across multiple CPU cores, the overall work is completed in a reduced time frame.</p><p>To implement multi-threading we use the <a href="https://crates.io/crates/rayon">Rayon.rs</a> library. Below is the create_series_from_type() method from earlier, but now with the additional method into_par_iter() added.</p><pre>fn create_series_from_type&lt;L&gt;(<br>    type_name: &amp;str,<br>    col_name: &amp;str,<br>    no_rows: usize,<br>    locale: L,<br>) -&gt; Series<br>where<br>    L: Data + Sync + Send + Copy,<br>{<br>    match type_name {<br>        &quot;FirstName&quot; =&gt; {<br>            let data: Vec&lt;String&gt; = (0..no_rows)<br>                .into_par_iter()  // enable multi-processing<br>                .map(|_| FirstName(locale).fake::&lt;String&gt;())<br>                .collect();<br>            Series::new(col_name, data)<br>        }<br>        // other type_names similar<br>        ... <br>    }<br>}<br></pre><p>The additional <a href="https://docs.rs/rayon/latest/rayon/iter/trait.IntoParallelIterator.html">into_par_iter()</a> provides a parallel iterator instead of the (default) sequential iterator. As a result, the Faker FirstName() call is run in parallel across multiple threads, which the operating system can distribute over available CPU cores.</p><pre># compile for release<br>cargo build --release<br><br># generate million rows on single thread (default)<br>target/release/rsfake -r 1000000<br><br># generate on 2, 4, 8 threads<br>target/release/rsfake -r 1000000 -t 2<br>target/release/rsfake -r 1000000 -t 4<br>target/release/rsfake -r 1000000 -t 8</pre><p>This outputs:</p><pre>Time taken to generate 1000000 people into a dataframe using 1 threads:<br>--- 1.267 seconds ---<br><br>Time taken to generate 1000000 people into a dataframe using 2 threads:<br>--- 0.740 seconds ---<br><br>Time taken to generate 1000000 people into a dataframe using 4 threads:<br>--- 0.486 seconds ---<br><br>Time taken to generate 1000000 people into a dataframe using 8 threads:<br>--- 0.357 seconds ---</pre><p>Using more threads helps our program to finish the job faster. However, as we scale our implementation, we do not observe perfect linearity in performance. This becomes particularly evident with higher thread counts. As we scale up, we encounter new system bottlenecks that diminish the incremental benefits of additional threads.</p><p>The emergence of new bottlenecks while scaling is not entirely unexpected. Although computation itself is more efficient, other system aspects, such as memory bandwidth, have their own limits. This is a limitation when scaling up on a single host.</p><p>As a side note, encountering these bottlenecks this quickly does indicate significant untapped potential for further performance improvements — this is something that is definately worth exploring in a separate story.</p><p><strong>Export the data to Parquet</strong></p><p>As a last and final step, we store the data on disk in Parquet format. The Parquet format is well-suited for storing tabular data intended for analytical purposes.</p><p>We have the option to consolidate the data into a single Parquet file or employ a partitioned strategy, spreading the data across multiple files.</p><p><strong>Single Parquet file</strong></p><p>For smaller datasets, typically less than 1GB, writing to a single Parquet file is efficient and straightforward. With <a href="https://pola.rs/">Polars</a>, exporting the DataFrame to a single Parquet file requires just a few lines of code.</p><pre>pub fn write_dataframe_to_single_parquet(<br>    df: &amp;mut DataFrame,<br>    file_path: &amp;str<br>) -&gt; Result&lt;(), Box&lt;dyn Error&gt;&gt; {<br>    let file = File::create(file_path)?;<br>    let writer = BufWriter::new(file);<br>    ParquetWriter::new(writer).finish(df)?;<br>    Ok(())<br>}</pre><p><strong>Multi-/ Partitioned Parquet file</strong></p><p>When dealing with larger datasets, it is common practice to distribute the data across multiple Parquet files, which are collectively stored in a directory structure, creating a partitioned Parquet file setup.</p><p>Partitioning is typically based on the anticipated patterns of data access, enabling efficient query handling for specific segments of the data, such as by column or date. In situations where data is processed in parallel, it’s typical to find multiple Parquet files within each partition. This is because parallel workers are configured to read from or write to a single file within the structure.</p><pre>sales_data.parquet/<br>│<br>├── year=2020/<br>│   ├── country=NL/<br>│   │   ├── part-00001.parquet<br>│   │   ├── part-00002.parquet<br>│   │   └── ...<br>│   └── ...<br>├── year=2021/<br>│   ├── country=UK/<br>│   │   ├── part-00001.parquet<br>│   │   ├── part-00002.parquet<br>│   │   └── ...<br>│   │<br>│   └── ...</pre><p>It’s important to highlight that in a partitioned Parquet file setup, each individual Parquet file can be independently imported as a complete DataFrame. Every file encompasses all columns and rows pertinent to a particular data subset, as determined by the partitioning criteria.</p><p><strong>Our Dataset in Parquet</strong><br>For our data generation case, each dataset is a collection of random names and addresses, to be used in separate test-runs. Therefore it makes most sense to partition based on an external attribute like dataset name or id, rather than on inherent data attributes such as columns or dates. This looks as follows;</p><p>For our data generation case, each dataset is a collection of random names and addresses, to be used in separate test-runs. Therefore it makes most sense to partition based on an external attribute like the name or id of the dataset, rather than on inherent data attributes such as columns or dates.</p><pre>output.parquet/<br>│<br>├── dataset=0/<br>│   ├── part-00000.parquet<br>│   ├── part-00001.parquet<br>│   └── ...</pre><p>The code for writing a partitioned Parquet file is similar to that used for writing a single file. The key difference is generating the appropriate file path.</p><pre><br>pub fn write_dataframe_chunk_to_parquet(<br>    df_chunk: &amp;mut DataFrame, <br>    dataset_id: &amp;str, <br>    base_dir: &amp;str,<br>    part_number: usize<br>) -&gt; Result&lt;(), Box&lt;dyn Error&gt;&gt; {<br>    // Path for the dataset directory<br>    let dataset_dir = format!(&quot;{}/dataset={}&quot;, base_dir, dataset_id);<br><br>    // Ensure the dataset directory exists<br>    if !std::path::Path::new(&amp;dataset_dir).exists() {<br>        fs::create_dir_all(&amp;dataset_dir)?;<br>    }<br>    // Generate the part file name<br>    let file_path = format!(&quot;{}/part-{:05}.parquet&quot;, dataset_dir, part_number);<br><br>    // Write the DataFrame chunk to the Parquet file<br>    let file = File::create(&amp;file_path)?;<br>    let writer = BufWriter::new(file);<br>    ParquetWriter::new(writer).finish(df_chunk)?;<br>    Ok(())<br>}</pre><p>The function above handles writing one part of the data and is invoked by this next function, that segments the DataFrame into smaller parts.</p><pre><br>pub fn write_dataframe_to_multi_parquet(<br>    df: &amp;DataFrame,<br>    dataset_id: &amp;str,<br>    base_dir: &amp;str,<br>    chunk_size: usize,<br>) -&gt; Result&lt;(), Box&lt;dyn Error&gt;&gt; {<br>    // Ensure the base directory and dataset directory exist<br>    let dataset_dir = format!(&quot;{}/dataset={}&quot;, base_dir, dataset_id);<br><br>    // create dataset directory if not exist, else clean up<br>    if !std::path::Path::new(&amp;dataset_dir).exists() {<br>        fs::create_dir_all(&amp;dataset_dir)?;<br>    } else {<br>        cleanup_dataset_parquet_files(&amp;dataset_dir)?;<br>    }<br><br>    let n_rows = df.height();<br>    let mut part_number = 0;<br><br>    for start in (0..n_rows).step_by(chunk_size) {<br>        let end = std::cmp::min(start + chunk_size, n_rows);<br>        let chunk = df.slice(start as i64, end - start);<br><br>        // Convert chunk to mutable for writing<br>        let mut chunk_mut = chunk.clone();<br><br>        // write the chunk<br>        write_dataframe_chunk_to_parquet(<br>            &amp;mut chunk_mut, dataset_id, base_dir, part_number)?;<br>        part_number += 1;<br>    }<br>    Ok(())<br>}</pre><p>It’s worth noting that in this implementation, while data generation is parallelized, the output files are written sequentially. Technically, we could modify the code to enable concurrent disk writes by each thread, but this approach would still have limited scalability due to the inherent write speed constraints of a single system.</p><p>To scale more effectively, we need to ‘scale out’ by running multiple instances of our program. Each instance would then write to a partitioned Parquet file on a distributed storage system that scales alongside, matching the high performance of our computational process.</p><p><strong>Next steps</strong></p><p>This marks the conclusion of the second installment in our series. We have effectively evolved our code into a versatile CLI tool, augmented with multi-threading, and adept at storing substantial datasets in Parquet format on disk.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49eaa0f388af" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/generating-data-100x-faster-with-rust-part-ii-49eaa0f388af">Generating data 100x faster with Rust — Part II</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating data 100x faster with Rust — Part I]]></title>
            <link>https://itnext.io/generating-data-100x-faster-with-rust-part-i-713182122d53?source=rss-99c3ef88ca96------2</link>
            <guid isPermaLink="false">https://medium.com/p/713182122d53</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[fakers]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Anthony Potappel]]></dc:creator>
            <pubDate>Tue, 02 Jan 2024 12:33:25 GMT</pubDate>
            <atom:updated>2024-02-05T09:31:30.204Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*fmuNV4_7dhVWH2SwxSvCnA.jpeg" /><figcaption>Picture of a High performance Rust processor</figcaption></figure><h3>Generating data 100x faster with Rust — Part I</h3><p>Testing and assessing data tools requires realistic test data, and for this task, I’ve frequently turned to Faker — a popular Python library for data generation.</p><p>However, as I started handling larger datasets — up to 1GB, sizeable, but not excessive — Faker’s performance constraints became problematic.</p><p>Python’s speed, or lack thereof, is well-known. But libraries can counter this by delegating heavy tasks to efficient, lower-level languages like C, C++, or Rust. Interestingly, Faker hasn’t gone down this road — at least not yet.</p><p>With my increasing passion for Rust, I couldn’t resist the urge to explore how Rust should be able to tackle this problem by offering a more efficient implementation.</p><p>Let’s find out!</p><p><strong>A case of generating a billion people</strong></p><p>In search of a challenging dataset, I came across <a href="https://motherduck.com/blog/python-faker-duckdb-exploration/">this article</a> about the creation of an extensive 36GB dataset, with data on one billion people.</p><p>While the article’s objective is to demonstrate Duckdb’s impressive querying capabilities, what caught my attention was the described method of data generation with Python Faker.</p><p>The process required an expensive 128-core machine to run for two hours. Given a modern day CPU can crunch gigabytes of data per second this highlights a significant opportunity for optimization.</p><p>The code creates a list of tuples, each representing an individual. I adopted most of the code as-is — be it with some minor adjustments. On analysis, it became clear that data generation consumed virtually all (&gt;95%) compute, so I left out the DataFrame and Parquet parts in this first part of the series— we’ll dig into that later.</p><pre>import sys<br>import random<br>import time<br><br>from faker import Faker<br>from typing import Any<br><br>NO_ROWS = 10000<br><br>fake = Faker()<br><br>def get_person() -&gt; dict[str, Any]:<br>    person = {<br>        &quot;id&quot;: random.randrange(1000, 9999999999999),<br>        &quot;first_name&quot;: fake.first_name(),<br>        &quot;last_name&quot;: fake.last_name(),<br>        &quot;email&quot;: fake.unique.ascii_email(),<br>        &quot;company&quot;: fake.company(),<br>        &quot;phone&quot;: fake.phone_number()<br>    }<br>    return person<br><br><br>def generate_person_list(count: int) -&gt; list[dict[str, Any]]:<br>    person_list = [get_person() for _ in range(count)]<br>    return person_list<br><br><br>def main() -&gt; int:<br>    start_time = time.time()<br>    person_list = generate_person_list(NO_ROWS)<br>    end_time = time.time()<br><br>    print(&quot;First 3 records:&quot;, person_list[:3])<br>    print(f&quot;Time taken to generate {NO_ROWS} people:&quot;)<br>    print(f&quot;--- {round((end_time - start_time), 3)} seconds ---&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    sys.exit(main())</pre><pre># run the code<br>python pyfake/generate_row.py<br><br>First 3 records:<br>....<br>Time taken to generate 10000 people:<br>--- 2.163 seconds ---</pre><p><strong>Python update</strong></p><p>Before moving to Rust, I explored a different approach within Python to potentially improve performance. I modified the original implementation, aiming for a more efficient way to handle the data.</p><p>My expectation was that writing the data via this new approach (lists of attributes instead of list of people; i.e. row -&gt; columnar) should run faster;</p><pre>import sys<br>import random<br>import time<br><br>from faker import Faker<br><br>NO_ROWS = 10000<br><br>fake = Faker()<br><br>class ColumnTable:<br>    def __init__(self, count: int):<br>        self.ids = [<br>            random.randrange(1000, 9999999999999) for _ in range(count)<br>        ]<br>        self.first_names = [fake.first_name() for _ in range(count)]<br>        self.last_names = [fake.last_name() for _ in range(count)]<br>        self.emails = [fake.unique.ascii_email() for _ in range(count)]<br>        self.companies = [fake.company() for _ in range(count)]<br>        self.phone_numbers = [fake.phone_number() for _ in range(count)]<br><br><br>def main() -&gt; int:  <br>    start_time = time.time()<br>    table = ColumnTable(NO_ROWS)<br>    end_time = time.time()<br><br>    print(&quot;First 3 records:&quot;)<br>    for i in range(3):<br>        print(<br>            f&quot;Record {i + 1}: {{ id: {table.ids[i]}, &quot;<br>            f&quot;first_name: \&quot;{table.first_names[i]}\&quot;, &quot;<br>            f&quot;last_name: \&quot;{table.last_names[i]}\&quot;, &quot;<br>            f&quot;email: \&quot;{table.emails[i]}\&quot;, &quot;<br>            f&quot;company: \&quot;{table.companies[i]}\&quot;, &quot;<br>            f&quot;phone_number: \&quot;{table.phone_numbers[i]}\&quot; }}&quot;<br>        )<br><br>    print(f&quot;Time taken to generate {NO_ROWS} people:&quot;)<br>    print(f&quot;--- {round((end_time - start_time), 3)} seconds ---&quot;)<br>    return 0<br><br><br>if __name__ == &quot;__main__&quot;:<br>    sys.exit(main())</pre><p>Let&#39;s run the code;</p><pre># run the code<br>python pyfake/generate.py <br><br>First 3 records:<br>....<br>Time taken to generate 10000 people:<br>--- 2.028 seconds ---</pre><p>To my surprise, this was not actually much faster in Python. Converting the table to a DataFrame did not matter that much either. In-memory it&#39;s all equally fast.</p><p>Still, there is a tiny (2-5%) performance bump — when looking hard enough — on the generating part that was fairly consistent on the second approach.</p><p>The final scores (average from 10 runs, 10K people) are as follows:</p><pre># benchmark row (average over 10 runs)<br>python -c &#39;import pyfake; pyfake.benchmark_row()&#39;<br><br>Average time taken to generate 10000 people:<br>--- 2.171 seconds ---<br><br># benchmark column (average over 10 runs)<br>python -c &#39;import pyfake; pyfake.benchmark_column()&#39;<br><br>Average time taken to generate 10000 people:<br>--- 2.116 seconds ---</pre><p>Note this is also still based on using 1 thread. We’ll add multi-threading in the next part of this series. You can find all the code in this <a href="https://github.com/aprxi/faster-data-generation">repository</a> and play with it yourself. Your hardware may yield different results.</p><p><strong>Rust implementation</strong></p><p>As we transition to the Rust implementation, you may notice a parallel between the Python version’s class and the Rust code’s struct. Understanding that Rust’s structs are similar to Python’s classes helped me a lot in grasping their usefulness.</p><p>Lets look at the Rust implementation and run it;</p><pre>use std::time::Instant;<br>use fake::Dummy;<br>use fake::{Fake, Faker};<br><br>use fake::faker::name::en::*;<br>use fake::faker::internet::en::*;<br>use fake::faker::company::en::*;<br>use fake::faker::phone_number::en::*;<br><br><br>const NO_ROWS: usize = 10000;<br><br><br>#[derive(Debug, Dummy)]<br>struct TableColumns {<br>    #[dummy(faker = &quot;(1000..9999999999999, NO_ROWS)&quot;)]<br>    pub ids: Vec&lt;i64&gt;,<br><br>    #[dummy(faker = &quot;(FirstName(), NO_ROWS)&quot;)]<br>    pub first_names: Vec&lt;String&gt;,<br><br>    #[dummy(faker = &quot;(LastName(), NO_ROWS)&quot;)]<br>    pub last_names: Vec&lt;String&gt;,<br><br>    #[dummy(faker = &quot;(FreeEmail(), NO_ROWS)&quot;)]<br>    pub emails: Vec&lt;String&gt;,<br><br>    #[dummy(faker = &quot;(CompanyName(), NO_ROWS)&quot;)]<br>    pub companies: Vec&lt;String&gt;,<br><br>    #[dummy(faker = &quot;(PhoneNumber(), NO_ROWS)&quot;)]<br>    pub phone_numbers: Vec&lt;String&gt;,<br>}<br><br><br>fn generate_table() {<br>    let start_time = Instant::now();<br>    let table: TableColumns = Faker.fake();<br>    let elapsed = start_time.elapsed().as_secs_f64();<br><br>    println!(&quot;First 3 records:&quot;);<br>    for i in 0..3 {<br>        println!(<br>        &quot;Record {}: {{ id: {}, first_name: \&quot;{}\&quot;, last_name: \&quot;{}\&quot;,\<br>         email: \&quot;{}\&quot;, company: \&quot;{}\&quot;, phone_number: \&quot;{}\&quot; }}&quot;, <br>            i + 1,<br>            table.ids[i], <br>            table.first_names[i], <br>            table.last_names[i], <br>            table.emails[i], <br>            table.companies[i], <br>            table.phone_numbers[i]<br>        );<br>    }<br><br>    println!(&quot;Time taken to generate {NO_ROWS} people:&quot;);<br>    println!(&quot;--- {:.3} seconds ---&quot;, elapsed);<br>}<br><br><br>fn main() {<br>    generate_table();<br>}</pre><pre># compile and run<br>cargo run<br><br>...<br>Running `target/debug/rsfake`<br>First 3 records:<br>...<br>Time taken to generate 10000 people:<br>--- 0.111 seconds ---</pre><p>That is not 100x faster! That is correct. This is because cargo run is optimized to minimize compile time, so it is quicker to test if code runs.</p><p>Let&#39;s try the optimized version;</p><pre># compile for release<br>cargo build --release<br><br># run compiled binary<br>target/release/rsfake<br><br>First 3 records:<br>...<br>Time taken to generate 10000 people:<br>--- 0.015 seconds ---</pre><p>This run is 133x faster. You may notice some variation in the performance gain still. Feel free to bump up the number of records from 10K to 100K or 1M to find out how it performs on larger numbers — you may be pleasantly surprised, just change the number for NO_ROWS in the code.</p><p><strong>Next steps</strong></p><p>You might be wondering: it’s impressive that the Rust implementation is so much faster, but how does one transform this into a usable DataFrame or a Parquet file? Indeed, there are a few more steps to cover:</p><ul><li>add CLI (Command Line Interface) parser functionality</li><li>convert the generated data into a DataFrame</li><li>export the data to a Parquet file</li><li>enable multi-threading to scale up the process — we’re aiming for 1 billion records, after all</li></ul><p>I will cover this in part II of this series. If you don’t like to wait, the code is already in this <a href="https://github.com/aprxi/faster-data-generation">repository</a>. It may still have some rough edges, but you’re welcome to explore and use this as you see fit.</p><p>Update: <a href="https://medium.com/p/49eaa0f388af">click here to read part II</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=713182122d53" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/generating-data-100x-faster-with-rust-part-i-713182122d53">Generating data 100x faster with Rust — Part I</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create, build and ship a Python3 pip module in 5 minutes]]></title>
            <link>https://itnext.io/create-build-and-ship-a-python3-pip-module-in-5-minutes-31dd6d9d5c8f?source=rss-99c3ef88ca96------2</link>
            <guid isPermaLink="false">https://medium.com/p/31dd6d9d5c8f</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Anthony Potappel]]></dc:creator>
            <pubDate>Fri, 27 Sep 2019 11:31:19 GMT</pubDate>
            <atom:updated>2023-12-24T13:40:41.396Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jXbr1V27rFkFMlN4" /><figcaption>Photo by <a href="https://unsplash.com/@randyfath?utm_source=medium&amp;utm_medium=referral">Randy Fath</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If you use Python, you likely encountered <strong>pip</strong> on your path. Pip is the most used package manager for Python modules.</p><blockquote><strong>Pip</strong> is a <a href="https://en.wikipedia.org/wiki/Recursive_acronym">recursive acronym</a> for “Pip Installs Packages” [<a href="https://en.wikipedia.org/wiki/Pip_(package_manager)">Wikipedia</a>]</blockquote><p>Last year was the <a href="https://pypi.org/project/pip/0.2/">10th birthday</a> of pip, which makes it a <a href="https://www.joelonsoftware.com/2001/07/21/good-software-takes-ten-years-get-used-to-it/">good software</a> — a rarity these days. I remember the time when pip was considered a liability. Pip had this label of being a second rebellious captain on a ship, known to conflict with the distribution’s primary packaging system, bringing up all sorts of security and stability related concerns.</p><p>Today, with containers — such as Docker — the worries are gone. Application code and their dependencies run in a controlled, isolated environment. Testing for vulnerabilities is easier, and no more need to fight the sys-admin.</p><p>Python’s use and popularity are skyrocketing as well — thanks to machine learning for that. This makes the world hungry for more Python modules.</p><p>Let’s ship some modules.</p><h3>Shipping pip packages — the automated way</h3><h4>Prerequisites</h4><ul><li>PyPi account. Register <a href="https://pypi.org/account/register/">here</a> and save your credentials in a safe place.</li><li>Docker. Install instructions: <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce">Ubuntu</a> or <a href="https://docs.docker.com/docker-for-mac/install/">Mac</a>.</li></ul><p>You also need the ability to run make, have a POSIX compliant shell and a code editor — nearly every Linux-, Unix- or Mac system will work.</p><h4>PyShipper</h4><p><a href="https://github.com/LINKIT-Group/pyshipper">PyShipper</a> is an automation part, doing the grunt-work of setting up a module structure, adds some boilerplate code, and provides a pipeline to build, test and ship a module — saving time when you update or create a module.</p><p>The relevant code is mostly in Makefile, complemented with some Docker configuration and a slightly modified setup.py. For the curious readers, this previous <a href="https://itnext.io/docker-makefile-x-ops-sharing-infra-as-code-parts-ea6fa0d22946?source=friends_link&amp;sk=1c42525c25039efadcbd25776a3019dd">Docker &amp; Makefile</a> article explains the base. <a href="https://github.com/LINKIT-Group/pyshipper">PyShipper</a> is a version that does the job of Python module delivery.</p><h4>Steps</h4><ol><li>Think of a name</li><li>Create a new module directory</li><li>Configure variables</li><li>Run and edit code</li><li>Make module</li><li>Publish</li></ol><p>I will go through each step, and explain where and how PyShipper kicks in to automate a few things along the way.</p><h4>1. Think of a name</h4><p>If you find it difficult to come up with a good name, you are not alone:</p><blockquote>There are only two hard things in Computer Science: cache invalidation and naming things. — Phil Karlton</blockquote><p>I always struggle with naming things. After hours in slow thinking mode, I usually end up with a name that describes what problem is solved — hence the name PyShipper. If a name is taken, pre- or postfixing the name with “Py”, “Script”, “Tool”, or something similar usually works out.</p><p>Naming is one of these things I have not yet figured out how to automate. If there are any machine learning related ideas, please share!</p><h4>2. Create a new module structure</h4><p>This <a href="https://python-packaging.readthedocs.io/en/latest/minimal.html">documentation</a> on Python packages describes the required package structure. Even as we automated this step, it goes without saying the documentation is still essential reading material — useful for debugging.</p><p>Let’s go to the one-liners.</p><pre># get a copy of PyShipper and change into it<br>git clone <a href="https://github.com/LINKIT-Group/pyshipper.git">https://github.com/LINKIT-Group/pyshipper.git</a><br>cd pyshipper</pre><pre># fork (copy/ paste) the contents to a new directory<br>make fork dest=~/${YOUR_NEW_MODULE_NAME}</pre><p>This forks a stripped version of PyShipper to a new directory called ~/${YOUR_NEW_MODULE_NAME}. Files like LICENSE and README.md are not copied. After all, you own the new module, and thus need to add a license and documentation to it — no license strings attached.</p><p>For the purpose of review, this is the Makefile fork target:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0d00137987cf8ee197baf73b40fe17a/href">https://medium.com/media/f0d00137987cf8ee197baf73b40fe17a/href</a></iframe><p>Safety note: files are only copied if there is either no directory or an empty directory.</p><h4>3. Configure variables</h4><p>Our next step is to change to the new directory and edit the /variables file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f0d00137987cf8ee197baf73b40fe17a/href">https://medium.com/media/f0d00137987cf8ee197baf73b40fe17a/href</a></iframe><pre># switch to the new module<br>cd ~/${YOUR_NEW_MODULE_NAME}</pre><pre># edit the /variables file<br>{replace_with_your_editor} variables</pre><p>The NAME variable in /variables is picked up by Makefile, and all variables are exported to the environment of a Docker container, used by setup.py during container execution.</p><p>The Makefile line took me a few hours to work, stripping of strong-quotes properly is nearly always painful. The problem became easy when I stumbled on <a href="https://www.gnu.org/software/gawk/manual/html_node/Quoting.html">this piece of documentation</a> — number 47 it is, for GNU AWK at least.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66dff58a08d9f30ff1aa21f1979dea41/href">https://medium.com/media/66dff58a08d9f30ff1aa21f1979dea41/href</a></iframe><p>While the single Makefile line was hard work, getting the variables passed to setup.py was much easier. I only needed a way to pass all variables to the container. A short ~/.profile file in the container space does that job — note, for this to work, the /variables file must also be present via a link or mount.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74b098dbd2a60dda00349f46d7dc0205/href">https://medium.com/media/74b098dbd2a60dda00349f46d7dc0205/href</a></iframe><p>To finalize this section, this is the /setup.py file that loads the variables in the container environment. The small lambda ensures an empty string is inserted if a variable is not present.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53eace1e318779942d5af0327574b963/href">https://medium.com/media/53eace1e318779942d5af0327574b963/href</a></iframe><h4>4. Run and edit code</h4><p>PyShipper ships with a /module directory containing boilerplate code for a module. There is also a coding pattern, with a minimal “hello-world”-like function, that can be called both CLI- and import-style.</p><p>Of course all Python3 — as Python2 goes <a href="https://www.ncsc.gov.uk/blog-post/time-to-shed-python-2">EOL Januari 2020</a>!</p><pre># enter the container runtime<br>make shell</pre><pre># test run the module in CLI<br>python3 -m module --name &quot;PyShipper&quot;</pre><pre># start Python3, import the module and test<br>python3</pre><pre>&gt;&gt;&gt; import module<br>&gt;&gt;&gt; module.main(name=&quot;PyShipper&quot;)</pre><p>This runs the code under /module. Instead of using the name “module”, you can also replace it with the name of your own module. A symlink is created to make both work — in the shipped version, only your own name can be used to reference the module.</p><p>If you are a Python developer, I am sure you need to know what do next. All module code to edit is in the /module — have fun tinkering!</p><h4>5. Make module</h4><p>When you have a minimal working version it is time to build the package. One thing you may want to do first, is to check the VERSION in /variables and ensure it is updated — I just made a note to myself to automate that in a next version, who likes keeping track of versions? — right ;).</p><pre># build the module -- this runs setup.py in the container<br>make module</pre><pre># better practice version of the former<br># includes pylint code quality testing<br>make pylint module</pre><p>This is the Makefile configuration connected to the one-liners above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07938629a36f33768ddc7c630c68a71f/href">https://medium.com/media/07938629a36f33768ddc7c630c68a71f/href</a></iframe><h4>6. Publish</h4><p>The output of the build process is a gzipped tar archive, present in the /dist directory. By uploading this file to <a href="https://pypi.org/">PyPi</a> —Python Package Index — the module is published, and install-able through Python pip by everyone.</p><pre># upload the module<br>make upload</pre><p>The command above prompts for a username and password. You need to insert your <a href="https://pypi.org/">PyPi</a> credentials—see Prerequisites at the beginning of this chapter.</p><p>The Makefile snippet for the upload target is:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae91963cfae170078414882f8e21a031/href">https://medium.com/media/ae91963cfae170078414882f8e21a031/href</a></iframe><p>After you created the module, published it on <a href="https://pypi.org/">PyPi</a>, you can install it on any capable system and use it like any other Python3 pip module.</p><pre># install the module on system<br>sudo python3 -m pip install ${YOUR_NEW_MODULE_NAME}</pre><h3>Final thoughts</h3><p>I already use this automation myself and I am quite happy with it. I regularly create small Python modules for specific tasks; it’s incredibly easy to import them in containers or server-less environments afterwards.</p><p>Now with this piece of automation more repetitive work is cut — that means more time to spend on other innovations.</p><p>I hope you find this equally useful. I’d be delighted if this helps, at least to some of you, to contribute to the Python eco-system.</p><p>Happy Python module writing!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31dd6d9d5c8f" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/create-build-and-ship-a-python3-pip-module-in-5-minutes-31dd6d9d5c8f">Create, build and ship a Python3 pip module in 5 minutes</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On automating everything — Cloud Container Provisioning on AWS]]></title>
            <link>https://itnext.io/on-automating-everything-cloud-container-provisioning-on-aws-ae88db5efad3?source=rss-99c3ef88ca96------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae88db5efad3</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Anthony Potappel]]></dc:creator>
            <pubDate>Wed, 11 Sep 2019 06:54:39 GMT</pubDate>
            <atom:updated>2023-12-24T13:33:57.315Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kmXc3rshix_M48_d" /><figcaption>Photo by <a href="https://unsplash.com/@danielkcheung?utm_source=medium&amp;utm_medium=referral">Daniel Cheung</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>On automating everything — Cloud Container Provisioning on AWS</h3><p>Let’s talk about applying continuous deployment and integration (CI/CD) to put some services online — the GitOps way. In a nutshell, we like to commit new features to Git, and have these features pushed to users immediately via micro-services. The command-line for Developers looks like this:</p><pre>git commit -am &quot;add new feature X for product Y&quot; \<br>  &amp;&amp; git push</pre><p>After typing the line above, a pipeline starts, things get tested and (re-)deployed (<a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">blue-green</a> style), after which end-users consume the updated service. Version rollbacks simply work by reverting to a previous version in Git, the service(s) change back accordingly.</p><p>This article is mostly about putting the automation parts in place before applying the infra-as-code itself — serving this never-ending itch of finding a faster way to deploy platforms. At the end we create a platform that converts Dockerfiles to running services on AWS Fargate.</p><h3>Low Maintenance Ops</h3><p>Modern day infra work involves serving and managing countless platforms, instant delivery on request. That means — yet more — automation.</p><p>We start by complementing this <a href="https://itnext.io/docker-makefile-x-ops-sharing-infra-as-code-parts-ea6fa0d22946">Docker &amp; Makefile</a> part to push infra-as-code to public cloud platforms. These are the one-liners that should eventually do the job:</p><pre># Make-Git: create a build repository<br>make git url=${GIT_URL_NEW_PLATFORM}</pre><pre># Make-Infra: deploy the infrastructure<br>make infra</pre><p>Excluding commentary, that is two lines of code. The first — Make-Git — is there to pull code parts in a new deployment object. The second — Make-Infra — handles the deployment itself.</p><p>Note that all code is available on Github. Links to the repositories are provided in the final section of this article (Dockerfile to Fargate).</p><h4>Make-Git</h4><p>Make-Git facilitates re-using existing code pieces in a new project. The tool is a wrapper around GIT -subtrees and -submodules. We maintain individual code parts in one spot, and pull in when needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*TFTvyuuZI0Zp0QmU4DfZ1A.png" /><figcaption>Figure 1: Make-Git</figcaption></figure><p>Make-Git (Figure 1) pulls in existing — re-usable — parts into a new platform, based on a configuration in .gitmodules. In our example we create two re-usable parts, Elastic Container Service (ECS) and Code Pipeline.</p><p>Make-Git is a Python module, installed through <a href="https://pypi.org/">PyPi</a>:</p><pre>python3 -m pip install makegit</pre><p>There is no need to copy this line. The module is added by Dockerfile, demonstrated in the final chapter of this article. Entry in Makefile:</p><pre>git:<br>     $(RUN_DOCK) &quot;GIT_URL=\&quot;$(URL)\&quot; makegit&quot;</pre><p>That git target triggers this ~/.aliases function in container runtime:</p><pre># compressed for readability purposes<br>function __makegit(){<br>    ...<br>    python3 -m makegit --url &quot;${GIT_URL}&quot;<br>    ...<br>    return $return_code<br>}<br>alias makegit=&#39;__makegit&#39;</pre><p>The setup allows for running the Python3 module inside the container, from where it collects the required GIT repositories.</p><p>By using the ~/.aliases file we can add code before and after, and even use special characters — quotes, pipes and redirection — that are notoriously difficult to escape in Makefile itself.</p><h4>Deploy infra by Commit — VIM users bonus</h4><p>I grew accustomed to commit &amp; push code continuously — every few to 10 minutes — to get immediate feedback on tiny increments. If you are a VIM user, you may like this ~/.vimrc snippet:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f95d103dcd790f7c2114b6f7172645ac/href">https://medium.com/media/f95d103dcd790f7c2114b6f7172645ac/href</a></iframe><p>Type <em>:P</em> — sorry, no pun intended — to commit and push code, or <em>:W</em> to only commit. This is for VIM, but every <a href="https://en.wikipedia.org/wiki/Integrated_development_environment">IDE</a> should be capable of running a similar function — feel free to copy from the ~/.vimrc snippet selectively.</p><h4>Make-Infra</h4><p>The infra-as-code pieces are written in Terraform as I had the code readily available — it required limited contribution on my part. Good arguments to use Terraform are:</p><ul><li><strong>Integrates:</strong> one code base to connect to all relevant cloud-providers.</li><li><strong>Documented:</strong> available blueprints help to kick-start a new project.</li><li><strong>Configurable </strong><a href="https://www.terraform.io/docs/state/remote.html"><strong>Remote State</strong></a><strong>:</strong> this helps working on a shared project.</li></ul><p>Disclaimer. If you use AWS only, CloudFormation is arguably a better choice — its automatic rollbacks and deep AWS integration are golden. I will share the CloudFormation version in a next article, in this article I stick with Terraform.</p><p>In a production setting, Terraform requires setting up and retrieving a state first, before running the actual deployment code. Thanks to some Python automation and Makefile magic, we compressed it all into one call.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*aV6UCub6E7Vk_Q-i9z-myg.png" /><figcaption>Figure 2: Make-Infra</figcaption></figure><p>Make-Infra schematics in Figure 2. Via Makefile the Python module RemoteState is called to setup Terraform state material (1), followed by plain Terraform build commands to deploy the infra (2).</p><h4>On Terraform RemoteState</h4><p>In an enterprise environment, Terraform is typically used by one or more teams, to ensure safe operations <a href="https://medium.com/@jessgreb01/how-to-terraform-locking-state-in-s3-2dc9a5665cb6">Remote State &amp; Locking</a> is applied.</p><ul><li><strong>Remote State</strong>. Infra details, such as service IDs and endpoints, are written in a <a href="https://www.terraform.io/docs/state/">state-file</a> by Terraform. Storing this file in a remote location, provides a single source of truth on deployed infrastructure — stored in S3.</li><li><strong>Locking</strong>. Only one update process can run at a time. We enforce this by setting a lock entry in a database — stored in DynamoDB.</li></ul><p>One common pain, is that an incorrect backend file can easily cause a faulty state, which is painful to troubleshoot and recover from. To solve that problem, the RemoteState module creates this file automatically.</p><h4>On RemoteState — Module</h4><p>The RemoteState module creates:</p><ul><li>DynamoDB and S3 parts on AWS, if the parts do not yet exist.</li><li>A backend file for Terraform to use — terraform/backend-auto.tf</li></ul><p>The module derives information from the current GIT repository — pulled by Make-Git. AWS keys and region information are loaded from the default ~/.aws/credentials and ~/.aws/config files (<a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html">see AWS CLI instructions</a>).</p><p>This installs the RemoteState module:</p><pre>python3 -m pip install remotestate</pre><p>Similar to Make-Git, this one-liner is included in a Dockerfile. Entry in Makefile:</p><pre>remote:<br>        $(RUN_DOCK) &quot;remotestate&quot;</pre><p>The wrapper function in ~/.aliases:</p><pre># compressed for readability purposes<br>function __remotestate(){<br>    ...<br>    cd &quot;${PROJECT_PATH}&quot; &amp;&amp; \<br>        python3 -m remotestate --git &quot;build/buildrepo&quot;<br>    ...<br>    return $return_code<br>}<br>alias remotestate=&#39;__remotestate&#39;</pre><p>RemoteState expects a GIT repository in the “build/buildrepo” directory, created by Make-Git in earlier chapter, to retrieve the repository URL defined in .git/config. From the URL, RemoteState derives the names for the DynamoDB tables and S3-Bucket automatically, in the next section we explain how this works</p><h4>RemoteState: Naming Scheme</h4><p>RemoteState takes care of naming DynamoDB tables and S3 Bucket, based on the input of a GIT repository URL. Assume the following URL:</p><pre>GIT_URL = <a href="https://github.com/LINKIT-Group/aws-cicd-demo">https://github.com/LINKIT-Group/aws-cicd-demo</a></pre><p>This can be decomposed in to the following parts:</p><pre># composed from ${GIT_URL}<br>GIT_HOST = github.com<br>GIT_ORGANISATION = LINKIT-Group<br>GIT_REPO_NAME = aws-cicd-demo<br>GIT_URI = ${GIT_HOST}/${GIT_ORGANISATION}/${GIT_REPO_NAME}</pre><pre># defaults -- optionally retrieved from ${GIT_URL}<br>GIT_BRANCH = master</pre><p>This information is used to create names for DynamoDB and S3. All characters are lower-cased:</p><pre># note that one set of DynamoDB tables is created to hold all repositories within one group (GIT_ORGANISATION).</pre><pre># DynamoDB Table base name -- not a Table itself<br>DYNAMO_TABLE_NAME = terraform.${GIT_HOST}.${GIT_ORGANISATION}</pre><pre># table for locking<br>DYNAMO_TABLE_NAME_LOCK = ${DYNAMO_TABLE_NAME}.lock</pre><pre># table to store key=${GIT_URI}, value=${S3_BUCKET_NAME} -pairs<br>DYNAMO_TABLE_NAME_LOCK = ${DYNAMO_TABLE_NAME}.s3</pre><pre># Python Code example<br>import time<br>def time_string():<br>   &quot;&quot;&quot;Return current time in micro-second (usec) as a string&quot;&quot;&quot;<br>   return str(round(time.time()*10**6))</pre><pre>CREATION_TIME_MICROSEC = time_string()<br># Bucketname must be globally unique--why we add the microseconds.</pre><pre>S3_BUCKET_NAME = ${DYNAMO_TABLE_NAME}-${CREATION_TIME_MICROSEC}</pre><h4>Troubleshoot Terraform</h4><p>To run Terraform, these are the lines in Makefile:</p><pre>infra:<br>        $(RUN_DOCK) &quot;remotestate&quot;<br>        $(RUN_DOCK) &quot;terraform init \<br>        &amp;&amp; terraform plan \<br>        &amp;&amp; terraform apply -auto-approve \<br>        &amp;&amp; terraform output&quot;<br>destroy:<br>        $(RUN_DOCK) &quot;terraform destroy -auto-approve&quot;</pre><p>That is nice, but how about all the other Terraform commands to manage or troubleshoot deployed infra? Enter the container as follows:</p><pre>make shell</pre><p>After entering the container, you can directly type individual Terraform commands to troubleshoot and fix the issue.</p><h4>Low Maintenance Ops: Wrap-up</h4><p>A team member can now pull the GIT repository by applying Make-Git, and create an infrastructure using Make-Infra.</p><p>The main advantage in both of these processes is that team members do not have to deal with internal configuration, speeding up delivery while reducing risk. Needless to say, all is automated.</p><h3>Dockerfile to Fargate</h3><p>The previous sections were about creating and deploying new platforms efficiently. In this part it’s time to deliver the promised CI/CD platform.</p><p><strong>Additional Prerequisites</strong></p><ul><li>An AWS account — this can be acquired for <a href="https://aws.amazon.com/free">free</a>.</li><li>Docker. Install instructions: <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce">Ubuntu</a> or <a href="https://docs.docker.com/docker-for-mac/install/">Mac</a>.</li><li>Ability to run “<a href="https://www.tutorialspoint.com/unix_commands/make.htm">make</a>” on your host system. Most Unix, Linux and Mac systems have this by default — on Windows I’d recommend an Ubuntu VM.</li></ul><p>I recommend using a clean AWS account or a dedicated test region to play in. I am also assuming you have some experience using AWS and understand what we are building.</p><h4>A build pipeline on AWS for server-less containers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*dGHuN7_ljlFbm_qprFYIrw.png" /><figcaption>Figure 3: CI/CD + ECS in AWS</figcaption></figure><p>Assuming your system is ready — see list of prerequisites above — the following lines deploy this platform:</p><pre># retrieve deploytools, this repository holds MakeGit and MakeInfra<br>git clone <a href="https://github.com/LINKIT-Group/deploytools.git">https://github.com/LINKIT-Group/deploytools</a><br>cd deploytools</pre><pre># collect the code to deploy our demo<br># code is put in the build/ directory<br>make git url=https://github.com/linkit-group/aws-cicd-demo</pre><pre># deploy the demo<br># note: ensure ~/.aws is setup properly!<br>make infra</pre><p>Building the infrastructure takes about 5 minutes — just enough time to grab a cup of coffee, tea or a snack. If all goes well, you should see this Terraform output on your console:</p><pre># Apply complete! Resources: 47 added, 0 changed, 0 destroyed.</pre><p>Do note the AWS bill is now running. Approximately a few dollars a day.</p><p>Now it’s time to access the app. Login to AWS console and go to EC2 -&gt; Loadbalancers, here you will find an application loadbalancer: “app-demo-backend-test-alb”. Click on it, open the “Description” tab, copy the value under “DNS name”, and paste it in a browser window.</p><p>You should get this error message back from the application loadbalancer:</p><pre>503 Service Temporarily Unavailable</pre><p>You get this error message because there is no application yet. We only created the build platform, waiting for us to push application code.</p><h4>Pushing application code</h4><p>Time to switch hats with a developer. Let’s install an Nginx container and say hello to the world. We clone <a href="https://github.com/LINKIT-Group/nginx-docker-helloworld.git">this repository</a>, copy/ paste the contents to the CodeCommit repository, and deploy via commit and push.</p><p>Retrieve the ${CODECOMMIT_URL} by browsing to AWS console -&gt; CodeCommit. There should be a repository named “app-${random-chars}”, click it and go to “Clone URL”. Choose “Clone SSH” and copy the string.</p><p>If this is your first time to clone a CodeCommit repository, you get an access denied error. First, setup an SSH key following these <a href="https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-ssh-unixes.html?icmpid=docs_acc_console_connect_np">instructions</a> (Step 3).</p><pre># clone CodeCommit repository -- first time its empty<br>git clone ${CODECOMMIT_URL}</pre><pre># clone nginx demo code<br>git clone <a href="https://github.com/LINKIT-Group/nginx-docker-helloworld.git">https://github.com/LINKIT-Group/nginx-docker-helloworld</a></pre><pre># copy/ paste to CodeCommit repo<br>cp nginx-docker-helloworld/* ${CODECOMMIT_REPO}/</pre><pre># commit and push -- this will trigger a container deployment<br>cd ${CODECOMMIT_REPO}<br>git add *<br>git commit -am &quot;add feature X for product Y&quot;<br>git push</pre><p>Now it’s time to sit back and relax. You can view the process by browsing to AWS console -&gt; CodePipeLine, and refresh the URL pointing to the container.</p><p>If you get the “Hello world” message, it works as designed and we are done.</p><pre># delete the platform<br>make destroy</pre><pre># note this will not clean up the S3 bucket and DynamoDB table used by Terraform backend. These must be deleted separately if no longer needed.</pre><h3>The End of the Beginning</h3><p>I hope to have proven that you can stop using EC2-instances for simple services. Did you notice ECS-Fargate does not insert EC2 either? — these are server-less containers.</p><p>Next step is to see how far we can scale this up, and start saving time on managing big container clusters — no more clunky EC2-instances!</p><p>Another work-item for me is to replace all the Terraform code with pure CloudFormation and switch to Lambda-only for the automation glue.</p><p>Hope to share either (or both) in a future article.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae88db5efad3" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/on-automating-everything-cloud-container-provisioning-on-aws-ae88db5efad3">On automating everything — Cloud Container Provisioning on AWS</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker & Makefile | X-Ops — sharing infra-as-code parts]]></title>
            <link>https://itnext.io/docker-makefile-x-ops-sharing-infra-as-code-parts-ea6fa0d22946?source=rss-99c3ef88ca96------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea6fa0d22946</guid>
            <category><![CDATA[makefile]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Anthony Potappel]]></dc:creator>
            <pubDate>Tue, 19 Feb 2019 15:20:27 GMT</pubDate>
            <atom:updated>2023-12-24T13:31:05.352Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rYWn26XjeK8bqumw" /><figcaption>Photo by <a href="https://unsplash.com/@zhenhu2424?utm_source=medium&amp;utm_medium=referral">Zhen Hu</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Docker &amp; Makefile | X-Ops — sharing infra-as-code parts</h3><p>In our life as X-Ops — i.e. roles like DevOps, CloudOps, GitOps, SysOps — we focus on building infra-as-code. We frequently use Docker to build and test, commit and pull the code to/from repositories, and when all tests pass, infra moves in production — completely automated.</p><p>A big opportunity now is scaling and integrating our X-Ops teams. Two goals on that shortlist are; 1. low- (preferably zero-) barriers to start, helping to get more colleagues come along, and 2. easy-to-follow build and test habits, getting parts into production faster and improve re-use.</p><p>In this article I focus on a few easy technical practices — namely applying Docker, Makefile and getting UIDs right — that, in my experience, put us much closer to those goals.</p><h3>Using a Makefile</h3><p>Docker brings us the automation to build systems fast and effectively, and we can even boost to running services with docker-compose.</p><p>However, in the majority of — in-production — cases there is a separate container platform. The container we build, test and run locally, is part of a larger context. One issue that often pops up, is that when you share docker code parts within a team, or the world, you will also need to explain how your container requires to be run.</p><p>Personally, I want all my projects to have something like this working:</p><pre>git pull &amp;&amp; make test &amp;&amp; make build &amp;&amp; make deploy</pre><p>It is extremely powerful to toss around a one-liner in this format. You can easily ask a colleague to take your product (-component) for a spin, without having him/ her plough through a big README. It either works or does not.</p><p>Enter the good old Makefile. A 40-year old classic, still too young to retire, is now becoming reborn within automation projects (nice short read on that: <a href="https://localheinz.com/blog/2018/01/24/makefile-for-lazy-developers/">makefile-for-lazy-developers</a>).</p><p>This is my version of a <strong>Makefile</strong> that plays nice with Docker:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e05832a9991179421461ec91904791f/href">https://medium.com/media/3e05832a9991179421461ec91904791f/href</a></iframe><p>If you are not familiar with Makefile, this may look intimidating. Before you start, you will need Docker. Docker install instructions are here: <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce">Ubuntu</a>, <a href="https://docs.docker.com/docker-for-mac/install/">Mac</a> or <a href="https://docs.docker.com/docker-for-windows/install/">Windows</a>. I am also assuming, you also know how to open a shell, install a git client and can run make. A Linux distribution is recommended, but I have also tested it on Mac. If you are set, here are some commands to try:</p><pre># download our files<br>git clone <a href="https://github.com/LINKIT-Group/dockerbuild">https://github.com/LINKIT-Group/dockerbuild</a></pre><pre># enter directory<br>cd dockerbuild</pre><pre># build, test and run a command<br>make build test shell cmd=&quot;whoami&quot;</pre><pre># my favorite for container exploration<br>make shell</pre><pre># shell-target is the default (first item), so this also works: <br>make cmd=&quot;whoami&quot;<br>make cmd=&quot;ls /&quot;</pre><pre># force a rebuild, test and cleanup<br>make rebuild test clean</pre><p>Notice I not used “&amp;&amp;”-chars anywhere? You can use a single make-command with a list of targets (e.g. build, test, clean), make stops once an error is hit.</p><p>In Makefile, You can customise the one-liners to run at a specific target. Personally, I always like to add “ — rm” to a run-statement, to prevent ending up with a trail of abandoned containers. If you read the Makefile carefully, you will see more interesting things to try-out.</p><p>You should want to customise the test-target, and make testing become standard practice. When your build part gets integrated in CI/CD pipelines (eventually, it will), it is incredibly helpful for “make test” to exist.</p><p>From here, it is only one more step to add a deploy- target, pushing the container to Docker-hub or a similar repository. Continuous deployment is where things get really exciting. Unfortunately, this also requires much more writing space, which is why I need to save this for a follow-up article (or two).</p><p>Before we move in the direction of deployments, we need to cover one more basic configuration practice. Noticed the HOST_USER and HOST_UID variables in Makefile? These are filled at runtime, and re-exported for docker-compose to read. Which leads me to the next chapter on UIDs.</p><h3><strong>Dealing with UIDs</strong></h3><p>Docker defaults a container to user root when a UserID (UID) is not configured. When you get to deal with production systems, it is required practice to configure users correctly. You can read about it on <a href="https://docs.docker.com/develop/develop-images/dockerfile_best-practices/">Dockers’ best-practices-list</a>, and <a href="https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/#8-run-containers-as-a-non-root-user">this piece from K8s</a> covers it nicely just as well.</p><p>When you run tests locally on Mac, root is mapped to your own user so all works fine. Production platforms — should — be configured for user-isolation, but that is not always the default. If for example, you run your tests on a Linux system (without re-mapping), you will likely stumble upon the issue that files generated by Docker are <a href="https://medium.com/redbubble/running-a-docker-container-as-a-non-root-user-7d2e00f8ee15">owned by root</a>.</p><p>You will also run into issues when you have multiple users on one system. Even when you do not share a system, running parallel tests requires the separation just as well. And, how would you even pass credentials (e.g. by linking volumes as ~/.ssh and ~/.aws) without a correct UID setup? The latter is a common pattern when dealing with infra deployments.</p><p>If you start becoming a more intensive docker-consumer, your team grows and things get complex, eventually you will either want or need to embed UID separation in all of your projects.</p><p>Fortunately, configuring UID early-on is (relatively) easy. While it took me some practice to get a good setup, I now have a template that (assuming use of Makefile) does it all automatically, costing near zero effort.</p><p>Below is a copy of a <strong>docker-compose</strong> file to use:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc44973495bd7fa121b6907b9828c6d7/href">https://medium.com/media/cc44973495bd7fa121b6907b9828c6d7/href</a></iframe><p>Some magic is in these variable statements:</p><pre>${HOST_USER:-nodummy}<br>${HOST_UID:-4000}</pre><p>This copies variables from your runtime, and if not exist, default to “nodummy” and “4000” respectively. If you don’t like defaults, just do this:</p><pre>${HOST_USER:?You forgot to set HOST_USER in .env!}<br>${HOST_UID:?You forgot to set HOST_UID in .env!}</pre><p>Note on the “HOST_” prefix. I steer away from using USER and UID directly. These variables are not guaranteed to be available in a runtime. USER often is available in the shell, but UID is mostly an environment variable which Docker will not pick up. Having a separate naming scheme prevents accidents, and allows flexibility in configuring automation pipelines.</p><p><strong>Dockerfile</strong> looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a960c024051888caa5682967398c2862/href">https://medium.com/media/a960c024051888caa5682967398c2862/href</a></iframe><p>Here we build a small (its just 10MB, micro-services FTW!) Alpine container with bash added for fun and practice. We apply the concept of so-called <a href="https://docs.docker.com/develop/develop-images/multistage-build/">staged builds</a> to keep base-images (re-usable build components) separated from user-images (image prepared for a specific run).</p><p>All the variables are set automatically when using the Makefile as explained in previous chapter. This Dockerfile also works fine without a Makefile though, users can still configure variables in their own runtime or a separate <a href="https://docs.docker.com/compose/env-file/">env-file</a>. Personally, I just use Makefiles because I enjoy the zero-effort.</p><p>One last thing. If you are in developing mode, you will probably hit a certain obstacle requiring you to troubleshoot as user root. Simply type:</p><pre>make shell user=root</pre><h3>Wrapping it up</h3><p>I hope you found this article useful. Full code is available at <a href="https://github.com/LINKIT-Group/dockerbuild">github</a>.</p><p>Now, when we build a new service, we can simply kickstart our project by copy/pasting these files — Makefile, Dockerfile and docker-compose.yml — , modify where needed, and have our new service up and running in no-time.</p><p>Using the Docker+Makefile process not just makes your parts portable, but also re-usable to your team (or the world), and allows for shipping with testing-rules attached. Welcome to the world of X-Ops!</p><p>Happy coding, and see you in next article!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea6fa0d22946" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/docker-makefile-x-ops-sharing-infra-as-code-parts-ea6fa0d22946">Docker &amp; Makefile | X-Ops — sharing infra-as-code parts</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>