<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samhita Alla on Medium]]></title>
        <description><![CDATA[Stories by Samhita Alla on Medium]]></description>
        <link>https://medium.com/@allasamhita?source=rss-7e65fabdea53------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*v7yefO_Il0rz4Hoq6b1CmA.jpeg</url>
            <title>Stories by Samhita Alla on Medium</title>
            <link>https://medium.com/@allasamhita?source=rss-7e65fabdea53------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:23:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@allasamhita/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fine-Tuning Insights: Lessons from Experimenting with RedPajama Large Language Model on Flyte Slack…]]></title>
            <link>https://allasamhita.medium.com/fine-tuning-insights-lessons-from-experimenting-with-redpajama-large-language-model-on-flyte-slack-3c64e1fad0b9?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c64e1fad0b9</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[slack]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Tue, 04 Jul 2023 04:15:33 GMT</pubDate>
            <atom:updated>2023-07-04T04:15:33.503Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Fine-Tuning Insights: Lessons from Experimenting with RedPajama Large Language Model on Flyte Slack Data</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qL3SgC-TfhFX6mMQ0Az1DA.jpeg" /></figure><p>Large language models (LLMs) have taken the world by storm, revolutionizing our understanding and generation of human-like text. These models have showcased remarkable capabilities across a range of tasks, including question-answering, chatbots and even creative writing. Naturally, like many others, I was filled with excitement to explore and experience the potential of these models firsthand.</p><p>As an open-source contributor at <a href="https://union.ai/">Union</a>, my aim was to enhance users’ ability to find solutions to their queries independently on Slack.</p><p>Building upon this goal, I had the <a href="https://flyte.org/community#slack">Flyte Slack</a> data on hand, and I set out to <a href="https://www.union.ai/blog-post/fine-tuning-vs-prompt-tuning-large-language-models#fine-tuning-updating-model-parameters">fine-tune</a> an LLM using that data. My ultimate objective was to develop a Slack bot that could run within the Flyte Slack workspace.</p><p>Instead of relying on embeddings, <a href="https://flyte.org/blog/building-flytegpt-on-flyte-with-langchain">which have already been extensively explored</a>, I opted for (supervised) fine-tuning to examine its performance and potential benefits. This decision was driven by my desire not only to incorporate knowledge into the model but also allow it to learn the subtleties and nuances of Slack messages. Fine-tuning appeared to be the better option for achieving this objective.</p><p><em>While fine-tuning itself proved to be a straightforward process, I was surprised to discover that the surrounding factors and implications turned out to be much more complex than I had anticipated …</em></p><h3><strong>Slack scraper</strong></h3><p><em>Goal: Create a scraper that extracts data from Slack channels and saves it on HuggingFace. The extracted data will be used later for fine-tuning an LLM.</em></p><p>Data is vital for fine-tuning LLMs — or any other model, for that matter. Therefore, the first crucial step is to extract Slack data and generate a dataset from it, finding a way to store it appropriately.</p><p>I exported all the Slack data into a folder, which consists of directories that align with each Slack channel. Within each channel directory, there are neatly arranged JSON files, in which each file represents the data for a specific day.</p><pre>[<br>   {<br>    &quot;client_msg_id&quot;: &quot;530038bc-718a-4708-9d05-0e2811795ffe&quot;,<br>    &quot;type&quot;: &quot;message&quot;,<br>    &quot;text&quot;: &quot;Are you trying to deploy Flyte on EKS?&quot;,<br>    &quot;user&quot;: &quot;U01J90KBSU9&quot;,<br>    &quot;ts&quot;: &quot;1684221986.531759&quot;,<br>    &quot;blocks&quot;: [<br>        {<br>            &quot;type&quot;: &quot;rich_text&quot;,<br>            &quot;block_id&quot;: &quot;79X&quot;,<br>            &quot;elements&quot;: [<br>                {<br>                    &quot;type&quot;: &quot;rich_text_section&quot;,<br>                    &quot;elements&quot;: [<br>                        {<br>                            &quot;type&quot;: &quot;text&quot;,<br>                            &quot;text&quot;: &quot;Are you trying to deploy Flyte on EKS?&quot;<br>                        }<br>                    ]<br>                }<br>            ]<br>        }<br>    ],<br>    &quot;team&quot;: &quot;TN89P6GGK&quot;,<br>    &quot;user_team&quot;: &quot;TN89P6GGK&quot;,<br>    &quot;source_team&quot;: &quot;TN89P6GGK&quot;,<br>    &quot;user_profile&quot;: {<br>        &quot;avatar_hash&quot;: &quot;4f24b225c547&quot;,<br>        &quot;image_72&quot;: &quot;https:\/\/avatars.slack-edge.com\/2023-04-28\/5184926466722_4f24b225c547392670bd_72.jpg&quot;,<br>        &quot;first_name&quot;: &quot;Samhita&quot;,<br>        &quot;real_name&quot;: &quot;Samhita Alla&quot;,<br>        &quot;display_name&quot;: &quot;Samhita Alla&quot;,<br>        &quot;team&quot;: &quot;TN89P6GGK&quot;,<br>        &quot;name&quot;: &quot;aallasamhita&quot;,<br>        &quot;is_restricted&quot;: false,<br>        &quot;is_ultra_restricted&quot;: false<br>    },<br>    &quot;thread_ts&quot;: &quot;1684211605.360509&quot;,<br>    &quot;parent_user_id&quot;: &quot;U057SAH5KA7&quot;<br>   },<br>   ...<br>]</pre><p>I proceeded to write a <a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html">Flyte task</a> that retrieves the names of the directories, or channels, within the parent directory.</p><pre>@task(cache=True, cache_version=&quot;2.0&quot;)<br>def get_channel_dirs(parent_dir: str) -&gt; List[str]:<br>    return glob(parent_dir + &quot;/*/&quot;)</pre><p>Next, I developed a task that extracts question-response pairs from the Slack data, involving the organization of messages into coherent threads. Within each Slack thread, I extracted question-response pairs, ensuring that a question corresponds to one user and the response corresponds to a different user. In cases where the same user posted multiple consecutive messages, I combined them into a single question or response.</p><p>This approach allowed me to retrieve multiple question-response pairs from each Slack thread, which made more sense to me than considering the first message to be the question and the remaining messages to be the response. Considering the possibility of multiple users contributing to a single Slack thread, it was important to capture the various contexts within the conversation.</p><p>Finally, the resulting question-response pairs were stored in a JSON file.</p><pre>@task(cache=True, cache_version=&quot;2.0&quot;)<br>def question_response_pairs(channel_dir: str) -&gt; Optional[FlyteFile]:<br>    threads = []<br>    thread_ts_list_index_pairs = {}<br>    sorted_list_of_files = sorted(glob(channel_dir + &quot;*.json&quot;), key=os.path.getctime)<br><br>    ...<br>    pairs = []<br>    for ts in threads:<br>        ...<br>        if input_messages and output_messages:<br>            pairs.append(<br>                {<br>                    &quot;input&quot;: list(input_messages.values())[0],<br>                    &quot;output&quot;: list(output_messages.values())[0],<br>                }<br>            )<br>    if pairs:<br>        json_file_name = os.path.join(<br>            flytekit.current_context().working_directory,<br>            f&quot;flyte_slack_data_{Path(channel_dir).parts[1]}.json&quot;,<br>        )<br>        with open(json_file_name, &quot;w&quot;) as f:<br>            json.dump(pairs, f)<br>        return FlyteFile(json_file_name)<br>    return None</pre><p><a href="https://github.com/unionai-oss/llm-fine-tuning/blob/1bc0187bb9eaf4d31106f1a084e51537a0d55f93/redpajama-lora/workflows/scrape_slack.py#L32-L127">This task</a> is applied to all the channels, and the resulting question-response pairs are stored in individual JSON files. Next these files are merged together to create a comprehensive dataset, which is then <a href="https://huggingface.co/datasets/unionai/flyte-slack-data">saved on HuggingFace</a> for further use and analysis.</p><pre>def merge_json_files(json_files):<br>    json_result_file = os.path.join(<br>        flytekit.current_context().working_directory, &quot;flyte_slack_data.json&quot;<br>    )<br>    result = []<br>    for json_file in json_files:<br>        if json_file:<br>            with open(json_file, &quot;r&quot;) as infile:<br>                result.extend(json.load(infile))<br>    with open(<br>        json_result_file,<br>        &quot;w&quot;,<br>    ) as f:<br>        json.dump(result, f)<br>    return json_result_file<br><br><br>@task(<br>    secret_requests=[<br>        Secret(<br>            group=SECRET_GROUP, key=SECRET_NAME, mount_requirement=Secret.MountType.FILE<br>        )<br>    ],<br>)<br>def push_to_hub(json_files: List[Optional[FlyteFile]]):<br>    HF_TOKEN = flytekit.current_context().secrets.get(SECRET_GROUP, SECRET_NAME)<br>    dataset = load_dataset(&quot;json&quot;, data_files=merge_json_files(json_files))<br>    dataset.push_to_hub(&quot;unionai/flyte-slack-data&quot;, token=HF_TOKEN)</pre><p><strong>I have accumulated a substantial amount of data, totaling approximately 28.2k rows! You can analyze and use this dataset as needed.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cHFqvHF-YGNYJzcGCUBHCg.png" /><figcaption>The question-response pairs extracted are stored in <a href="https://huggingface.co/datasets/unionai/flyte-slack-data">HuggingFace Datasets</a>.</figcaption></figure><p>At first glance, the large size of this dataset instilled a sense of excitement, raising expectations for the LLM to deliver impressive performance. However, there are a few gotchas to keep in mind:</p><ul><li>Outputs as inputs: It’s crucial to note that the outputs from previous question-response pairs are treated as inputs for subsequent pairs. This chaining of information can impact the model’s understanding and generation of responses.</li><li>Terse responses: Responses on Slack can often be brief and lacking in detail. This brevity can pose a challenge for the LLM, as it may struggle to provide comprehensive or informative answers.</li><li>Inaccurate responses: Not every response in the dataset can be guaranteed to be an accurate answer to the corresponding question. Some responses may diverge from the intended answer or may not fully address the question’s intent.</li><li>Non-question inputs: It’s important to acknowledge that not all inputs labeled as questions within the dataset are necessarily genuine queries. Some labeled questions might be statements, comments, or incomplete phrases, which may impact the model’s performance.</li></ul><p>I proceeded with the fine-tuning of the LLM regardless.</p><h3>Fine-Tuning RedPajama LLM</h3><p><em>Goal: Fine-tune the RedPajama LLM using the collected dataset, refining its ability to provide responses to queries within the Flyte Slack platform.</em></p><p>I chose the <a href="https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat">RedPajama 7B chat model</a> for fine-tuning. As for the prompt, I decided to go with the following:</p><pre>def generate_prompt(input, output=&quot;&quot;):<br>    return f&quot;&quot;&quot;As an advanced chatbot, you enjoy assisting users on a community Slack platform. Write a detailed response that appropriately answers the query.<br><br>### Query:<br>{input}<br><br>### Response:<br>{output}&quot;&quot;&quot;</pre><p>To carry out the fine-tuning process, I utilized the <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/kfpytorch/pytorch_mnist.html#pytorch-elastic-training-torchrun">PyTorch elastic training integration</a>, running the code on a single node equipped with five T4 GPUs. This integration provided a seamless way to utilize any PyTorch elastic training runner and its associated integrations effortlessly on Union Cloud. It provided a convenient approach to train any LLM, allowing for a straightforward replacement of the torchrun --nproc-per-node=1 --nnodes=1 … script with a simple @task(task_config=Elastic(nnodes=1)) annotation.</p><pre>@task(<br>    task_config=Elastic(nnodes=1),<br>    requests=Resources(mem=&quot;100Gi&quot;, cpu=&quot;50&quot;, gpu=&quot;5&quot;, ephemeral_storage=&quot;100Gi&quot;),<br>    pod_template=PodTemplate(<br>        primary_container_name=&quot;llm-fine-tuning&quot;,<br>        pod_spec=V1PodSpec(<br>            containers=[<br>                V1Container(<br>                    name=&quot;llm-fine-tuning&quot;,<br>                    image=&quot;ghcr.io/samhita-alla/redpajama-finetune:0.0.8&quot;,<br>                    volume_mounts=[V1VolumeMount(mount_path=&quot;/dev/shm&quot;, name=&quot;dshm&quot;)],<br>                )<br>            ],<br>            volumes=[<br>                V1Volume(<br>                    name=&quot;dshm&quot;,<br>                    empty_dir=V1EmptyDirVolumeSource(<br>                     medium=&quot;Memory&quot;, size_limit=&quot;60Gi&quot;<br>                    ),<br>                )<br>            ],<br>        ),<br>    ),<br>    environment={<br>        &quot;TRANSFORMERS_CACHE&quot;: &quot;/tmp&quot;,<br>        &quot;CUDA_LAUNCH_BLOCKING&quot;: &quot;1&quot;,<br>    },<br>    secret_requests=[<br>        Secret(<br>            group=SECRET_GROUP, key=SECRET_NAME, mount_requirement=Secret.MountType.FILE<br>        )<br>    ],<br>)<br>def redpajama_finetune(config: TrainerConfig) -&gt; FlyteDirectory:<br>    ...</pre><p>Moving forward, the subsequent step entails loading the tokenizer and LLM model into memory to proceed with fine-tuning.</p><pre>tokenizer = AutoTokenizer.from_pretrained(config.base_model)<br><br>tokenizer.pad_token_id = 0  # unk. we want this to be different from the eos token<br>tokenizer.padding_side = &quot;left&quot;  # Allow batched inference<br><br>model = AutoModelForCausalLM.from_pretrained(<br>    config.base_model, # &quot;togethercomputer/RedPajama-INCITE-7B-Chat&quot;<br>    trust_remote_code=True,<br>    device_map=device_map, # &quot;auto&quot;<br>)</pre><p>By utilizing the device_map=”auto” parameter, I can leverage <a href="https://huggingface.co/docs/accelerate/usage_guides/big_modeling#loading-weights">Hugging Face Accelerate to automatically determine the optimal placement of each layer</a> in the model based on the available resources. This approach maximizes the utilization of GPU memory space by initially storing the model’s weights on the GPU(s). If additional space is required, the remaining weights are stored on the CPU. In cases where there is insufficient RAM, the excess weights are stored on the hard drive as memory-mapped tensors.</p><p>Running the 7B model in full precision requires a total of 28 GB (7 * 4) GPU RAM, considering that each single precision (float32) floating-point number occupies 4 bytes of memory. However, the T4 instance I’ll use for fine-tuning has only 16 GB of GPU memory available. To ensure that the model fits within the available memory, I set torch_dtype to torch.float16. By using half precision, the memory footprint is reduced to approximately 14 GB, allowing the model to fit within the available 16 GB of GPU memory on the T4 instance.</p><pre>model = AutoModelForCausalLM.from_pretrained(<br>    config.base_model,<br>    trust_remote_code=True,<br>    torch_dtype=torch.float16,<br>    device_map=device_map,<br>)</pre><p>To further minimize memory consumption, which may not be necessary in this particular case but can be beneficial for larger models, you can employ <a href="https://huggingface.co/docs/transformers/main_classes/quantization">8-bit or 4-bit quantization offered by the bitsandbytes library</a>. This approach allows for even more efficient utilization of memory resources. By utilizing the load_in_8bit functionality, I can convert the loaded model into a mixed 8-bit quantized model. This feature <a href="https://huggingface.co/blog/hf-bitsandbytes-integration">enables the model to be loaded and operated with reduced memory requirements while maintaining acceptable performance</a>.</p><pre>model = AutoModelForCausalLM.from_pretrained(<br>    config.base_model,<br>    trust_remote_code=True,<br>    load_in_8bit=True,<br>    torch_dtype=torch.float16,<br>    device_map=device_map,<br>)</pre><p>After loading the model in 8-bit precision, the resulting memory footprint is as follows:</p><pre>model.get_memory_footprint()<br># Output: 7406375040</pre><p>The model only occupies 7.4GB of memory!</p><p>Note: If the torch_dtype is not specified when loading the model in 8-bit, you will encounter the following error:</p><pre>Overriding torch_dtype=None with torch_dtype=torch.float16 due to requirements of bitsandbytes to enable model loading in mixed int8. Either pass torch_dtype=torch.float16 or don&#39;t pass this argument at all to remove this warning.</pre><p>You can also use <a href="https://huggingface.co/docs/peft/index">Parameter-Efficient Fine-Tuning (PEFT)</a> techniques to address the challenges posed by computational and storage requirements. PEFT focuses on fine-tuning only a small subset of additional model parameters, resulting in substantial reductions in both computational and storage costs. Remarkably, these techniques achieve performance levels that are comparable to full fine-tuning approaches.</p><p>I will be utilizing a PEFT method called <a href="https://arxiv.org/pdf/2106.09685.pdf">Low Rank Adapters (LoRA)</a> from the PEFT library. Instead of fine-tuning the entire model, LoRa enables fine-tuning of specific adapters, which are then appropriately loaded within the model.</p><p>Before training the int8 model using PEFT, there are some pre-processing steps that need to be performed. To help with this, I will incorporate a utility function called prepare_model_for_int8_training that performs the following tasks:</p><ol><li>It casts all the non int8 modules to full precision (fp32) to ensure stability during training.</li><li>It adds a forward_hook to the input embedding layer, enabling gradient computation of the input hidden states. This is important for accurate gradient calculations.</li><li>It enables gradient checkpointing, which optimizes memory usage during training by selectively storing and recomputing intermediate activations.</li></ol><pre># 8-bit training<br>model = prepare_model_for_int8_training(model)<br><br># LoRA<br>lora_config = LoraConfig(<br>    r=config.lora_r, # 8<br>    lora_alpha=config.lora_alpha, # 16<br>    target_modules=config.lora_target_modules, # [&quot;query_key_value&quot;]<br>    lora_dropout=config.lora_dropout, # 0.05<br>    bias=&quot;none&quot;,<br>    task_type=&quot;CAUSAL_LM&quot;,<br>)<br>model = get_peft_model(model, lora_config)</pre><p>The next step in the process is to define a Trainer that will handle the training loop.</p><pre>trainer = Trainer(<br>    model=model,<br>    train_dataset=train_data,<br>    eval_dataset=val_data,<br>    args=TrainingArguments(<br>        per_device_train_batch_size=config.micro_batch_size,<br>        gradient_accumulation_steps=gradient_accumulation_steps,<br>        warmup_steps=100,<br>        num_train_epochs=config.num_epochs,<br>        learning_rate=config.learning_rate,<br>        fp16=True,<br>        logging_steps=10,<br>        optim=&quot;adamw_torch&quot;,<br>        evaluation_strategy=&quot;steps&quot; if config.val_set_size &gt; 0 else &quot;no&quot;,<br>        save_strategy=&quot;steps&quot;,<br>        eval_steps=eval_steps if config.val_set_size &gt; 0 else None,<br>        save_steps=save_steps,<br>        output_dir=config.output_dir,<br>        save_total_limit=3,<br>        load_best_model_at_end=True if config.val_set_size &gt; 0 else False,<br>        ddp_find_unused_parameters=False if ddp else None,<br>        group_by_length=config.group_by_length,<br>        report_to=&quot;wandb&quot; if use_wandb else None,<br>        run_name=wandb_run_name if use_wandb else None,<br>    ),<br>    data_collator=DataCollatorForSeq2Seq(<br>        tokenizer, pad_to_multiple_of=8, return_tensors=&quot;pt&quot;, padding=True<br>    ),<br>    callbacks=[SavePeftModelCallback],<br>)</pre><p>I stored the resulting fine-tuned model LoRA weights on the Hugging Face models hub for easy accessibility and future use.</p><p>You can access the complete fine-tuning code in the <a href="https://github.com/unionai-oss/llm-fine-tuning/blob/main/redpajama-lora/workflows/llm_finetune.py">GitHub repository</a>. The code includes various libraries and techniques, such as PyTorch elastic training for native torchrun usage, 8-bit quantization, PEFT and LoRA. By integrating these techniques into a Flyte <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task.html">task</a>, I successfully conducted the fine-tuning process on T4 GPUs, leveraging the capabilities of Union Cloud as a unified platform.</p><h3><strong>Inference</strong></h3><p><em>Goal: Implement an inference pipeline using the fine-tuned RedPajama LLM to generate accurate predictions on new user queries.</em></p><p>During the inference phase, I retrieved the pre-trained model and instantiated a LoRA model using the pre-trained LoRA configuration and weights.</p><pre>@task(requests=Resources(gpu=&quot;1&quot;, mem=&quot;50Gi&quot;, cpu=&quot;10&quot;))<br>def generate_output(<br>    input: str,<br>    temperature: float,<br>    top_p: float,<br>    top_k: int,<br>    num_beams: int,<br>    max_new_tokens: int,<br>    load_8bit: bool,<br>    base_model: str,<br>    lora_weights: str,<br>) -&gt; str:<br>    base_model = base_model or os.environ.get(&quot;BASE_MODEL&quot;, &quot;&quot;)<br>    assert (<br>        base_model<br>    ), &quot;Please specify a --base_model, e.g. --base_model=&#39;huggyllama/llama-7b&#39;&quot;<br><br>    tokenizer = AutoTokenizer.from_pretrained(base_model)<br>    model = AutoModelForCausalLM.from_pretrained(<br>        base_model,<br>        load_in_8bit=load_8bit,<br>        torch_dtype=torch.float16,<br>        device_map=&quot;auto&quot;,<br>    )<br>    model = PeftModel.from_pretrained(<br>        model,<br>        lora_weights,<br>        torch_dtype=torch.float16,<br>    )<br><br>    # unwind broken decapoda-research config<br>    model.config.pad_token_id = tokenizer.pad_token_id = 0  # unk<br>    model.config.bos_token_id = 1<br>    model.config.eos_token_id = 2<br><br>    if not load_8bit:<br>        model.half()  # seems to fix bugs for some users.<br><br>    model.eval()<br>    if torch.__version__ &gt;= &quot;2&quot; and sys.platform != &quot;win32&quot;:<br>        model = torch.compile(model)<br><br>    prompt = generate_prompt(input)<br>    inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;)<br>    input_ids = inputs[&quot;input_ids&quot;].to(device)<br>    generation_config = GenerationConfig(<br>        temperature=temperature,<br>        top_p=top_p,<br>        top_k=top_k,<br>        num_beams=num_beams,<br>    )<br><br>    with torch.no_grad():<br>        generation_output = model.generate(<br>            input_ids=input_ids,<br>            generation_config=generation_config,<br>            return_dict_in_generate=True,<br>            output_scores=True,<br>            max_new_tokens=max_new_tokens,<br>            repetition_penalty=1.3,<br>         )<br>    s = generation_output.sequences[0]<br>    output = tokenizer.decode(s)<br><br>    return output.split(&quot;### Response:&quot;)[1].strip()</pre><p>The predictions I generated were suboptimal.</p><p><strong>Question</strong>: quick question — is there a way to register Flyte tasks that are defined in a separate imported repository or package? i.e., repository A installs the package produced by repository B and uses the tasks written in repository B and when I register Flyte workflows in repo A I want it to register all the tasks from both repos</p><p><strong>Response</strong>: &amp;gt; quick question — is there a way to register Flyte tasks that are defined in a separate imported repository or package?</p><p>Yes, you can use the register_task decorator to register tasks defined in a separate repository. You can also use the register_workflow decorator to register workflows defined in a separate repository.</p><p>&amp;gt; i.e., repository A installs the package produced by repository B and uses the tasks written in repository B and when I register Flyte workflows in repo A I want it to register all the tasks from both repos</p><p>You can use the `</p><p><strong>Question</strong>: Hi! Is there a way to shorten ttlSecondsAfterFinished? By default, it is 3600s (1 hour) and we’d like to tear down a cluster right after a job is complete. Thanks for your help! $ k describe rayjobs feb5da8c2a2394fb4ac8-n0–0 -n flytesnacks-development … Ttl Seconds After Finished: 3600</p><p><strong>Response</strong>: cc Yuvraj (union.ai) &lt;|endoftext|&gt;</p><p>It seems that the prompt I used could have been more effective. RedPajama recommended an <a href="https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat#quick-start">alternative prompt</a>, but unfortunately, it did not yield improved results Interestingly, I also noticed a significant number of empty outputs for some unknown reason.</p><pre>&lt;human&gt;: [Instruction]<br>&lt;bot&gt;:</pre><h3><strong>Key takeaways</strong></h3><p>I observed that the performance of the fine-tuned model did not meet my initial expectations. The responses generated by the fine-tuned model tended to be concise, and there were occasional repetitions in the output.</p><p>The brevity of responses can be attributed to the characteristics of the Slack data used for training. To address this, one possible approach could be to filter and consider only longer responses or involve human intervention to select and provide appropriate responses.</p><p>Regarding the issue of repetitions, it seemed to stem from the model being undertrained, as the training loss plateaued while the evaluation loss continued to decrease. Although I attempted to mitigate this problem by applying a repetition penalty, it did not completely eliminate the repetitions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LM_bdJCrUdnFZbHYLRrH2Q.png" /><figcaption>The training loss reached a plateau</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ge0_zeo5UedvXkO_KQ1kzQ.png" /><figcaption>The evaluation loss continued to decrease.</figcaption></figure><p>Interestingly, when I fine-tuned the model on a smaller subset of 1,000 data samples instead of the entire dataset, the responses became longer, although the model occasionally generated hallucinated content. Nevertheless, the responses remained polite and detailed.</p><p><strong>Fine-tuning seems like a promising approach, but its effectiveness heavily relies on the availability and quality of the data</strong>. However, considering the requirement for high-end GPUs solely for training purposes, it may not be the most efficient use of resources. When fine-tuning, it is important to anticipate the response style that aligns with the output format of the training dataset. I noticed that the fine-tuned model often heavily relied on the specific characteristics of the training data, sometimes disregarding the desired response style (even after modifying the prompt to guide it toward generating detailed explanations.)</p><p>In contrast, generating semantic embeddings proved to be a quick, straightforward and effective alternative in most use cases. This cost-effective solution allows you to harness the power of larger and more advanced models like GPT to generate embeddings. Since embeddings operate as distinct modules, they retain the pre-trained knowledge and are less prone to catastrophic forgetting, unlike fine-tuning, which can lead to the loss of previously learned information while adapting to new tasks. When the objective is to impart knowledge to the model, embeddings provide a more consistent response style, primarily focusing on delivering factual information in most cases.</p><p>To summarize my experimentation journey and key takeaways: I initially began fine-tuning the model with a limited understanding of dataset curation and prompt engineering. I quickly learned that the quality of the dataset plays a vital role in the fine-tuning process. On the other hand, dealing with embeddings proved to be relatively simpler and didn’t require extensive dataset cleanup, although it never hurts to ensure cleanliness.</p><p>Moving forward, the next steps involve deciding between fine-tuning and utilizing semantic embeddings. If fine-tuning appears to be a suitable approach, I will proceed with the creation of a high-quality Slack dataset. I would greatly appreciate your feedback and any ideas you may have regarding improvements or further steps that I could consider.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c64e1fad0b9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing COVID-19 Impact on NYC Taxis with Flyte and DuckDB]]></title>
            <link>https://allasamhita.medium.com/analyzing-covid-19-impact-on-nyc-taxis-with-flyte-and-duckdb-b196eb614e26?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/b196eb614e26</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Thu, 23 Mar 2023 15:40:24 GMT</pubDate>
            <atom:updated>2023-03-23T15:40:24.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u0g0JkWemqtly7w0" /><figcaption>Photo by <a href="https://unsplash.com/@dame_visuals?utm_source=medium&amp;utm_medium=referral">Damon Rice</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The world of data analytics is constantly evolving, with new tools and technologies emerging every day. One popular new tool is <a href="https://duckdb.org/">DuckDB</a>, open-source software touted as a faster, more efficient alternative to traditional databases.</p><p>So, what exactly is DuckDB, and why is it gaining so much attention in the data analytics community? In simple terms, DuckDB is a SQL database management system that is designed to handle analytical workloads. Unlike traditional databases that are optimized for transactional workloads, DuckDB is optimized for analytical workloads, making it the preferred choice for data analysts and data scientists.</p><p>DuckDB stands out for its speed and capacity to handle sizable datasets. In this article, we’ll look at DuckDB’s capabilities by running analytical queries on a few gigabytes of NYC taxi data, all within a <a href="https://flyte.org/">Flyte</a> workflow. In the process, we’ll provide insight into how COVID-19 affected pickups, drop-offs, and peak and off-peak hours for NYC taxis, and how Flyte can expedite analytical queries.</p><h3>Fetching pickup and drop-off hours over a span of 4 years</h3><p>Start by retrieving the pickup and drop-off hours of NYC taxis from 2019 to 2022 from the <a href="https://github.com/toddwschneider/nyc-taxi-data/blob/master/setup_files/raw_data_urls.txt">parquet files</a>. Since each parquet file corresponds to a specific year and month, execute queries on each data file using a loop. To handle this dynamic task, utilize flytekit’s <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/dynamics.html">@dynamic</a> workflow.</p><p>Send the resulting list of pickup and drop-off structured datasets to a coalesce_dfs task. This task is responsible for coalescing the structured datasets into a single list, grouping the pickup and drop-off values by year.</p><pre>@dynamic(requests=Resources(mem=&quot;1Gi&quot;, cpu=&quot;2&quot;))<br>def fetch_trips_data(years: List[int], months: List[int]) -&gt; vaex_dfs_list:<br>    result = []<br>    for year in years:<br>        for month in months:<br>            url = f&quot;https://d37ci6vzurychx.cloudfront.net/trip-data/green_tripdata_{year}-{month:02d}.parquet&quot;<br>            pickup_sd = pickup_query(params=[url])<br>            dropoff_sd = dropoff_query(params=[url])<br>            result.append(<br>                {<br>                    &quot;pickup_df&quot;: pickup_sd,<br>                    &quot;dropoff_df&quot;: dropoff_sd,<br>                }<br>            )<br>    return coalesce_dfs(sds=result, years=years)</pre><p>Define peak_hour_output and vaex_dfs_list types as follows:</p><pre>peak_hour_output = List[Dict[str, StructuredDataset]]<br>vaex_dfs_list = List[vaex.dataframe.DataFrameLocal]</pre><p>pickup_query and dropoff_query refer to the DuckDB queries that need to be executed on the parquet files. In Flytekit, a DuckDB query can be defined using the <a href="https://docs.flyte.org/projects/flytekit/en/latest/plugins/generated/flytekitplugins.duckdb.DuckDBQuery.html">DuckDBQuery</a> task. This task requires specification of the task name, the query or list of queries to be executed, and any input parameters required by the query.</p><p>To install the DuckDB plugin, run the following command:</p><pre>pip install flytekitplugins-duckdb</pre><p>To obtain the number of trips for every hour, along with their corresponding pickup and drop-off values, we construct two queries. Design the queries to operate on the parquet file, which will be used as the input data source.</p><pre>from flytekitplugins.duckdb import DuckDBQuery <br><br><br>pickup_query = DuckDBQuery(<br>    name=&quot;pickup_query&quot;,<br>    query=[<br>        &quot;install httpfs&quot;,<br>        &quot;load httpfs&quot;,<br>        &quot;&quot;&quot;select hour(lpep_pickup_datetime) as hour,<br>        count(*) as count<br>        from read_parquet(?)<br>        group by hour&quot;&quot;&quot;,<br>    ],<br>    inputs=kwtypes(params=List[str]),<br>)<br><br>dropoff_query = DuckDBQuery(<br>    name=&quot;dropoff_query&quot;,<br>    query=[<br>        &quot;install httpfs&quot;,<br>        &quot;load httpfs&quot;,<br>        &quot;&quot;&quot;<br>        select hour(lpep_dropoff_datetime) as hour,<br>        count(*) as count<br>        from read_parquet(?)<br>        group by hour<br>        &quot;&quot;&quot;,<br>    ],<br>    inputs=kwtypes(params=List[str]),<br>)</pre><p><strong>The </strong><strong>pickup_query and </strong><strong>dropoff_query tasks are triggered simultaneously in the </strong><strong>@dynamic workflow, resulting in expedited executions!</strong></p><p>Now, we define a coalesce_df task to merge the structured datasets for each year.</p><ol><li>For each pickup and drop-off structured dataset dictionary, add the corresponding year as a key-value pair to the dictionary.</li><li>Sort the structured datasets by year.</li><li>Iterate through the structured datasets, and for each dataset, load it into a Vaex dataframe. Concatenate the pickup dataframe with the previous pickup dataframe, and the drop-off dataframe with the previous drop-off dataframe until all the 12 months have been considered.</li><li>Once the Vaex dataframes for a year are obtained, group the pickup and drop-off dataframes by hour, and join them together.</li><li>Store the resulting dataframe in a list and repeat the process for the next year.</li></ol><p>The result will be a list of Vaex dataframes.</p><pre>@task(requests=Resources(mem=&quot;500Mi&quot;, cpu=&quot;2&quot;))<br>def coalesce_dfs(sds: peak_hour_output, years: List[int]) -&gt; vaex_dfs_list:<br>    # add year to every dictionary<br>    partition_index = int(len(sds) / len(years))<br>    for index, item in enumerate(sds):<br>        item[&quot;year&quot;] = years[index // partition_index]<br><br>    # sort by year<br>    sds.sort(key=lambda x: x[&quot;year&quot;])<br><br>    result_list = []<br>    end_index = len(sds) / len(years)<br>    pickup_df, dropoff_df = None, None<br><br>    for index, item in enumerate(sds, start=1):<br>        materialized_pickup_df = (<br>            item[&quot;pickup_sd&quot;].open(vaex.dataframe.DataFrameLocal).all()<br>        )<br>        materialized_dropoff_df = (<br>            item[&quot;dropoff_sd&quot;].open(vaex.dataframe.DataFrameLocal).all()<br>        )<br>        pickup_df = (<br>            vaex.concat([pickup_df, materialized_pickup_df])<br>            if pickup_df<br>            else materialized_pickup_df<br>        )<br>        dropoff_df = (<br>            vaex.concat([dropoff_df, materialized_dropoff_df])<br>            if dropoff_df<br>            else materialized_dropoff_df<br>        )<br>        # for every year, store the number of pickups and drop-offs<br>        if index % end_index == 0:<br>            year = item[&quot;year&quot;]<br>            pickup_df = pickup_df.groupby(by=&quot;hour&quot;).agg({&quot;count&quot;: &quot;sum&quot;})<br>            dropoff_df = dropoff_df.groupby(by=&quot;hour&quot;).agg({&quot;count&quot;: &quot;sum&quot;})<br>            pickup_df.rename(&quot;count&quot;, &quot;pickup_count&quot;)<br>            dropoff_df.rename(&quot;count&quot;, &quot;dropoff_count&quot;)<br>            result_df = pickup_df.join(dropoff_df, on=&quot;hour&quot;)<br>            result_df[&quot;year&quot;] = np.array([year] * len(result_df))<br>            result_list.append(result_df)<br>            pickup_df, dropoff_df = None, None<br><br>    return result_list</pre><p><strong>Note</strong>: Vaex is designed to handle and process large datasets. It can be helpful in our case to expedite the aggregations and joins of dataframes. Flyte provides native support for Vaex dataframes. You can install the library by running the following command:</p><pre>pip install flytekitplugins-vaex</pre><h3>Visualizing data through plot rendering</h3><p>To estimate the impact of COVID-19 on NYC taxi trips, we can create a visual representation of the pickups and drop-offs between 2019 and 2022.</p><pre>@task(disable_deck=False)<br>def generate_plot(dfs: vaex_dfs_list):<br>    fig = plt.figure()<br>    ax = fig.add_subplot(1, 1, 1)<br>    for df in dfs:<br>        ax.plot(<br>            df.hour,<br>            df.pickup_count,<br>            label=f&quot;{df.year.values[0]} Pickup Count&quot;,<br>            linestyle=&quot;--&quot;,<br>            marker=&quot;o&quot;,<br>        )<br>        ax.plot(<br>            df.hour,<br>            df.dropoff_count,<br>            label=f&quot;{df.year.values[0]} Dropoff Count&quot;,<br>            linestyle=&quot;--&quot;,<br>            marker=&quot;o&quot;,<br>        )<br>    ax.set_xlabel(&quot;hour&quot;)<br>    ax.set_ylabel(&quot;count&quot;)<br>    ax.set_title(&quot;Number of pickups/dropoffs&quot;)<br>    ax.legend(loc=&quot;best&quot;)<br>    fig.savefig(&quot;plot.png&quot;)<br><br>    flytekit.Deck(&quot;matplotlib&quot;, mpld3.fig_to_html(fig))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pwNXGE8WCDV9fkSp.png" /><figcaption><em>Pickups and drop-offs between 2019 and 2022</em></figcaption></figure><p>Install the following libraries prior to rendering the plot:</p><pre>pip install mpld3<br>pip install vaex-viz</pre><p>The plot suggests that in 2019, there were significantly more instances of pickups and drop-offs compared to 2020, 2021, and 2022. At 18:00 hour, for instance, there was a noticeable drop in pickups from about 400,000 in 2019 to roughly 90,000 in 2020, and even lower at around 50,000 in 2021 and 2022.</p><h3>Retrieving the peak and off-peak hours</h3><p>To retrieve the peak and off-peak hours for each year from 2019 to 2022, we can follow these steps:</p><ol><li>Define a @dynamic workflow to loop over the Vaex dataframes retrieved from the coalesce_dfs task.</li><li>Generate an Arrow table from every Vaex dataframe.</li><li>Execute two DuckDB queries to retrieve peak and off-peak hours by sending the arrow tables to the queries.</li><li>Define a post-processing task to retrieve the peak and off-peak hours from the dataframes.</li><li>Return the result.</li></ol><pre>@task<br>def peak_offpeak_postprocess(<br>    df: vaex.dataframe.DataFrameLocal,<br>    peak_hour_sd: StructuredDataset,<br>    off_peak_hour_sd: StructuredDataset,<br>) -&gt; PeakOffPeak:<br>    peak_hour = peak_hour_sd.open(pd.DataFrame).all().iloc[0][&quot;hour&quot;].item()<br>    off_peak_hour = off_peak_hour_sd.open(pd.DataFrame).all().iloc[0][&quot;hour&quot;].item()<br><br>    return PeakOffPeak(df.year.values[0].as_py(), peak_hour, off_peak_hour)<br><br><br>@dynamic(requests=Resources(mem=&quot;500Mi&quot;, cpu=&quot;2&quot;))<br>def peak_offpeak_hours(dfs: vaex_dfs_list) -&gt; List[PeakOffPeak]:<br>    result = []<br>    for df in dfs:<br>        arrow_table = df.to_arrow_table()<br>        peak_hour_sd = peak_hour_query(arrow_table=arrow_table)<br>        off_peak_hour_sd = off_peak_hour_query(arrow_table=arrow_table)<br>        df_result = peak_offpeak_postprocess(<br>            df=df,<br>            peak_hour_sd=peak_hour_sd,<br>            off_peak_hour_sd=off_peak_hour_sd,<br>        )<br>        result.append(df_result)<br>    return result</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K738ONgnNBfdqSom.png" /><figcaption><em>Peak and off-peak hours</em></figcaption></figure><p>peak_hour_query and off_peak_hour_query execute SQL queries on the Arrow tables and retrieve the peak and off-peak hours, respectively.</p><p>Peak hour refers to the hour with the highest number of pickups and drop-offs, while off-peak hour refers to the hour with the lowest number of pickups and drop-offs.</p><pre>peak_hour_query = DuckDBQuery(<br>    name=&quot;peak_hour_query&quot;,<br>    query=&quot;&quot;&quot;select hour<br>        from arrow_table<br>        where pickup_count + dropoff_count = (select max(pickup_count + dropoff_count)<br>                                               from arrow_table)&quot;&quot;&quot;,<br>    inputs=kwtypes(arrow_table=pa.Table),<br>)<br><br>off_peak_hour_query = DuckDBQuery(<br>    name=&quot;off_peak_hour_query&quot;,<br>    query=&quot;&quot;&quot;select hour<br>        from arrow_table<br>        where pickup_count + dropoff_count = (select min(pickup_count + dropoff_count)<br>                                               from arrow_table)&quot;&quot;&quot;,<br>    inputs=kwtypes(arrow_table=pa.Table),<br>)</pre><p>PeakOffPeak is a data class that specifies the format for storing the year, peak hour, and off-peak hour.</p><pre>@dataclass_json<br>@dataclass<br>class PeakOffPeak:<br>    year: int<br>    peak_hour: int<br>    off_peak_hour: int</pre><h3>Running the workflow</h3><p>You can execute the workflow either locally or on a Flyte cluster. To establish dependencies between tasks, build a <a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.workflow.html#flytekit-workflow">Flyte workflow</a> using the following steps:</p><pre>@workflow<br>def duckdb_wf(<br>    years: List[int] = [2019, 2020, 2021, 2022], months: List[int] = list(range(1, 12))<br>) -&gt; List[PeakOffPeak]:<br>    result_dfs = fetch_trips_data(years=years, months=months)<br>    generate_plot(dfs=result_dfs)<br>    return peak_offpeak_hours(dfs=result_dfs)</pre><p><strong>Note</strong>: The source code is available on <a href="https://gist.github.com/samhita-alla/003c3f409e8caa88470f6f7206b54ae3">GitHub Gist</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lBNvHwrC9FkjLVe-.png" /><figcaption><em>An illustration of the end-to-end workflow presented on the Flyte UI</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JzfS8SZF2t3P04I4.png" /><figcaption><em>Zoomed in…</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yoBwc__epBp040xW.png" /><figcaption><em>An illustration of the </em><em>peak_offpeak_hours dynamic workflow</em></figcaption></figure><p><strong>Note</strong>: You may come across access denied errors when hitting multiple CloudFront URLs to retrieve the data. In such a scenario, wait for about an hour and recover the existing execution. This way, you can prevent re-execution of successful Parquet file executions. Flyte’s <a href="https://flyte.org/features#recover-from-failures">recoverability</a> feature can help!</p><h3>Summary</h3><p>In conclusion, leveraging the Flyte x DuckDB integration can simplify the execution of complex analytical workloads on Flyte. If you are interested in running DuckDB queries within Flyte, here are some valuable resources to explore:</p><ul><li>Review our <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/flytekit_plugins/duckdb_examples/index.html">documentation</a> to gain a better understanding of how to use the Flyte x DuckDB integration.</li><li>Take a look at our <a href="https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-duckdb">plugin code</a> to get a better sense of how the integration works under the hood.</li><li>Join our <a href="https://slack.flyte.org/">Slack</a> community if you have any questions or want to engage with other users who are using Flyte x DuckDB.</li><li>Give the <a href="https://github.com/flyteorg/flyte">Flyte repository</a> a ⭐ on GitHub and follow us on <a href="https://twitter.com/flyteorg">Twitter</a> to stay informed about the latest updates and enhancements.</li><li><a href="https://github.com/flyteorg/flyte/issues">Create a GitHub issue</a> if you run into any bugs and let us know.</li></ul><p>By leveraging the Flyte x DuckDB integration, you can optimize your analytical workflows for greater efficiency.</p><p><em>Originally published at </em><a href="https://flyte.org/blog/analyzing-covid-19-impact-on-nyc-taxis-with-duckdb"><em>https://flyte.org</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b196eb614e26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10-Minute Guide to Accelerate Your ML Pipeline from Data to Deployment]]></title>
            <link>https://allasamhita.medium.com/10-minute-guide-to-accelerate-your-ml-pipeline-from-data-to-deployment-2440a9847642?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/2440a9847642</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 10:42:22 GMT</pubDate>
            <atom:updated>2023-03-13T10:42:22.963Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn to build a production-grade ML pipeline with Flyte, Banana, and Hugging Face</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G--Ick0ZB6cscOoA" /><figcaption>Photo by <a href="https://unsplash.com/@spacex?utm_source=medium&amp;utm_medium=referral">SpaceX</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Before you train an ML model requires you to procure, prepare and analyze the data you’ll use to build and train it. When there’s additional data, you may need to repeat the steps multiple times — a process known as “retraining.” To ensure reproducibility, it may be necessary to version a retrained model. Additionally, in order to keep the execution cost-effective, it may be necessary to cache the outputs of the model. To complete the loop, the model must be deployed so that end users can generate predictions.</p><p>I set out on a journey to build an application that would <a href="https://www.union.ai/blog-post/orchestration-for-data-machine-learning-and-infrastructure">orchestrate</a> a fine-tuned pipeline and enable on-demand, scalable model serving. My approach:</p><ul><li>Build a fine-tuning pipeline leveraging 🤗 <a href="https://huggingface.co/docs/transformers/index">Hugging Face transformers</a>.</li><li>Allocate additional memory for resource-intensive operations and a GPU to expedite model training.</li><li>Cache the dataset to reduce the execution time.</li><li>Upon completion of training, push the trained model to 🤗 Hub.</li><li>Enable the user to approve or reject the model deployment.</li><li>If approved, retrieve the model from the Hub and deploy it.</li><li>Invoke the endpoint to generate predictions.</li></ul><p>In addition, I wanted to ensure that the end-to-end pipeline maintained data lineage and versioning.</p><p>In the following sections, I’ll take you through a code walkthrough of the <a href="https://flyte.org/">Flyte</a> x <a href="https://banana.dev/">Banana</a> integration. Let’s dive in!</p><blockquote><em>Do you want to jump directly to the code? It’s on </em><a href="https://github.com/samhita-alla/flyte-banana"><em>GitHub</em></a><em>.</em></blockquote><h3>⛓️ The fine-tuning pipeline</h3><p>Let’s use a <a href="https://huggingface.co/bert-base-uncased">bert-base-uncased</a> pre-trained model and fine-tune it on the <a href="https://huggingface.co/datasets/yelp_review_full">Yelp dataset</a>. We can break the pipeline down into four steps:</p><ol><li>Downloading the dataset</li><li>Tokenizing the dataset</li><li>Splitting the dataset into training and evaluation subsets</li><li>Training an ML model using the subsets.</li></ol><p>Let’s use <a href="https://github.com/flyteorg/flytekit">Flytekit</a> — Flyte’s Python SDK — to build the pipeline.</p><h4>📥 Downloading the dataset</h4><p>Download the <a href="https://huggingface.co/datasets/yelp_review_full">yelp_review_full</a> dataset and store it in a directory. Use <a href="https://flyte.org/features#FlyteFile-FlyteDirectory">FlyteDirectory</a>, a custom Flyte type that facilitates smooth communication between Flyte tasks, to enable automatic uploading and downloading of directories. <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/customizing_resources.html">Configure the resources</a> to guarantee that the task has adequate resources to acquire the dataset. Enable faster execution by <a href="https://flyte.org/features#cache-outputs">caching</a> the task output.</p><pre>@task(<br>    cache=True,<br>    cache_version=&quot;1.0&quot;,<br>    requests=Resources(mem=&quot;1Gi&quot;, cpu=&quot;2&quot;, ephemeral_storage=ephemeral_storage),<br>)<br>def download_dataset() -&gt; FlyteDirectory:<br>    dataset = load_dataset(&quot;yelp_review_full&quot;)<br><br>    local_dir = create_local_dir(dir_name=&quot;yelp_data&quot;)<br><br>    dataset.save_to_disk(dataset_dict_path=local_dir)<br>    return FlyteDirectory(path=str(local_dir))</pre><h4>🧹 Tokenizing the data</h4><p>Load a predefined bert-base-uncased tokenizer, apply it to the Yelp dataset, and store the resulting output in a directory. In addition, cache the tokenized data and allocate the necessary resources for the task. This will improve the task’s execution speed and efficiency.</p><pre>@task(<br>    cache=True,<br>    cache_version=&quot;1.0&quot;,<br>    requests=Resources(mem=&quot;1Gi&quot;, cpu=&quot;2&quot;, ephemeral_storage=ephemeral_storage),<br>)<br>def tokenize(dataset: FlyteDirectory) -&gt; FlyteDirectory:<br>    downloaded_path = dataset.download()<br>    loaded_dataset = load_from_disk(downloaded_path)<br>    tokenized_dataset = loaded_dataset.map(tokenize_function, batched=True)<br><br>    local_dir = create_local_dir(dir_name=&quot;yelp_tokenized_data&quot;)<br><br>    tokenized_dataset.save_to_disk(dataset_dict_path=local_dir)<br>    return FlyteDirectory(path=str(local_dir))</pre><h4>🖖 Splitting the dataset</h4><p>Divide the dataset into two separate subsets for training and evaluation, each with the datasets.Dataset type, which is already defined in the <a href="https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-huggingface">flytekitplugins-huggingface</a> library.</p><pre>@task(requests=Resources(mem=&quot;1Gi&quot;, cpu=&quot;2&quot;, ephemeral_storage=ephemeral_storage))<br>def get_train_eval(<br>    tokenized_dataset: FlyteDirectory,<br>) -&gt; datasets_tuple:<br>    downloaded_path = tokenized_dataset.download()<br>    loaded_tokenized_dataset = load_from_disk(downloaded_path)<br><br>    small_train_dataset = (<br>        loaded_tokenized_dataset[&quot;train&quot;].shuffle(seed=42).select(range(dataset_size))<br>    )<br>    small_eval_dataset = (<br>        loaded_tokenized_dataset[&quot;test&quot;].shuffle(seed=42).select(range(dataset_size))<br>    )<br>    return datasets_tuple(<br>        train_dataset=StructuredDataset(dataframe=small_train_dataset),<br>        eval_dataset=StructuredDataset(dataframe=small_eval_dataset),<br>    )</pre><h4>🤖 Fine-tuning BERT model</h4><p>Allocate a GPU to the train task to accelerate fine-tuning. Initialize secret_requests to retrieve the 🤗 token. Load the pre-trained bert-base-uncased model, initialize TrainingArguments, and use Trainer to train the model using tokenized data. Finally, publish the trained model to 🤗 hub, and return the GitHub SHA of the published model, Flyte execution ID (to maintain lineage), and repository name.</p><blockquote><em>To store secrets in Flyte, you can utilize a secrets management system, with Kubernetes secrets being the default. In the following section, we’ll see how to create a secret.</em></blockquote><pre>@task(<br>    requests=Resources(mem=mem, cpu=&quot;2&quot;, gpu=gpu, ephemeral_storage=ephemeral_storage),<br>    secret_requests=[<br>        Secret(<br>            group=HF_SECRET_GROUP,<br>            key=HF_SECRET_NAME,<br>            mount_requirement=Secret.MountType.FILE,<br>        )<br>    ],<br>)<br>def train(<br>    small_train_df: StructuredDataset, small_eval_df: StructuredDataset, hf_user: str<br>) -&gt; dict:<br>    HUGGING_FACE_HUB_TOKEN = flytekit.current_context().secrets.get(<br>        HF_SECRET_GROUP, HF_SECRET_NAME<br>    )<br>    repo = f&quot;{hf_user}/{HUGGINGFACE_REPO}&quot;<br>    execution_id = flytekit.current_context().execution_id.name<br>    small_train_dataset = small_train_df.open(Dataset).all()<br>    small_eval_dataset = small_eval_df.open(Dataset).all()<br><br>    model = AutoModelForSequenceClassification.from_pretrained(<br>        &quot;bert-base-uncased&quot;, num_labels=5<br>    )<br><br>    training_args = TrainingArguments(<br>        output_dir=HUGGINGFACE_REPO,<br>        evaluation_strategy=&quot;epoch&quot;,<br>        push_to_hub=True,<br>        hub_token=HUGGING_FACE_HUB_TOKEN,<br>    )<br>    trainer = Trainer(<br>        model=model,<br>        args=training_args,<br>        train_dataset=small_train_dataset,<br>        eval_dataset=small_eval_dataset,<br>        compute_metrics=compute_metrics,<br>    )<br><br>    trainer.train()<br>    trainer.push_to_hub(<br>        commit_message=f&quot;End of training - Flyte execution ID {execution_id}&quot;<br>    )<br>    return {<br>        &quot;sha&quot;: model_info(repo).sha,<br>        &quot;execution_id&quot;: execution_id,<br>        &quot;repo&quot;: repo,<br>    }</pre><h3>🗄️ Sending model metadata to GitHub</h3><p>Because Banana must be deployed through GitHub, locate all Banana deployment files in a GitHub repository. The deployment is triggered by a push event once a Banana GitHub action is included in the appropriate repository.</p><p>Initiating Banana deployment from within a Flyte workflow requires triggering a push event in this instance, we will push the model metadata.</p><p>Take the model metadata retrieved from the train task and add it to a model_metadata.json file. When transmitting the data through the GitHub API, it must be converted to base64 encoding. Prior to generating the commit, obtain the most recent commit SHA to be used when sending the push event. Utilize the subprocess library to hit the createCommitOnBranch endpoint.</p><pre>@task(<br>    secret_requests=[<br>        Secret(<br>            group=SECRET_GROUP, key=SECRET_NAME, mount_requirement=Secret.MountType.FILE<br>        )<br>    ],<br>    requests=Resources(mem=&quot;1Gi&quot;, ephemeral_storage=ephemeral_storage),<br>)<br>def push_to_github(<br>    model_metadata: dict, gh_owner: str, gh_repo: str, gh_branch: str<br>) -&gt; str:<br>    token = flytekit.current_context().secrets.get(SECRET_GROUP, SECRET_NAME)<br><br>    additions = [<br>        {<br>            &quot;path&quot;: f&quot;banana/model_metadata.json&quot;,<br>            &quot;contents&quot;: f&quot;{base64.urlsafe_b64encode(json.dumps(model_metadata).encode()).decode(&#39;utf-8&#39;)}&quot;,<br>        }<br>    ]<br><br>    sha_result = subprocess.run(<br>        f&quot;&quot;&quot;<br>        curl -s -H &quot;Authorization: bearer {token}&quot; \<br>        -H &quot;Accept: application/vnd.github.VERSION.sha&quot; \<br>        &quot;https://api.github.com/repos/{gh_owner}/{gh_repo}/commits/main&quot;<br>        &quot;&quot;&quot;,<br>        shell=True,<br>        capture_output=True,<br>        text=True,<br>    )<br><br>    github_sha = sha_result.stdout<br><br>    cmd = f&quot;&quot;&quot;<br>curl https://api.github.com/graphql -s -H &quot;Authorization: bearer {token}&quot; --data @- &lt;&lt; GRAPHQL | jq &#39;.data.createCommitOnBranch.commit.url[0:56]&#39;<br>{{<br>   &quot;query&quot;: &quot;mutation (\$input: CreateCommitOnBranchInput!) {{<br>   createCommitOnBranch(input: \$input) {{<br>       commit {{<br>         url<br>       }}<br>   }}<br>   }}&quot;,<br>   &quot;variables&quot;: {{<br>     &quot;input&quot;: {{<br>       &quot;branch&quot;: {{<br>         &quot;repositoryNameWithOwner&quot;: &quot;{gh_owner}/{gh_repo}&quot;,<br>         &quot;branchName&quot;: &quot;{gh_branch}&quot;<br>       }},<br>       &quot;message&quot;: {{<br>       &quot;headline&quot;: &quot;Update the model artifact&quot;<br>       }},<br>       &quot;fileChanges&quot;: {{<br>         &quot;additions&quot;: {json.dumps(additions)}<br>       }},<br>       &quot;expectedHeadOid&quot;: &quot;{github_sha}&quot;<br>     }}<br>   }}<br>}}<br>GRAPHQL<br>&quot;&quot;&quot;<br><br>    result = subprocess.run(<br>        cmd,<br>        shell=True,<br>        capture_output=True,<br>        text=True,<br>    )<br><br>    return result.stdout</pre><p>Refer to the <a href="https://github.com/samhita-alla/flyte-banana/blob/main/flyte/workflows/ml_pipeline.py">end-to-end pipeline available on the GitHub repository</a> for a comprehensive overview.</p><h3>▶️ Running the pipeline locally</h3><p>In order to execute the model locally, you must first store the secrets in a local file.</p><ul><li>Obtain access tokens for HuggingFace and GitHub and store them in files within a secrets directory, as follows:</li></ul><pre>&lt;your-secrets-dir&gt;/deployments-secrets/flyte-banana-creds<br>&lt;your-secrets-dir&gt;/hf-secrets/flyte-banana-hf-creds</pre><ul><li>Within a .env file, set the following two variables to enable local code execution:</li></ul><pre>FLYTE_SECRETS_DEFAULT_DIR=&lt;your-secrets-dir&gt;<br>DEMO=&quot;1&quot;</pre><p>Next, install the necessary <a href="https://github.com/samhita-alla/flyte-banana/blob/main/flyte/requirements.txt">requirements</a> in a virtual environment.</p><p>To run the Flyte workflow, use the following command:</p><pre>pyflyte run ml_pipeline.py --gh_owner &lt;your-github-username&gt; --gh_repo &lt;your-github-repo&gt; --gh_branch &lt;your-github-repo-branch&gt; --hf_user &lt;your-huggingface-username&gt;</pre><p>The pipeline retrieves the data, tokenizes it and trains a model. Then it waits for the user to approve the push event before pushing the model metadata to GitHub, which would trigger a Banana deployment.</p><h3>🍌 Serving on Banana</h3><p>Banana is an ML inference solution that does inference on serverless GPUs.</p><h4>🛠️ Setting it up</h4><p>To activate the Banana deployment, you need to:</p><ol><li>Create an account.</li><li>Configure your Banana account by creating a deployment and linking it to the forked GitHub repository. To do this, navigate to Team &gt; Integrations &gt; GitHub &gt; Manage Repos and Deploy &gt; Deploy from GitHub &gt; your repository.</li><li>That’s all! Every push to the configured GitHub repository will now trigger a deployment.</li></ol><h4>🪚 Adding code for inference</h4><p>The Banana inference code needs to be encapsulated in the app.py and server.py files.</p><p>In the app.py file, the following steps should be taken:</p><ul><li>Define an init() method. This should involve fetching the model from the 🤗 hub and loading it onto a GPU.</li><li>Define an inference() method. This should involve tokenizing the user-given prompt and generating a prediction.</li></ul><pre>import json<br><br>import torch<br>from transformers import AutoModelForSequenceClassification, AutoTokenizer<br><br><br># Init is ran on server startup<br># Load your model to GPU as a global variable here using the variable name &quot;model&quot;<br>def init():<br>    global model<br><br>    with open(&quot;model_metadata.json&quot;) as f:<br>        model_metadata = json.load(f)<br><br>    device = &quot;cuda:0&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>    model = AutoModelForSequenceClassification.from_pretrained(<br>        pretrained_model_name_or_path=model_metadata[&quot;repo&quot;],<br>        num_labels=5,<br>        revision=model_metadata[&quot;sha&quot;],<br>    ).to(device)<br><br><br># Inference is ran for every server call<br># Reference your preloaded global model variable here.<br>def inference(model_inputs: dict) -&gt; dict:<br>    global model<br><br>    # Parse out your arguments<br>    prompt = model_inputs.get(&quot;prompt&quot;, None)<br>    if prompt == None:<br>        return {&quot;message&quot;: &quot;No prompt provided&quot;}<br><br>    tokenizer = AutoTokenizer.from_pretrained(&quot;bert-base-cased&quot;)<br><br>    device = &quot;cuda:0&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>    encoding = tokenizer(<br>        prompt, padding=&quot;max_length&quot;, truncation=True, return_tensors=&quot;pt&quot;<br>    ).to(device)<br><br>    # Run the model<br>    outputs = model(**encoding)<br>    prediction = outputs.logits.argmax(-1)<br><br>    # Return the result as a dictionary<br>    return {&quot;result&quot;: prediction.item()}</pre><p>Refer to the <a href="https://github.com/samhita-alla/flyte-banana/blob/main/banana/server.py">server.py file code on GitHub</a>.</p><h4>🧪 Testing the endpoint</h4><p>To run the Banana server locally, execute the command python server.py. Confirm that the Banana API endpoint is functional by running the following test case:</p><pre>import argparse<br><br>import requests<br><br><br>def generate_predictions(args):<br>    model_inputs = {&quot;prompt&quot;: args.prompt}<br>    res = requests.post(args.url, json=model_inputs)<br>    print(res.json())<br><br><br>if __name__ == &quot;__main__&quot;:<br>    parser = argparse.ArgumentParser()<br>    parser.add_argument(<br>        &quot;--url&quot;, default=&quot;http://localhost:8000/&quot;, type=str, help=&quot;API endpoint URL&quot;<br>    )<br>    parser.add_argument(<br>        &quot;--prompt&quot;,<br>        default=&quot;The service is terrible, the staff seem to be generally clueless, the management is inclined to blame the staff for their own mistakes, and there&#39;s no sense of FAST in their fast food.&quot;,<br>        type=str,<br>        help=&quot;Prompt&quot;,<br>    )<br>    args = parser.parse_args()<br>    generate_predictions(args=args)</pre><blockquote><em>Install the </em><a href="https://github.com/samhita-alla/flyte-banana/blob/main/banana/requirements.txt"><em>Banana requirements</em></a><em> prior to running the command.</em></blockquote><h4>📤 Deploying the model</h4><p>To deploy the model on Banana, prepare a Dockerfile and place it at the root of your GitHub repository:</p><pre>#####################<br># BANANA DOCKERFILE #<br>#####################<br><br># Must use cuda version 11+<br>FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime<br><br>WORKDIR /root<br><br># Install git<br>RUN apt-get update &amp;&amp; apt-get install -y git<br><br># Install python packages<br>RUN pip3 install --upgrade pip<br>ADD banana/requirements.txt requirements.txt<br>RUN pip3 install -r requirements.txt<br><br># We add the banana boilerplate here<br>ADD banana/server.py /root<br><br># Add your custom app code, init() and inference()<br>ADD banana/app.py /root<br><br># Add model metadata<br>ADD banana/model_metadata.json /root<br><br>EXPOSE 8000<br><br>CMD python3 -u server.py</pre><p>Once the files are in place, run the Flyte pipeline locally again. Approve the model metadata push to GitHub, and the model should be built and deployed on Banana!</p><h3>🫐 Running the pipeline on Flyte cluster</h3><p>Create a Dockerfile that includes the necessary requirements to package and register your Flyte workflow.</p><pre>FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime<br><br>WORKDIR /root<br>ENV VENV /opt/venv<br>ENV LANG C.UTF-8<br>ENV LC_ALL C.UTF-8<br>ENV PYTHONPATH /root<br><br>RUN apt-get update &amp;&amp; apt-get install -y build-essential git-lfs curl<br>RUN pip3 install awscli<br><br>ENV VENV /opt/venv<br><br># Virtual environment<br>RUN python3 -m venv ${VENV}<br>ENV PATH=&quot;${VENV}/bin:$PATH&quot;<br><br># Install Python dependencies<br>COPY ./requirements.txt /root<br>RUN pip install -r /root/requirements.txt<br><br>COPY workflows /root/workflows<br><br># This tag is supplied by the build script and will be used to determine the version<br># when registering tasks, workflows, and launch plans<br>ARG tag<br>ENV FLYTE_INTERNAL_IMAGE $tag</pre><p>Create Kubernetes secrets to store GitHub and HuggingFace tokens as follows:</p><pre>kubectl create secret generic deployment-secrets --namespace flytesnacks-development --from-file=flyte-banana-creds=&lt;your-secrets-dir&gt;/deployment-secrets/flyte-banana-creds<br><br>kubectl create secret generic hf-secrets --namespace flytesnacks-development --from-file=flyte-banana-hf-creds=&lt;your-secrets-dir&gt;/hf-secrets/flyte-banana-hf-creds</pre><p>To run the Flyte workflow on an actual Flyte backend, <a href="https://docs.flyte.org/en/latest/deployment/deployment/index.html">set up a Flyte cluster</a>. The simplest way to get started is by running flytectl demo start command, which spins up a mini-replica of the Flyte deployment.</p><p>Register tasks and workflows using the following command, which can leverage the<a href="https://docs.flyte.org/en/latest/deployment/deployment/sandbox.html#start-the-sandbox"> docker registry included with the demo cluster</a> for image pushing and pulling:</p><pre>pyflyte register --image &lt;flyte-docker-image&gt; ml_pipeline.py</pre><p>And then, launch the registered workflow on the UI. To deploy the retrained model on Banana, click “Approve.” This action saves the model metadata in the GitHub repository, and the push action triggers a deployment on Banana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CrArX9iJqocHL6W-.png" /><figcaption>Execution graph in the Flyte UI</figcaption></figure><h3>🧪 Testing the Banana deployment</h3><p>To test your deployment, retrieve the API and model keys. Store these keys in your local environment, and then execute the following test:</p><pre>import argparse<br>import os<br><br>import banana_dev as banana<br>from dotenv import load_dotenv<br><br>load_dotenv()<br><br>api_key = os.getenv(&quot;BANANA_API_KEY&quot;)<br>model_key = os.getenv(&quot;BANANA_MODEL_KEY&quot;)<br><br><br>def generate_predictions(args):<br>    print(banana.run(api_key, model_key, {&quot;prompt&quot;: args.prompt})[&quot;modelOutputs&quot;][0])<br><br><br>if __name__ == &quot;__main__&quot;:<br>    parser = argparse.ArgumentParser()<br>    parser.add_argument(<br>        &quot;--prompt&quot;,<br>        default=&quot;The service is terrible, the staff seem to be generally clueless, the management is inclined to blame the staff for their own mistakes, and there&#39;s no sense of FAST in their fast food.&quot;,<br>        type=str,<br>        help=&quot;Prompt&quot;,<br>    )<br>    args = parser.parse_args()<br>    generate_predictions(args=args)</pre><p>Upon running the code, you should observe a predicted label being returned.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F807448098%3Fh%3D78525d650b%26app_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F807448098&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1630912628-31ab7bd1e5dd7eaaebb11d1015d9e666f38c17755193f32d97b3ad6d485dbe8a-d_1280&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1200" frameborder="0" scrolling="no"><a href="https://medium.com/media/eebe19358515df1853a137352ef5f6f8/href">https://medium.com/media/eebe19358515df1853a137352ef5f6f8/href</a></iframe><h3>🥡 Takeaways</h3><p>Hopefully, you’ve learned a bunch about Flyte and Banana! Orchestration and inference can go hand-in-hand seamlessly as demonstrated in this piece. You can ensure your ML pipelines are versioned, cached and reproducible, and at the same time, run online inference at scale on GPUs.</p><p>Here are some key takeaways from this application:</p><ul><li>Ensure coherence between retraining ML models and deployment</li><li>Human-in-the-loop can power your deployment with Flyte orchestration</li><li>GPU-powered serverless inference with Banana</li><li>Comprehensive data lineage across the entire model development and deployment pipelines for easier debugging</li><li>Every model is versioned through 🤗 hub</li><li>Flyte versioning ensures versioned ML pipeline executions</li></ul><p>Flyte and Banana offer the potential to create production-grade ML pipelines with ease. If this resonates with your needs, I encourage you to try these tools.</p><p>Thanks for reading this far! I hope you found this helpful.</p><p><em>Originally published at </em><a href="https://flyte.org/blog/how-to-serve-ml-models-with-banana"><em>https://flyte.org</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2440a9847642" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is Airflow the Right Choice for Machine Learning Too?]]></title>
            <link>https://medium.com/better-programming/is-airflow-the-right-choice-for-machine-learning-too-bead4eb6207d?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/bead4eb6207d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Thu, 21 Jul 2022 14:44:28 GMT</pubDate>
            <atom:updated>2022-07-22T04:43:14.840Z</atom:updated>
            <content:encoded><![CDATA[<h4>A look at the differences between ETL and machine learning tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GAvfsjX5JGyVO3Wt" /><figcaption>Photo by <a href="https://unsplash.com/@charlesdeluvio">charlesdeluvio</a> from <a href="https://images.unsplash.com/photo-1580064201713-696b91df7712?ixid=MXwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHw%3D&amp;ixlib=rb-1.2.1&amp;auto=format&amp;fit=crop&amp;w=668&amp;q=80">Unsplash</a></figcaption></figure><p>Apache Airflow is an open source platform that can be used to author, monitor, and schedule data pipelines. It is used by companies like Airbnb, Lyft, and Twitter and has been the go-to tool in the data engineering ecosystem.</p><p>With an increased necessity for orchestration of data pipelines, Airflow witnessed tremendous growth. It has broadened its scope from data to machine learning and is now being used for a variety of use cases.</p><p>But since machine learning in itself demands a distinctive orchestration, Airflow needs to be extended to accommodate all the MLOps requirements.</p><p>To understand this better, let’s dive deeper into the differences between machine learning and ETL (extract-transform-load) tasks and the machine learning engineering requirements.</p><h3>Machine Learning vs. ETL Tasks</h3><p>Machine learning tasks have different requirements from ETL tasks.</p><ul><li>Machine learning needs to run resource-intensive computations.</li><li>Machine learning needs data lineage in place or a data-aware platform to simplify the debugging process.</li><li>Machine learning needs code to be versioned to ensure reproducibility.</li><li>Machine learning needs infrastructure automation.</li></ul><p>Moreover, there has been a substantial shift from static to dynamic workflows. The processes are fast and unpredictable, more computing capacity is being utilized to run workflows, and iteration is a primitive requirement, among others. Hence, the legacy way of handling pipelines wouldn’t work.</p><h3>Why Extend Airflow?</h3><p>Airflow was designed to orchestrate data workflows. It is useful to run ETL tasks due to the ease with which it connects to a standard set of third-party sources to achieve data orchestration. Machine Learning practitioners, however, have additional requirements.</p><ul><li>Resource-intensive tasks: Airflow doesn’t propose a direct way to handle this, although you can use a <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html">KubernetesPodOperator</a> to overcome resource constraints by containerizing the code.</li><li>Data lineage: Airflow is not a data-aware platform; it simply constructs the flow but is oblivious to the inner details comprising passing around the data.</li><li>Versioning: There have been some indirect techniques to handle versioning of the code in Airflow, but it still isn’t an evident feature. Moreover, Airflow also doesn’t support revisiting the DAGs and rerunning the workflows on demand.</li><li>Infrastructure automation: For the most part, machine learning code needs high processing power. However, we do not want to keep the resources idle when the workflows aren’t running or when a certain number of tasks in a workflow do not require more computing capacity—this is difficult to achieve through Airflow.</li><li>Caching: Caching task outputs can help expedite the executions and eliminate the reuse of resources, which, again, Airflow doesn’t support.</li></ul><p>Airflow supports building machine learning pipelines by leveraging operators such as Amazon Sagemaker and Databricks. The issue is that we wouldn’t be receiving all the orchestration benefits as such, and no reproducibility or any of the aforementioned guarantees can be ensured at the Airflow end. In a nutshell, the fundamental missing piece in this scenario is fine-grained machine learning orchestration.</p><p>Imagine a scenario where machine learning engineers are brought into an existing Airflow project that has been set up by data engineers. There is often such a big gap in terminology, conceptual approaches, and lack of understanding of the work efforts of each team because Airflow’s primarily a data-orchestration-oriented platform. To make Airflow a full-fledged machine learning orchestration service requires greater effort than usual and might not be the right choice, after all, considering Airflow’s inclination towards ETL.</p><p>That said, Airflow can still be used to construct “static” machine learning pipelines, but if your use case is to adapt to the ever-evolving and fast-paced ecosystem around machine learning with an extreme focus on iteration (and, of course, deployment), you may have to extend Airflow.</p><h3>Airflow + Flyte = Maximum Flexibility</h3><p>Flyte was born out of solving the challenges that Airflow presented, especially to machine learning engineers, practitioners, and data scientists. It’s a full-fledged workflow orchestration service built on top of Kubernetes to ensure the scalability of machine learning pipelines. So, why not leverage the best of both worlds — Airflow and Flyte — to scale Airflow for machine learning tasks?</p><p>Introducing <a href="https://github.com/flyteorg/airflow-provider-flyte/tree/master">Flyte Airflow Provider</a>. You can now construct your ETL pipelines in Airflow and machine learning pipelines in Flyte and use the provider to trigger the machine learning or Flyte pipelines from within Airflow.</p><p>If you’re a longtime user of Airflow, you can approach the code piecemeal without worrying about how Airflow and Flyte intertwine together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wp9YyarmCfE45EAb" /><figcaption><em>An example DAG on the Airflow UI where </em><em>train_model is a </em><em>FlyteOperator (Image by Author)</em></figcaption></figure><h4>Flyte operator</h4><p>You can use FlyteOperator to trigger a Flyte execution — be it a task or a workflow execution — from within Airflow, before which you&#39;d have to register your Flyte workflows on the backend. On triggering the Airflow DAG, the corresponding Flyte execution gets triggered, and voila!, you&#39;d receive all the benefits of Flyte within Airflow.</p><h4>Flyte sensor</h4><p>If you need to wait for execution to complete, FlyteSensor comes into the picture. Monitoring with FlyteSensor allows you to trigger downstream processes only when the Flyte executions are complete.</p><h4>Example</h4><p>Place the following under dags/ directory after <a href="https://www.astronomer.io/guides/get-started-airflow-2/">installing and setting up Airflow</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5ab8b074b5d8e693665169b946b9ccd3/href">https://medium.com/media/5ab8b074b5d8e693665169b946b9ccd3/href</a></iframe><p>The source code of the pima_diabetes Python file is available <a href="https://github.com/flyteorg/flytesnacks/blob/master/cookbook/case_studies/ml_training/pima_diabetes/diabetes.py">here</a>.</p><p>Add airflow-provider-flyte to the requirements.txt file and create an Airflow connection to Flyte by setting the Conn Id to flyte_conn and Conn Type to Flyte.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qNeZkpdsmZMYg0AI" /><figcaption><em>Flyte execution UI (Image by Author)</em></figcaption></figure><p>On terminating the execution either at the Airflow or Flyte end, the concerning event would be propagated to the other platform, and the execution would be canceled. If the execution fails at the Flyte end, the execution at the Airflow end would fail too.</p><p>If you’d like to know a practical use case of this provider, watch <a href="https://www.youtube.com/watch?v=F2JyA0O2U4M">this video</a>. It demonstrates an example of pulling NYC taxi data from S3, uploading it to CrateDB, and building an XGBoost model.</p><p>Airflow is a powerful tool to run ETL pipelines; however, Airflow needs to be extended to run machine learning pipelines.</p><p>With Flyte, you can version control your code, audit the data, reproduce executions, cache the outputs, and insert checkpoints without dwelling on the scalability of your machine learning pipelines. Give Flyte Airflow Provider a try, and let me know what you think of it!</p><p><em>Originally published at </em><a href="https://blog.flyte.org/scale-airflow-for-machine-learning-tasks-with-the-flyte-airflow-provider"><em>https://blog.flyte.org</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bead4eb6207d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/is-airflow-the-right-choice-for-machine-learning-too-bead4eb6207d">Is Airflow the Right Choice for Machine Learning Too?</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps with Flyte: The Convergence of Workflows Between Machine Learning and Engineering]]></title>
            <link>https://medium.com/union-ai/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering-717e4f1375c9?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/717e4f1375c9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Thu, 26 May 2022 14:11:05 GMT</pubDate>
            <atom:updated>2022-06-09T17:55:35.725Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*PgZtt8kPvxQUInRw" /></figure><p>So your company is building jaw-dropping machine learning (ML) models that are performant and outputting the best results. The next task is to promote the models to production.</p><p>It seems easy. After all, the most complex phase — building ML models — yielded a fruitful result. And <em>shouldn’t</em> it be easy considering how well DevOps (development operations) has matured into a set of practices that can ship software in minutes?</p><p>At first glance, DevOps does seem like the go-to choice to push machine learning models to production. But in fact, there’s more to productionizing ML.</p><h3>DevOps</h3><p>We’ve been writing software and building applications for a very long time, and we’ve been thinking about its wider deployment for almost as long. DevOps is a structured way to scale and maintain software reliably. Typical DevOps practices include:</p><ul><li>Continuous Integration (CI): Frequently merging code changes</li><li>Continuous Delivery (CD): Continuous promotion from development to staging to production</li><li>Infrastructure as Code (IaC): Managing infrastructure through code</li><li>Monitoring and Logging: Constant monitoring of the deployment</li><li>Communication and Collaboration: Team collaboration, multi-tenancy</li><li>Microservices: Building an application as a set of small services</li></ul><p>You may think that if the aforementioned practices aid in the development and deployment of software, they assuredly have to help with the deployment of machine learning models. ML is <em>code</em> after all!</p><p>However, it isn’t true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*vWHvVrtR4UZwLHjsD_GJZg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*JOOezQP9wOCZK6OW" /><figcaption>ML = model + data + code + ?</figcaption></figure><p>Unlike software engineering, ML development has multiple branches to deal with. Although the typical DevOps practices are helpful to deploy the models, they just <em>aren’t enough</em>.</p><h3>MLOps Is a Superset of DevOps</h3><blockquote><em>Build models like you build software with a hefty dose of ML-native practices.</em></blockquote><p>MLOps is a combination of machine learning, DevOps, and data engineering. Here’s how MLOps differs from DevOps:</p><ul><li><strong>Models can deteriorate over time. </strong>Because models and data are mutable, they require constant iteration and experimentation. Software development isn’t as variable and nondeterministic as ML models. Ensuring ML models climb the performance curve requires additional effort.</li><li><strong>Data may be very complex. </strong>In ML, data can be huge. It may need to be pulled from multiple sources.</li><li><strong>Infrastructure overhead &amp; hardware requirements are significant. </strong>Models are tied to higher processing and compute power. More often than not, ML models need to run on GPUs, and they rely on more memory than software development does.</li><li><strong>Scale of operations. </strong>Scalability of models is crucial for handling the ever-mutating data.</li></ul><p>When running models at scale, we’ll also have to take care of:</p><ul><li>Retraining the models on a cadence</li><li>Versioning the data and code</li><li>Data quality</li><li>Monitoring and logging</li><li>Adding tests to CI</li></ul><h4>Requirements for ML Pipelines</h4><p>MLOps at large needs to ensure:</p><ul><li><strong>Reproducibility. </strong>Determinism of a run. We should be able to reproduce a run given the same set of inputs; this facilitates debugging.</li><li><strong>Recoverability. </strong>If there’s a pipeline failure, we should be able to recover from the failure.</li><li><strong>Reliability. </strong>A holistic approach to ensure consistency of a pipeline or system.</li><li><strong>Auditability. </strong>Ability to backtrack to the error source when a pipeline fails. This in turn leads to <em>data lineage</em> — understanding how the data is transformed over time.</li><li><strong>Adaptability. </strong>Dynamism of an ML system. How convenient it is to switch among various ML models within a pipeline.</li><li><strong>Maintainability. </strong>A holistic approach to ensure maintenance of ML pipelines, which includes support for iterative development, team collaboration, automation, and more.</li></ul><p>MLOps is still a challenge: <em>There’s a wide gap between machine learning and engineering</em>. Deployment of models is a hard nut to crack, primarily because of backend leakage into the data world that data scientists are expected to wrap their heads around.</p><p>On the surface, it may seem like it’s the job of data scientists to handle all the MLOps challenges. However, it is unreasonable to expect them to know about low-level infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/0*Jp2E0PbpII36kpuY" /><figcaption>Let’s plug the leak!</figcaption></figure><p>We ought to separate the concerns of the platform/infrastructure team and the user team.</p><h3>Meet Flyte!</h3><p><a href="https://flyte.org/">Flyte</a> is a data- and machine learning-aware workflow automation tool built on top of Kubernetes. The goal of Flyte is to provide a <em>reproducible</em>, <em>incremental</em>, <em>iterative</em>, and <em>extensible</em> workflow automation platform-as-a-service (PaaS) for any organization.</p><p>The following are the features of Flyte that aim to solve MLOps challenges by converging the workflows between machine learning and engineering:</p><p><em>Note: SDKs for Flytekit (the tool with which you can write Flyte-compatible code) are available in Python, Java, and Scala. This article quotes code snippets that use the Python SDK.</em></p><p>1/<strong> Separation of concerns between platform and user teams.</strong> The immediate MLOps challenge is to incorporate an <em>infrastructure abstraction</em> and <em>workflow automation</em> platform that can <em>separate</em> the responsibilities between the platform and user teams. At the same time, it must <em>unify</em> the teams along the lines of a single platform.</p><p>Flyte is adept at enabling segregation of concerns and collaboration of teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cO6YUWRIRFKmJxCn" /><figcaption>Segregation of ML and Ops with Flyte</figcaption></figure><p>2/<strong> Develop incrementally and constantly iterate. </strong>An ML model needs to be retrained on a cadence (or sometimes, updated) because data is variable; this requires continuous integration and delivery in place.</p><p>Flyte supports the incremental and iterative development of code locally and on production setups (with the same ease as running locally). The promotion of code from local to staging to production can be done with ease using a CI/CD setup, such as GitHub Actions. Flyte also ships with a <a href="https://docs.flyte.org/en/latest/concepts/component_architecture/native_scheduler_architecture.html#native-scheduler-architecture">native scheduler</a> that helps run pipelines regularly and ensure they don’t go stale.</p><p>3/ <strong>Infrastructure as Code (IaC). </strong>ML models heavily rely on GPUs, and they may need to be scaled to multiple GPUs if data is huge or training is compute-intensive.</p><p>Flyte provides a declarative infrastructure-as-code (IaC) approach to access resources such as CPUs, GPUs, and memory.</p><pre><strong>@task(limits=Resources(cpu=&quot;2&quot;, mem=&quot;150Mi&quot;))<br></strong>def pay_multiplier(df: pandas.DataFrame, scalar: int) -&gt; pandas.DataFrame:<br>    df[&quot;col&quot;] = 2 * df[&quot;col&quot;]<br>    return df</pre><p><em>A </em><a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html"><em>@task</em></a><em> is a fully-independent unit of execution and first-class entity of Flyte. The </em><a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.Resources.html"><em>Resources</em></a><em> class accepts “cpu”, “gpu”, “mem”, “storage”, and “ephemeral_storage”.</em></p><p>4/<strong> Caching, retry, and recovery. </strong>Executing ML code requires greater processing power than a typical software application. If your feature engineering component succeeds and the training component fails, you do not want to burn out your resources by running the code right from scratch.</p><p>Flyte ships with three interesting features to handle the re-executions, namely, <em>caching</em>, <em>retry</em>, and <em>recovery</em>.</p><p><em>Caching</em> skips re-execution of code if the output of a task is cached, regardless of who ran it in the past. <em>Retry</em> triggers the execution again if the preceding execution fails. The <em>recover</em> option reruns an execution starting from the failed node by copying all the successful task executions.</p><pre><strong>@task(cache=True, retries=2)</strong><br>def pay_multiplier(df: pandas.DataFrame, scalar: int) -&gt; pandas.DataFrame:<br>    df[&quot;col&quot;] = 2 * df[&quot;col&quot;]<br>    return df</pre><p><em>“cache” can be set to True to cache the output of a @task. The number of retries is an integer.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/0*p5f2oXZnzAUGFMCz" /><figcaption>Caching: after the successful caching of a task output, when the execution is run again with the previously used input, the cached output is retrieved.</figcaption></figure><p>5/<strong> Multi-tenancy. </strong>Models may need to be used or worked upon by several teams within an organization. A single ML pipeline can comprise several components, each assigned to a team. The team members should be able to <strong>share</strong> their code and <strong>collaborate</strong> with other teams when the need arises.</p><p>As a Kubernetes-native platform, Flyte is multi-tenant from the ground up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/0*DwDAUZzGUcfda0Nd" /><figcaption>Team A and Team B work on projects titled “experimentation” and “training.” Since ML training relies on picking the best model from the experimentation phase, Team B has to be able to fetch the desired code from Team A, and there has to also be a separation of concerns between the two teams. Both the separation and collaboration are supported by Flyte.</figcaption></figure><p>6/<strong> Versioning</strong>. Versioning is crucial to reproduce ML models and data. Since models and data are mutable, maintaining separate versions is important to enable a switch between the versions and choose the best-performing model.</p><p>In Flyte, the entire <a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.workflow.html#flytekit-workflow">@workflow</a> (a Flyte entity that captures the dependencies between tasks) is versioned and all the tasks and workflows are immutable.</p><p>7/<strong> Parallelism. </strong>Running ML code requires more time than a typical software engineering job. Hence, it’s essential to run the code in parallel whenever possible.</p><p>Flyte determines if two <a href="https://docs.flyte.org/en/latest/concepts/nodes.html">nodes</a> can be run in parallel, meaning less wait time for users to get their results.</p><p>8/<strong> Compile-time validation. </strong>Rather than catching the errors at run time, it’s beneficial to capture the errors at compile time because ML is compute- and time-expensive.</p><p>Flyte’s <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/deploying_workflows.html#serialize-your-workflows-and-tasks">serialization</a> and <a href="https://docs.flyte.org/en/latest/concepts/registration.html#divedeep-registration">registration</a> process have many validations and tests like type enforcement available to users natively.</p><p>9/<strong> Data lineage. </strong>When an error pops up, there has to be a mechanism to backtrack to the error source. Moreover, the user should be able to understand the intermediate data (inputs and outputs) at every point in time.</p><p>Flyte is a <strong>data-aware </strong>platform. It understands data in all respects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bvTbMT-1S-E19YL_" /><figcaption>Inputs and outputs on Flyte UI can be viewed for every task.</figcaption></figure><p>10/<strong> BONUS: Intra-task checkpointing. </strong>Checkpointing is an important feature to enable when training ML models because training is expensive. Checkpoints can be helpful: Storing snapshots of the model until the current state allows you to continue subsequent executions from the failed state.</p><p><a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/checkpoint.html#sphx-glr-auto-core-control-flow-checkpoint-py">Flyte provides an intra-task checkpointing feature</a> that can be leveraged from within <em>@task</em>. This feature supports the use of <em>AWS spot instances</em> or <em>GCP preemptible instances</em> to lower the costs.</p><h3>How to Start Adopting Flyte for Your Machine Learning Pipelines</h3><p>Your company could be using a customized setup to handle ML workflows, or it might be using other orchestration tools. It’s time for you to make the switch to the right workflow orchestration service that caters to all your needs.</p><p>Here’s a chronological overview of how you could port your existing ML pipelines to Flyte, which involves the collaboration between user and platform teams to create a consolidated solution. This isn’t a one-size-fits-all solution, but it does include some best practices.</p><p><em>Let’s assume</em></p><p><em>User team = Data scientists, ML/Data engineers</em></p><p><em>Platform team = Infrastructure engineers, DevOps engineers</em></p><h4><strong>Write an ML job</strong></h4><p><em>Team: User</em></p><p>The first step to getting started with building an ML pipeline is to write a job (or @task as per Flyte terminology). The job could be a training job, data preprocessing job, or spark job. If the user team has the code to implement it, they’ll have to convert it to Flyte-compatible code to be able to run it on the Flyte backend. (This requires <em>little</em> refactoring.) If the task interacts with a tool or an external service, the user team can take a look at the <a href="https://docs.flyte.org/projects/cookbook/en/latest/integrations.html">supported plugins</a> before implementing the code from scratch.</p><h4><strong>Run it locally</strong></h4><p><em>Team: User</em></p><p>Despite writing the code as per Flyte DSL (domain-specific language), it’s important that the user team can run the code locally. Flyte supports local executions out of the box. After the team constructs Flyte-compatible workflows, it can run the code locally just like how one would run a Python file.</p><h4><strong>Scale the job</strong></h4><p><em>Team: User</em></p><p>Depending on how compute-intensive the tasks are, the user team may need to tune resources to run them on a production setup. Since Flyte provides a declarative IaC approach, the team can use knobs to set the required <em>CPU</em>, <em>GPU</em>, and <em>mem</em>.</p><h4><strong>Create a pipeline</strong></h4><p><em>Team: User</em></p><p>Stitch all the jobs/tasks together to create a pipeline/workflow.</p><h4><strong>Test the pipeline locally</strong></h4><p><em>Team: User</em></p><p>Run the code locally to verify the output.</p><h4><strong>Set up the Flyte local environment</strong></h4><p><em>Team: User</em></p><p>Before the user team tests the code in production, it’s useful to test it on a bare-bones Flyte setup. The <a href="https://docs.flyte.org/en/latest/getting_started/index.html#creating-a-demo-flyte-cluster">Getting Started guide</a> should help the team set up a Kubernetes cluster on their local machines and run the workflows on the Flyte backend. “Kubernetes” might intimidate data scientists; nonetheless, given that Docker, an OS-level virtualization platform, is installed, it should take <em>less than three minutes</em> to get the Flyte workflows up and running on the backend.</p><h4><strong>Set up Flyte</strong></h4><p><em>Team: Platform</em></p><p>Setting up Flyte in production involves configuring the database, object store, authentication, and so on. This can be done on an on-prem setup or in the cloud — be it AWS, GCP, Azure, or any cloud service. The <a href="https://docs.flyte.org/en/latest/deployment/index.html">deployment guide</a> in the Flyte documentation should help with setting up a production-ready Flyte backend with a centrally managed infrastructure. The platform team interacts with Flyte directly to ensure that the production environment is functioning properly, and the user team works on the data and models.</p><h4><strong>Set up DevOps</strong></h4><p><em>Team: Platform</em></p><p>To automate the process of packaging code and registering it on the Flyte backend to run the pipeline at scale, it’s important to implement some DevOps practices.</p><p>After the user team builds an ML pipeline, the code needs to be packaged and registered on the Flyte backend. This can turn out to be a complex process if it needs to be done for every code or dependency change. Here’s how <strong>Lyft</strong> roughly handles <em>1+ million</em> Flyte workflows:</p><ul><li>Every time a new pull request (PR) is created by the user team, a Docker container that captures the dependencies is built from the PR, and the tasks and workflows are registered that will use the Docker image when they run.</li><li>If dependencies are modified by the user team — say, when a new library is to be installed — PR securely builds the container again.</li><li>Straight code changes use fast registration: The code is registered on the Flyte backend without building any Docker image.</li><li>The user team can run the registered tasks and workflows on the Flyte backend (let’s assume the <a href="https://docs.flyte.org/en/latest/concepts/domains.html">domain</a> is <em>development </em>when testing through a PR).</li><li>Once a PR is merged to the master/main branch, tasks and workflows are promoted to the <em>production</em> domain. The user team can then simulate the deployment through a deployment pipeline.</li><li>At each stage of the deployment, tasks and workflows are registered in Flyte with a specific domain (such as development, staging, and production).</li><li>Logs and metrics are automatically tracked for production deployments.</li></ul><p><strong><em>How to version Flyte tasks and workflows during the registration process? </em></strong><em>Tasks can be versioned using the GIT-SHA or hash of the entire code artifact. Workflows can be versioned using the GIT-SHA of the containing repository.</em></p><h4><strong>Execute the pipeline on-demand</strong></h4><p><em>Team: User</em></p><p>After the automation is in place, the user team can test and deploy the ML pipelines with ease.</p><h4><strong>Schedule workflows</strong></h4><p><em>Team: User</em></p><p><a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/core/scheduled_workflows/lp_schedules.html#sphx-glr-auto-core-scheduled-workflows-lp-schedules-py">Schedules can be created from within the code</a> using <a href="https://docs.flyte.org/en/latest/concepts/launchplans.html">LaunchPlan</a> — a plan to execute the workflows.</p><h4><strong>Retrieve results</strong></h4><p><em>Team: User</em></p><p>The user team can retrieve the pipeline’s intermediate or final outputs from a Jupyter notebook using <a href="https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#design-control-plane">FlyteRemote</a> or from the CLI using <a href="https://docs.flyte.org/projects/flytectl/en/latest/">Flytectl</a>, or view them via the UI.</p><h4><strong>Maintenance</strong></h4><p><em>Team: Platform</em></p><p>Maintenance includes periodically upgrading Flyte components and ensuring that the Flyte backend is operational.</p><p>All the processes together portray a clear picture of how MLOps can be implemented with ease through Flyte.</p><p>The convergence of workflows between machine learning and engineering is a longstanding problem that we need to address now. We believe Flyte bridges the gap between the two and adds a lot more ML flavor to the engineering discipline. Flyte aims to unify data scientists, ML engineering, and platform engineers on a single platform by erecting strong interface boundaries.</p><p>If Flyte interests you, feel free to check out our <a href="https://github.com/flyteorg/flyte">GitHub</a>, explore our <a href="https://docs.flyte.org/">documentation</a>, and join our <a href="https://github.com/flyteorg/flyte">Slack</a>!</p><p><em>Originally published at </em><a href="https://mlops.community/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering/"><em>https://mlops.community</em></a><em> on May 26, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=717e4f1375c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/union-ai/mlops-with-flyte-the-convergence-of-workflows-between-machine-learning-and-engineering-717e4f1375c9">MLOps with Flyte: The Convergence of Workflows Between Machine Learning and Engineering</a> was originally published in <a href="https://medium.com/union-ai">Union-ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build an Event-Driven Neural Style Transfer Application Using AWS Lambda]]></title>
            <link>https://medium.com/data-science/build-an-event-driven-neural-style-transfer-application-using-aws-lambda-18fa8145ef5b?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/18fa8145ef5b</guid>
            <category><![CDATA[workflow-automation]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Wed, 11 May 2022 18:50:36 GMT</pubDate>
            <atom:updated>2022-05-11T18:50:36.239Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KeLhxdf0WCUYshIyi2gfYw.png" /><figcaption>A neural style transfer image (on the right) is generated after 5 epochs and 100 steps per epoch (Image by Author)</figcaption></figure><p>To build a production-ready ML application and ensure its stability in the long run, we need to take care of a long checklist of requirements which include the ease with which the models could be iterated, reproducibility, infrastructure, automation, resources, memory, and so on. On top of that, we need a <em>seamless developer experience</em>. How hard could it be?</p><p><a href="https://github.com/flyteorg/flyte">Flyte</a> can handle the former set of issues because:</p><ul><li>it’s a workflow automation platform that helps maintain and reproduce pipelines.</li><li>it provides the control knobs for infrastructure, resources, and memory.</li></ul><p>Also, Flyte simplifies the developer experience. In this blog post, we’ll see how by building a neural style transfer application using Flyte and AWS Lambda. We’ll code the end-to-end pipeline and assign the required compute to run the code. Further, we’ll design an event-driven mechanism to trigger the pipeline and output a stylized image when a user uploads an image. <em>From a user perspective, a stylized output image has to be generated upon uploading an image.</em></p><p>Since the application has to be triggered on an event, i.e., image upload, a more suitable choice for Flyte would be to use AWS Lambda. It is serverless and an event-driven compute service. Our neural style transfer application will leverage the “event-driven feature” of AWS Lambda.</p><p>Let’s look at how we could stitch the pipeline automation and event-driven service together using Flyte and AWS Lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMa64HiiEbe2afrmfU1B8A.png" /><figcaption><em>Figure 1. An overview of the application (Image by Author)</em></figcaption></figure><h3>Application Code</h3><p>Neural style transfer is applying the style of the style image onto the content image. The output image would be a blend of the content and style images.</p><p>To get started with the code, first import and configure the dependencies.</p><p><em>Note: This code is an adaption of the </em><a href="https://www.tensorflow.org/tutorials/generative/style_transfer"><em>Neural style transfer example from the TensorFlow documentation</em></a><em>. To run the code, ensure </em><em>tensorflow, </em><em>flytekit, and </em><em>Pillow libraries are installed through </em><em>pip.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D728%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520os%25250Afrom%252520typing%252520import%252520Tuple%25250A%25250Aimport%252520flytekit%25250Aimport%252520numpy%252520as%252520np%25250Aimport%252520PIL.Image%25250Aimport%252520tensorflow%252520as%252520tf%25250Afrom%252520flytekit%252520import%252520Resources%25252C%252520dynamic%25252C%252520task%25252C%252520workflow%25250Afrom%252520flytekit.types.file%252520import%252520FlyteFile%25250A%25250Astyle_weight%252520%25253D%2525201e-2%25250Acontent_weight%252520%25253D%2525201e4%25250Atotal_variation_weight%252520%25253D%25252030%25250A%25250Acontent_layers%252520%25253D%252520%25255B%252522block5_conv2%252522%25255D%25250Astyle_layers%252520%25253D%252520%25255B%25250A%252520%252520%252520%252522block1_conv1%252522%25252C%25250A%252520%252520%252520%252522block2_conv1%252522%25252C%25250A%252520%252520%252520%252522block3_conv1%252522%25252C%25250A%252520%252520%252520%252522block4_conv1%252522%25252C%25250A%252520%252520%252520%252522block5_conv1%252522%25252C%25250A%25255D%25250AIMAGE_URL%252520%25253D%252520%252522https%25253A%25252F%25252Fstorage.googleapis.com%25252Fdownload.tensorflow.org%25252Fexample_images%25252FYellowLabradorLooking_new.jpg%252522%25250A%25250Arequest_resources%252520%25253D%252520Resources%2528cpu%25253D%2525221%252522%25252C%252520mem%25253D%252522500Mi%252522%25252C%252520storage%25253D%252522500Mi%252522%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D728%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520os%2525250Afrom%25252520typing%25252520import%25252520Tuple%2525250A%2525250Aimport%25252520flytekit%2525250Aimport%25252520numpy%25252520as%25252520np%2525250Aimport%25252520PIL.Image%2525250Aimport%25252520tensorflow%25252520as%25252520tf%2525250Afrom%25252520flytekit%25252520import%25252520Resources%2525252C%25252520dynamic%2525252C%25252520task%2525252C%25252520workflow%2525250Afrom%25252520flytekit.types.file%25252520import%25252520FlyteFile%2525250A%2525250Astyle_weight%25252520%2525253D%252525201e-2%2525250Acontent_weight%25252520%2525253D%252525201e4%2525250Atotal_variation_weight%25252520%2525253D%2525252030%2525250A%2525250Acontent_layers%25252520%2525253D%25252520%2525255B%25252522block5_conv2%25252522%2525255D%2525250Astyle_layers%25252520%2525253D%25252520%2525255B%2525250A%25252520%25252520%25252520%25252522block1_conv1%25252522%2525252C%2525250A%25252520%25252520%25252520%25252522block2_conv1%25252522%2525252C%2525250A%25252520%25252520%25252520%25252522block3_conv1%25252522%2525252C%2525250A%25252520%25252520%25252520%25252522block4_conv1%25252522%2525252C%2525250A%25252520%25252520%25252520%25252522block5_conv1%25252522%2525252C%2525250A%2525255D%2525250AIMAGE_URL%25252520%2525253D%25252520%25252522https%2525253A%2525252F%2525252Fstorage.googleapis.com%2525252Fdownload.tensorflow.org%2525252Fexample_images%2525252FYellowLabradorLooking_new.jpg%25252522%2525250A%2525250Arequest_resources%25252520%2525253D%25252520Resources%252528cpu%2525253D%252525221%25252522%2525252C%25252520mem%2525253D%25252522500Mi%25252522%2525252C%25252520storage%2525253D%25252522500Mi%25252522%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e7c32cf28a6cfca2a65b5942bff1dda/href">https://medium.com/media/5e7c32cf28a6cfca2a65b5942bff1dda/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D728%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25250A%252540task%2528requests%25253Drequest_resources%2529%25250Adef%252520tensor_to_image%2528tensor%25253A%252520tf.Variable%2529%252520-%25253E%252520FlyteFile%25253A%25250A%252520%252520%252520%252520tensor%252520%25253D%252520tensor%252520%252A%252520255%25250A%252520%252520%252520%252520tensor%252520%25253D%252520np.array%2528tensor%25252C%252520dtype%25253Dnp.uint8%2529%25250A%252520%252520%252520%252520if%252520np.ndim%2528tensor%2529%252520%25253E%2525203%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520assert%252520tensor.shape%25255B0%25255D%252520%25253D%25253D%2525201%25250A%252520%252520%252520%252520%252520%252520%252520%252520tensor%252520%25253D%252520tensor%25255B0%25255D%25250A%252520%252520%252520%252520working_dir%252520%25253D%252520flytekit.current_context%2528%2529.working_directory%25250A%252520%252520%252520%252520image_path%252520%25253D%252520os.path.join%2528working_dir%25252C%252520%252522neural_style_transferred_img.png%252522%2529%25250A%252520%252520%252520%252520image%252520%25253D%252520PIL.Image.fromarray%2528tensor%2529%25250A%252520%252520%252520%252520image.save%2528image_path%2529%25250A%252520%252520%252520%252520return%252520image_path&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D728%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%2525250A%25252540task%252528requests%2525253Drequest_resources%252529%2525250Adef%25252520tensor_to_image%252528tensor%2525253A%25252520tf.Variable%252529%25252520-%2525253E%25252520FlyteFile%2525253A%2525250A%25252520%25252520%25252520%25252520tensor%25252520%2525253D%25252520tensor%25252520%25252A%25252520255%2525250A%25252520%25252520%25252520%25252520tensor%25252520%2525253D%25252520np.array%252528tensor%2525252C%25252520dtype%2525253Dnp.uint8%252529%2525250A%25252520%25252520%25252520%25252520if%25252520np.ndim%252528tensor%252529%25252520%2525253E%252525203%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520assert%25252520tensor.shape%2525255B0%2525255D%25252520%2525253D%2525253D%252525201%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520tensor%25252520%2525253D%25252520tensor%2525255B0%2525255D%2525250A%25252520%25252520%25252520%25252520working_dir%25252520%2525253D%25252520flytekit.current_context%252528%252529.working_directory%2525250A%25252520%25252520%25252520%25252520image_path%25252520%2525253D%25252520os.path.join%252528working_dir%2525252C%25252520%25252522neural_style_transferred_img.png%25252522%252529%2525250A%25252520%25252520%25252520%25252520image%25252520%2525253D%25252520PIL.Image.fromarray%252528tensor%252529%2525250A%25252520%25252520%25252520%25252520image.save%252528image_path%252529%2525250A%25252520%25252520%25252520%25252520return%25252520image_path&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3e902da13948b757f72653ca5fbad9f/href">https://medium.com/media/c3e902da13948b757f72653ca5fbad9f/href</a></iframe><p>content_layers and style_layers are the layers of the VGG19 model, which we’ll use to build our model, and the tensor_to_image task converts a tensor to an image.</p><p>The first step of the model building process is to fetch the image and preprocess it. Define a <a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.task.html">@task</a> to load the image and limit its maximum dimension to 512 pixels.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D502%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%252520load_img%2528path_to_img%2529%25253A%25250A%252520%252520%252520%252520max_dim%252520%25253D%252520512%25250A%252520%252520%252520%252520img%252520%25253D%252520tf.io.read_file%2528path_to_img%2529%25250A%252520%252520%252520%252520img%252520%25253D%252520tf.image.decode_image%2528img%25252C%252520channels%25253D3%2529%25250A%252520%252520%252520%252520img%252520%25253D%252520tf.image.convert_image_dtype%2528img%25252C%252520tf.float32%2529%25250A%25250A%252520%252520%252520%252520shape%252520%25253D%252520tf.cast%2528tf.shape%2528img%2529%25255B%25253A-1%25255D%25252C%252520tf.float32%2529%25250A%252520%252520%252520%252520long_dim%252520%25253D%252520max%2528shape%2529%25250A%252520%252520%252520%252520scale%252520%25253D%252520max_dim%252520%25252F%252520long_dim%25250A%25250A%252520%252520%252520%252520new_shape%252520%25253D%252520tf.cast%2528shape%252520%252A%252520scale%25252C%252520tf.int32%2529%25250A%25250A%252520%252520%252520%252520img%252520%25253D%252520tf.image.resize%2528img%25252C%252520new_shape%2529%25250A%252520%252520%252520%252520img%252520%25253D%252520img%25255Btf.newaxis%25252C%252520%25253A%25255D%25250A%252520%252520%252520%252520return%252520img&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D502%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%25252520load_img%252528path_to_img%252529%2525253A%2525250A%25252520%25252520%25252520%25252520max_dim%25252520%2525253D%25252520512%2525250A%25252520%25252520%25252520%25252520img%25252520%2525253D%25252520tf.io.read_file%252528path_to_img%252529%2525250A%25252520%25252520%25252520%25252520img%25252520%2525253D%25252520tf.image.decode_image%252528img%2525252C%25252520channels%2525253D3%252529%2525250A%25252520%25252520%25252520%25252520img%25252520%2525253D%25252520tf.image.convert_image_dtype%252528img%2525252C%25252520tf.float32%252529%2525250A%2525250A%25252520%25252520%25252520%25252520shape%25252520%2525253D%25252520tf.cast%252528tf.shape%252528img%252529%2525255B%2525253A-1%2525255D%2525252C%25252520tf.float32%252529%2525250A%25252520%25252520%25252520%25252520long_dim%25252520%2525253D%25252520max%252528shape%252529%2525250A%25252520%25252520%25252520%25252520scale%25252520%2525253D%25252520max_dim%25252520%2525252F%25252520long_dim%2525250A%2525250A%25252520%25252520%25252520%25252520new_shape%25252520%2525253D%25252520tf.cast%252528shape%25252520%25252A%25252520scale%2525252C%25252520tf.int32%252529%2525250A%2525250A%25252520%25252520%25252520%25252520img%25252520%2525253D%25252520tf.image.resize%252528img%2525252C%25252520new_shape%252529%2525250A%25252520%25252520%25252520%25252520img%25252520%2525253D%25252520img%2525255Btf.newaxis%2525252C%25252520%2525253A%2525255D%2525250A%25252520%25252520%25252520%25252520return%25252520img&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6ce47edb19ed1d531fbb4b4caf64f59/href">https://medium.com/media/a6ce47edb19ed1d531fbb4b4caf64f59/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D502%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252540task%2528requests%25253Drequest_resources%2529%25250Adef%252520preprocess_img%2528%25250A%252520%252520%252520content_img%25253A%252520FlyteFile%25252C%252520style_img%25253A%252520FlyteFile%25250A%2529%252520-%25253E%252520Tuple%25255Btf.Tensor%25252C%252520tf.Tensor%25255D%25253A%25250A%252520%252520%252520%252520content_path%252520%25253D%252520content_img.download%2528%2529%25250A%252520%252520%252520%252520style_path%252520%25253D%252520style_img.download%2528%2529%25250A%25250A%252520%252520%252520%252520content_image%252520%25253D%252520load_img%2528content_path%2529%25250A%252520%252520%252520%252520style_image%252520%25253D%252520load_img%2528style_path%2529%25250A%25250A%252520%252520%252520%252520return%252520content_image%25252C%252520style_image&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D502%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252540task%252528requests%2525253Drequest_resources%252529%2525250Adef%25252520preprocess_img%252528%2525250A%25252520%25252520%25252520content_img%2525253A%25252520FlyteFile%2525252C%25252520style_img%2525253A%25252520FlyteFile%2525250A%252529%25252520-%2525253E%25252520Tuple%2525255Btf.Tensor%2525252C%25252520tf.Tensor%2525255D%2525253A%2525250A%25252520%25252520%25252520%25252520content_path%25252520%2525253D%25252520content_img.download%252528%252529%2525250A%25252520%25252520%25252520%25252520style_path%25252520%2525253D%25252520style_img.download%252528%252529%2525250A%2525250A%25252520%25252520%25252520%25252520content_image%25252520%2525253D%25252520load_img%252528content_path%252529%2525250A%25252520%25252520%25252520%25252520style_image%25252520%2525253D%25252520load_img%252528style_path%252529%2525250A%2525250A%25252520%25252520%25252520%25252520return%25252520content_image%2525252C%25252520style_image&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/408c79e7767b624a6a20921d426f3820/href">https://medium.com/media/408c79e7767b624a6a20921d426f3820/href</a></iframe><p>The preprocess_img task downloads the content and style image files, and resizes them using the load_img function.</p><p>With the data ready to be used by the model, define a VGG19 model that returns the style and content tensors.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D682%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%252520vgg_layers%2528layer_names%2529%25253A%25250A%252520%252520%252520%252520%252522%252522%252522Creates%252520a%252520vgg%252520model%252520that%252520returns%252520a%252520list%252520of%252520intermediate%252520output%252520values.%252522%252522%252522%25250A%252520%252520%252520%252520%252523%252520Load%252520our%252520model.%252520Load%252520pretrained%252520VGG%25252C%252520trained%252520on%252520imagenet%252520data%25250A%252520%252520%252520%252520vgg%252520%25253D%252520tf.keras.applications.VGG19%2528include_top%25253DFalse%25252C%252520weights%25253D%252522imagenet%252522%2529%25250A%252520%252520%252520%252520vgg.trainable%252520%25253D%252520False%25250A%25250A%252520%252520%252520%252520outputs%252520%25253D%252520%25255Bvgg.get_layer%2528name%2529.output%252520for%252520name%252520in%252520layer_names%25255D%25250A%25250A%252520%252520%252520%252520model%252520%25253D%252520tf.keras.Model%2528%25255Bvgg.input%25255D%25252C%252520outputs%2529%25250A%252520%252520%252520%252520return%252520model%25250A%25250A%25250Adef%252520gram_matrix%2528input_tensor%2529%25253A%25250A%252520%252520%252520%252520result%252520%25253D%252520tf.linalg.einsum%2528%252522bijc%25252Cbijd-%25253Ebcd%252522%25252C%252520input_tensor%25252C%252520input_tensor%2529%25250A%252520%252520%252520%252520input_shape%252520%25253D%252520tf.shape%2528input_tensor%2529%25250A%252520%252520%252520%252520num_locations%252520%25253D%252520tf.cast%2528input_shape%25255B1%25255D%252520%252A%252520input_shape%25255B2%25255D%25252C%252520tf.float32%2529%25250A%252520%252520%252520%252520return%252520result%252520%25252F%252520%2528num_locations%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D682%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Ddef%25252520vgg_layers%252528layer_names%252529%2525253A%2525250A%25252520%25252520%25252520%25252520%25252522%25252522%25252522Creates%25252520a%25252520vgg%25252520model%25252520that%25252520returns%25252520a%25252520list%25252520of%25252520intermediate%25252520output%25252520values.%25252522%25252522%25252522%2525250A%25252520%25252520%25252520%25252520%25252523%25252520Load%25252520our%25252520model.%25252520Load%25252520pretrained%25252520VGG%2525252C%25252520trained%25252520on%25252520imagenet%25252520data%2525250A%25252520%25252520%25252520%25252520vgg%25252520%2525253D%25252520tf.keras.applications.VGG19%252528include_top%2525253DFalse%2525252C%25252520weights%2525253D%25252522imagenet%25252522%252529%2525250A%25252520%25252520%25252520%25252520vgg.trainable%25252520%2525253D%25252520False%2525250A%2525250A%25252520%25252520%25252520%25252520outputs%25252520%2525253D%25252520%2525255Bvgg.get_layer%252528name%252529.output%25252520for%25252520name%25252520in%25252520layer_names%2525255D%2525250A%2525250A%25252520%25252520%25252520%25252520model%25252520%2525253D%25252520tf.keras.Model%252528%2525255Bvgg.input%2525255D%2525252C%25252520outputs%252529%2525250A%25252520%25252520%25252520%25252520return%25252520model%2525250A%2525250A%2525250Adef%25252520gram_matrix%252528input_tensor%252529%2525253A%2525250A%25252520%25252520%25252520%25252520result%25252520%2525253D%25252520tf.linalg.einsum%252528%25252522bijc%2525252Cbijd-%2525253Ebcd%25252522%2525252C%25252520input_tensor%2525252C%25252520input_tensor%252529%2525250A%25252520%25252520%25252520%25252520input_shape%25252520%2525253D%25252520tf.shape%252528input_tensor%252529%2525250A%25252520%25252520%25252520%25252520num_locations%25252520%2525253D%25252520tf.cast%252528input_shape%2525255B1%2525255D%25252520%25252A%25252520input_shape%2525255B2%2525255D%2525252C%25252520tf.float32%252529%2525250A%25252520%25252520%25252520%25252520return%25252520result%25252520%2525252F%25252520%252528num_locations%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0baa10d9e08fbc94b0b5896aab8f4588/href">https://medium.com/media/0baa10d9e08fbc94b0b5896aab8f4588/href</a></iframe><pre>class <strong>StyleContentModel</strong>(tf.keras.models.Model):<br>    def <strong>__init__</strong>(self, style_layers, content_layers):<br>        super(StyleContentModel, self).__init__()<br>        self.vgg = vgg_layers(style_layers + content_layers)<br>        self.style_layers = style_layers<br>        self.content_layers = content_layers<br>        self.num_style_layers = len(style_layers)<br>        self.vgg.trainable = False</pre><pre>    def <strong>call</strong>(self, inputs):<br>        &quot;Expects float input in [0,1]&quot;<br>        inputs = inputs * 255.0<br>        preprocessed_input = tf.keras.applications.vgg19.preprocess_input(inputs)<br>        outputs = self.vgg(preprocessed_input)<br>        style_outputs, content_outputs = (<br>            outputs[: self.num_style_layers],<br>            outputs[self.num_style_layers :],<br>        )<br>        style_outputs = [gram_matrix(style_output) for style_output in style_outputs]<br>        <strong>content_dict</strong> = {<br>            content_name: value<br>            for content_name, value in zip(self.content_layers, content_outputs)<br>        }<br>        <strong>style_dict</strong> = {<br>            style_name: value<br>            for style_name, value in zip(self.style_layers,    style_outputs)<br>        }<br>        return {&quot;content&quot;: content_dict, &quot;style&quot;: style_dict}</pre><p>The vgg_layers function returns a list of intermediate layer outputs on top of which the model is built (note that we’re using a pretrained VGG network), and the gram_matrix function <em>literally</em> describes the style of an image. When the model is called on an image, it returns the gram matrix of the style_layers and the content of the content_layers.</p><p>Next comes the implementation of the style transfer algorithm. Calculate the total loss (style + content) by considering the weighted combination of the two losses.</p><pre>def <strong>style_content_loss</strong>(outputs, content_targets, style_targets):<br>    style_outputs = outputs[&quot;style&quot;]<br>    content_outputs = outputs[&quot;content&quot;]<br>    style_loss = tf.add_n(<br>        [<br>            tf.reduce_mean((style_outputs[name] - style_targets[name]) ** 2)<br>            for name in style_outputs.keys()<br>        ]<br>    )<br>    <strong>style_loss</strong> *= style_weight / len(style_layers)<br>    content_loss = tf.add_n(<br>        [<br>            tf.reduce_mean((content_outputs[name] - content_targets[name]) ** 2)<br>            for name in content_outputs.keys()<br>        ]<br>    )<br>    <strong>content_loss</strong> *= content_weight / len(content_layers)<br>    <strong>loss = style_loss + content_loss</strong><br>    return loss</pre><p>Call style_content_loss from within tf.GradientTape to update the image.</p><pre><a href="http://twitter.com/task"><strong>@task</strong></a><strong>(requests=Resources(cpu=&quot;1&quot;, mem=&quot;5Gi&quot;, storage=&quot;5Gi&quot;, ephemeral_storage=&quot;5Gi&quot;))</strong><br>def <strong>train_step</strong>(<br>    image: tf.Variable, content_image: tf.Tensor, style_image: tf.Tensor<br>) -&gt; tf.Variable:<br>    opt = tf.optimizers.Adam(learning_rate=0.02, beta_1=0.99, epsilon=1e-1)<br>    extractor = StyleContentModel(style_layers, content_layers)<br>    style_targets = extractor(style_image)[&quot;style&quot;]<br>    content_targets = extractor(content_image)[&quot;content&quot;]<br>    <strong>with tf.GradientTape() as tape:<br>        outputs = extractor(image)<br>        loss = style_content_loss(outputs, content_targets, style_targets)<br>        loss += total_variation_weight * tf.image.total_variation(image)</strong><br>    grad = tape.gradient(loss, image)<br>    opt.apply_gradients([(grad, image)])<br>    image.assign(clip_0_1(image))<br>    return image</pre><p>The train_step task initializes the style and content target values (tensors), computes the total variation loss, runs gradient descent, applies the processed gradients, and clips the pixel values of the image between 0 and 1. Define the clip_0_1 function as follows:</p><pre>def <strong>clip_0_1</strong>(image):<br>    return tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0)</pre><p>Create a <a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.dynamic.html#flytekit-dynamic">@dynamic</a> workflow to trigger the train_step task for a specified number of epochs and steps_per_epoch.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D708%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252540dynamic%2528requests%25253DResources%2528cpu%25253D%2525221%252522%25252C%252520mem%25253D%2525225Gi%252522%25252C%252520storage%25253D%2525225Gi%252522%25252C%252520ephemeral_storage%25253D%2525225Gi%252522%2529%2529%25250Adef%252520generate_image%2528%25250A%252520%252520%252520%252520content_image%25253A%252520tf.Tensor%25252C%252520style_image%25253A%252520tf.Tensor%25252C%252520epochs%25253A%252520int%25252C%252520steps_per_epoch%25253A%252520int%25250A%2529%252520-%25253E%252520FlyteFile%25253A%25250A%252520%252520%252520%252520image%252520%25253D%252520tf.Variable%2528content_image%2529%25250A%252520%252520%252520%252520step%252520%25253D%2525200%25250A%252520%252520%252520%252520for%252520n%252520in%252520range%2528epochs%2529%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520for%252520m%252520in%252520range%2528steps_per_epoch%2529%25253A%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520step%252520%25252B%25253D%2525201%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520image%252520%25253D%252520train_step%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520image%25253Dimage%25252C%252520content_image%25253Dcontent_image%25252C%252520style_image%25253Dstyle_image%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%2529%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520print%2528%252522.%252522%25252C%252520end%25253D%252522%252522%25252C%252520flush%25253DTrue%2529%25250A%252520%252520%252520%252520%252520%252520%252520%252520print%2528%252522Train%252520step%25253A%252520%25257B%25257D%252522.format%2528step%2529%2529%25250A%25250A%252520%252520%252520%252520return%252520tensor_to_image%2528tensor%25253Dimage%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D708%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252540dynamic%252528requests%2525253DResources%252528cpu%2525253D%252525221%25252522%2525252C%25252520mem%2525253D%252525225Gi%25252522%2525252C%25252520storage%2525253D%252525225Gi%25252522%2525252C%25252520ephemeral_storage%2525253D%252525225Gi%25252522%252529%252529%2525250Adef%25252520generate_image%252528%2525250A%25252520%25252520%25252520%25252520content_image%2525253A%25252520tf.Tensor%2525252C%25252520style_image%2525253A%25252520tf.Tensor%2525252C%25252520epochs%2525253A%25252520int%2525252C%25252520steps_per_epoch%2525253A%25252520int%2525250A%252529%25252520-%2525253E%25252520FlyteFile%2525253A%2525250A%25252520%25252520%25252520%25252520image%25252520%2525253D%25252520tf.Variable%252528content_image%252529%2525250A%25252520%25252520%25252520%25252520step%25252520%2525253D%252525200%2525250A%25252520%25252520%25252520%25252520for%25252520n%25252520in%25252520range%252528epochs%252529%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520for%25252520m%25252520in%25252520range%252528steps_per_epoch%252529%2525253A%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520step%25252520%2525252B%2525253D%252525201%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520image%25252520%2525253D%25252520train_step%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520image%2525253Dimage%2525252C%25252520content_image%2525253Dcontent_image%2525252C%25252520style_image%2525253Dstyle_image%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%252529%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520print%252528%25252522.%25252522%2525252C%25252520end%2525253D%25252522%25252522%2525252C%25252520flush%2525253DTrue%252529%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520print%252528%25252522Train%25252520step%2525253A%25252520%2525257B%2525257D%25252522.format%252528step%252529%252529%2525250A%2525250A%25252520%25252520%25252520%25252520return%25252520tensor_to_image%252528tensor%2525253Dimage%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/71bf6176044ec74affb8a6940e4bfbef/href">https://medium.com/media/71bf6176044ec74affb8a6940e4bfbef/href</a></iframe><p>tf.Variable stores the content image. When it is called from within tf.GradientTape, the image, a tf.Variable is watched and the operations are recorded for automatic differentiation.</p><p>Lastly, define a <a href="https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.workflow.html">@workflow</a> to encapsulate the tasks and generate a stylized image.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D708%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252540workflow%25250Adef%252520neural_style_transfer_wf%2528%25250A%252520%252520%252520%252520content_img%25253A%252520FlyteFile%252520%25253D%252520IMAGE_URL%25252C%25250A%252520%252520%252520%252520style_img%25253A%252520FlyteFile%252520%25253D%252520%252522https%25253A%25252F%25252Fstorage.googleapis.com%25252Fdownload.tensorflow.org%25252Fexample_images%25252FVassily_Kandinsky%2525252C_1913_-_Composition_7.jpg%252522%25252C%25250A%252520%252520%252520%252520epochs%25253A%252520int%252520%25253D%2525205%25252C%25250A%252520%252520%252520%252520steps_per_epoch%25253A%252520int%252520%25253D%252520100%25252C%25250A%2529%252520-%25253E%252520FlyteFile%25253A%25250A%252520%252520%252520%252520content_image%25252C%252520style_image%252520%25253D%252520preprocess_img%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520content_img%25253Dcontent_img%25252C%252520style_img%25253Dstyle_img%25250A%252520%252520%252520%252520%2529%25250A%252520%252520%252520%252520return%252520generate_image%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520content_image%25253Dcontent_image%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520style_image%25253Dstyle_image%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520epochs%25253Depochs%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520steps_per_epoch%25253Dsteps_per_epoch%25252C%25250A%252520%252520%252520%252520%2529%25250A%25250A%25250Aif%252520__name__%252520%25253D%25253D%252520%252522__main__%252522%25253A%25250A%252520%252520%252520%252520print%2528f%252522Running%252520%25257B__file__%25257D...%252522%2529%25250A%252520%252520%252520%252520print%2528f%252522Stylized%252520image%25253A%252520%25257Bneural_style_transfer_wf%2528%2529%25257D%252522%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dblackboard%26wt%3Dsharp%26l%3Dpython%26width%3D708%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252540workflow%2525250Adef%25252520neural_style_transfer_wf%252528%2525250A%25252520%25252520%25252520%25252520content_img%2525253A%25252520FlyteFile%25252520%2525253D%25252520IMAGE_URL%2525252C%2525250A%25252520%25252520%25252520%25252520style_img%2525253A%25252520FlyteFile%25252520%2525253D%25252520%25252522https%2525253A%2525252F%2525252Fstorage.googleapis.com%2525252Fdownload.tensorflow.org%2525252Fexample_images%2525252FVassily_Kandinsky%252525252C_1913_-_Composition_7.jpg%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520epochs%2525253A%25252520int%25252520%2525253D%252525205%2525252C%2525250A%25252520%25252520%25252520%25252520steps_per_epoch%2525253A%25252520int%25252520%2525253D%25252520100%2525252C%2525250A%252529%25252520-%2525253E%25252520FlyteFile%2525253A%2525250A%25252520%25252520%25252520%25252520content_image%2525252C%25252520style_image%25252520%2525253D%25252520preprocess_img%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520content_img%2525253Dcontent_img%2525252C%25252520style_img%2525253Dstyle_img%2525250A%25252520%25252520%25252520%25252520%252529%2525250A%25252520%25252520%25252520%25252520return%25252520generate_image%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520content_image%2525253Dcontent_image%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520style_image%2525253Dstyle_image%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520epochs%2525253Depochs%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520steps_per_epoch%2525253Dsteps_per_epoch%2525252C%2525250A%25252520%25252520%25252520%25252520%252529%2525250A%2525250A%2525250Aif%25252520__name__%25252520%2525253D%2525253D%25252520%25252522__main__%25252522%2525253A%2525250A%25252520%25252520%25252520%25252520print%252528f%25252522Running%25252520%2525257B__file__%2525257D...%25252522%252529%2525250A%25252520%25252520%25252520%25252520print%252528f%25252522Stylized%25252520image%2525253A%25252520%2525257Bneural_style_transfer_wf%252528%252529%2525257D%25252522%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9e4c4ad0171ace437a944a797aca2cc/href">https://medium.com/media/f9e4c4ad0171ace437a944a797aca2cc/href</a></iframe><p>Once the pipeline is <a href="https://docs.flyte.org/en/latest/deployment/aws/index.html">deployed</a>, the subsequent step would be to set up the S3 bucket and configure Lambda.</p><h3>Configure AWS S3 Bucket and Lambda</h3><p>Images will be uploaded to the S3 bucket, and Lambda will be used to trigger the Flyte workflow as soon as an image is uploaded.</p><h4>S3 Bucket</h4><p>To configure the S3 bucket,</p><ol><li>Open the <a href="https://console.aws.amazon.com/s3/">Amazon S3 console</a>.</li><li>Choose <strong>Buckets</strong>.</li><li>Choose <strong>Create bucket</strong>.</li><li>Give the bucket a name, e.g., “neural-style-transfer”.</li><li>Choose the appropriate AWS region (make sure Lambda is created in the same AWS region).</li><li>Block or unblock public access (this tutorial assumes that public access is granted).</li><li>Choose <strong>Create bucket</strong>.</li></ol><h4>Lambda</h4><p>A Lambda function can be created from scratch, through a blueprint, a container image, or a serverless app repository. Blueprint can be chosen to fetch sample lambda code, in our case, an S3 blueprint. However, since we need to connect to FlyteRemote from within Lambda, we have to install the flytekit library. Library installation within Lambda is possible through the <a href="https://docs.aws.amazon.com/lambda/latest/dg/python-package.html"><em>zip file</em></a> or <a href="https://docs.aws.amazon.com/lambda/latest/dg/images-create.html"><em>container image</em></a> approach.</p><p>Zip file is the easiest approach to get flytekit into Lambda, but due to the size limitations it imposes on the zip file, a much more feasible way would be to use the container image approach.</p><h4>Container Image</h4><p>To create a container image on your machine:</p><p>1. Create a project directory (e.g., lambda) to accommodate the lambda function.</p><p>2. Create 4 files in the directory: lambda_function.py, Dockerfile, requirements.txt, and flyte.config.</p><pre>lambda<br> ├── Dockerfile<br> ├── flyte.config<br> ├── lambda_function.py<br> └── requirements.txt</pre><p>3. lambda_function.py: encapsulate the code to fetch the uploaded image, instantiate a <a href="https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html#design-control-plane"><strong>FlyteRemote</strong></a> object, and trigger the Flyte workflow.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec23c79627693cd7e10e45c3f30451a0/href">https://medium.com/media/ec23c79627693cd7e10e45c3f30451a0/href</a></iframe><p>ℹ️ FlyteRemote provides a programmatic interface to interact with the Flyte backend.</p><p>Make sure to fill in the endpoint, default_project (e.g. flytesnacks), default_domain (e.g. development), and the name of the launch plan (e.g. neural_style_transfer.example.neural_style_transfer_wf).</p><p>4. <strong>flyte.config</strong>: add configuration to connect to Flyte through FlyteRemote.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5458be066b6b56074b4a488867a0ac66/href">https://medium.com/media/5458be066b6b56074b4a488867a0ac66/href</a></iframe><p>5. <strong>requirements.txt</strong></p><pre>flytekit&gt;=1.0.0</pre><p>6. <strong>Dockerfile</strong>: copy lambda_function.py, flyte.config, and requirements.txt to the root. Instantiate CMD to the handler that is used in the lambda_function.py file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac023c59181bf69d28bfb08200c9b72c/href">https://medium.com/media/ac023c59181bf69d28bfb08200c9b72c/href</a></iframe><p>7. Build a Docker image in the project directory using the command:</p><pre>docker build -t neural-style-transfer .</pre><p>8. Authenticate Docker CLI to the Amazon ECR registry.</p><pre>aws ecr get-login-password --region &lt;us-east-1&gt; | docker login --username AWS --password-stdin &lt;123456789012&gt;.dkr.ecr.&lt;us-east-1&gt;.amazonaws.com</pre><p>Make sure to replace text in &lt;&gt;.</p><p>9. Create a repository in the ECR.</p><ul><li>Open the <a href="https://console.aws.amazon.com/ecr/">Amazon ECR console</a>.</li><li>Choose <strong>Repositories</strong>.</li><li>Choose <strong>Create repository</strong> (e.g., lambda).</li></ul><p>10. Tag your Docker image and push the image to the newly-created repository.</p><pre>docker tag neural-style-transfer:latest &lt;123456789012&gt;.dkr.ecr.&lt;us-east-1&gt;.amazonaws.com/lambda:neural-style-transfer-latest</pre><pre>docker push &lt;123456789012&gt;.dkr.ecr.&lt;us-east-1&gt;.amazonaws.com/lambda:neural-style-transfer-latest</pre><p>Make sure to replace text in &lt;&gt; in the registry details.</p><p>That’s it! You now have your image in the ECR.</p><h4>Lambda Configuration</h4><p>To configure Lambda,</p><ol><li>Open the <a href="https://console.aws.amazon.com/lambda/home#/functions">Functions page</a> of the Lambda console.</li><li>Choose <strong>Create function</strong>.</li><li>Choose <strong>Container image</strong>.</li><li>Enter the function name (e.g., s3-lambda).</li><li>Give the <strong>Container Image URI</strong> (should be available in <a href="https://console.aws.amazon.com/ecr/">Amazon ECR console</a> -&gt; <strong>Repositories</strong> dashboard).</li><li>Choose <strong>Create function</strong>.</li></ol><p>You now have the lambda configured!</p><h3>Permissions</h3><p>S3 bucket and Lambda are currently separate entities. To trigger Lambda as soon as an image is uploaded to the S3 bucket, we must establish a connection between them.</p><p>Connecting them also requires setting up the required permissions. But before configuring the permissions, copy the bucket and Lambda ARNs.</p><p><strong>Bucket ARN</strong>:</p><ol><li>Open the <a href="https://console.aws.amazon.com/s3/">Amazon S3 console</a>.</li><li>Choose <strong>Buckets</strong>.</li><li>Choose your bucket.</li><li>Choose <strong>Properties</strong>.</li><li>Copy the ARN.</li></ol><p><strong>Lambda ARN</strong>:</p><ol><li>Open the <a href="https://console.aws.amazon.com/lambda/home#/functions">Functions page</a> of the Lambda console.</li><li>Choose <strong>Functions</strong>.</li><li>Choose your Lambda.</li><li>Choose <strong>Configuration</strong> and then choose <strong>Permissions</strong>.</li><li>Click on the role in <strong>Execution role</strong>.</li><li>Copy the ARN.</li></ol><h4>S3 Bucket</h4><p>To set up permissions for the S3 bucket:</p><ol><li>Go to the S3 bucket you created.</li><li>Select <strong>Permissions</strong>.</li><li>In the <strong>Bucket policy</strong>, choose <strong>Edit</strong>.</li><li>Add the following policy:</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78dcfd6222b2db6c0a9761ba6cfc1032/href">https://medium.com/media/78dcfd6222b2db6c0a9761ba6cfc1032/href</a></iframe><p>Make sure to fill in the Lambda execution role ARN and the S3 bucket ARN.</p><h4>Lambda</h4><p>To set up permissions for the Lambda:</p><p>1. Follow steps 1- 4 outlined in the <em>Lambda ARN</em> section.</p><p>2. Under Permissions, choose <strong>Add Permissions</strong>.</p><p>3. In the dropdown, choose <strong>Create inline policy</strong>.</p><p>4. Under the <strong>JSON</strong> tab, paste the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff1abde1f3ff6185395de670f9c1c987/href">https://medium.com/media/ff1abde1f3ff6185395de670f9c1c987/href</a></iframe><p>Make sure to fill in the S3 bucket ARN.</p><p>5. Choose <strong>Review policy</strong>.</p><p>6. For <strong>Name</strong>, enter a name for your policy.</p><p>7. Choose <strong>Create policy</strong>.</p><p>You can add FLYTE_CREDENTIALS_CLIENT_SECRET to the lambda’s environment variables as part of initializing FlyteRemote. To do so:</p><ol><li>Follow steps 1–3 outlined in the <em>Lambda ARN</em> section.</li><li>Choose <strong>Configuration</strong> and then choose <strong>Environment Variables</strong>.</li><li>Set the key as FLYTE_CREDENTIALS_CLIENT_SECRET, and the value should be your secret.</li></ol><p>Now comes the fun part — linking lambda to the S3 bucket!</p><h3>Trigger</h3><p>To set up the trigger:</p><ol><li>Follow steps 1–3 outlined in the <em>Lambda ARN</em> section.</li><li>Choose <strong>Configuration</strong> and then choose <strong>Triggers</strong>.</li><li>Click <strong>Add trigger</strong>.</li><li>In the <strong>Select a trigger</strong> dropdown, choose S3.</li><li>Choose your S3 bucket under <strong>Bucket</strong>.</li><li>Choose <strong>Add</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L8pcIi_WLAYytGlM" /><figcaption><em>Figure 2. There should be a link established between S3 and Lambda. (Image by Author)</em></figcaption></figure><h3>Test the Application</h3><p>To test the application, upload an image to the S3 bucket. On your Flyte console, under the neural style transfer workflow, check if the execution got triggered. The output of the execution should be your stylized image!</p><h3>Next Steps</h3><p>To summarize, we’ve built an event-driven application that triggers and executes an ML pipeline on the fly whenever there’s new data. It’s quite easy to productionize the pipeline with Flyte and AWS Lambda, as seen in this tutorial. We can also have a front-end application on top of this flow to make the application even more accessible.</p><p>If you want to give feedback about this tutorial or have questions regarding the implementation, please post in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18fa8145ef5b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/build-an-event-driven-neural-style-transfer-application-using-aws-lambda-18fa8145ef5b">Build an Event-Driven Neural Style Transfer Application Using AWS Lambda</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Open-Source Tools That Can Help You Build ML Pipelines With Ease]]></title>
            <link>https://medium.com/better-programming/5-open-source-tools-that-can-help-you-build-ml-pipelines-with-ease-46eb5d4c3488?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/46eb5d4c3488</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Fri, 11 Feb 2022 15:32:55 GMT</pubDate>
            <atom:updated>2022-02-12T07:41:20.779Z</atom:updated>
            <content:encoded><![CDATA[<h4>All production-friendly</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PC4RAUksq0HQxCLkNDh16g.png" /><figcaption>Image by Author</figcaption></figure><p>ML isn’t all about training a K-means classifier on Iris data in a Jupyter notebook<em>.</em> You might want to <em>train tons of data</em> using a <em>complex ML model</em> in the presence of <em>high-end infrastructure</em>. Now, this cannot be done with a single tool. We need a collection of them!</p><p>With the spike in interest to build production-ready, end-to-end ML pipelines, there has been an ever-growing demand to use the right toolset which could scale faster, and are actively being developed.</p><p>In this article, I’ll talk about 5 open-source tools that I have come across and used, which could assist you in multiple stages of building ML pipelines.</p><h4><a href="https://flyte.org/">Flyte</a></h4><p><a href="https://github.com/flyteorg/flyte">GitHub</a> . <a href="https://docs.flyte.org/en/latest/index.html">Docs</a> . <a href="https://slack.flyte.org/">Community</a></p><p><em>Current Deployments: Spotify, Lyft, Freenome, GoJek, etc.</em></p><p>To deploy production-grade ML <em>with</em> high development velocity and <em>without</em> the hassle of infrastructure, we need a platform that could help with the <em>Ops</em> aspect of <em>ML</em>. Precisely, we need a platform that facilitates collaboration between teams, handles repetitive runs of code without exhausting resources, stands by data scientists who do not like to intervene in the infrastructure aspects, is scalable and trustworthy, and the list could go on.</p><p><a href="https://flyte.org/">Flyte</a></p><p>Flyte is a workflow automation platform that helps automate critical ML and data pipelines. It makes distributed training easy, has no single point of failure, provides data lineage, and simplifies the ramp to bootstrap the infrastructure. You need not worry about handling the <em>Ops</em> of <em>ML</em> anymore with Flyte.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hY6JAC5sXF7M6wY794I8pg.png" /><figcaption>FlyteConsole screengrab from <a href="https://blog.flyte.org/from-incubation-to-graduation-and-beyond#heading-enter-the-world-of-ml-aware-orchestration">blog.flyte.org</a></figcaption></figure><p>Getting started with Flyte is pretty simple; have Docker, Python, Git installed, run a couple of commands, and you’re good to go!</p><p><strong>How to deploy?</strong> All the deployment guides and best practices are compiled in the <a href="https://docs.flyte.org/en/latest/deployment/index.html">deployment section of the docs</a>. A few deployments might need expertise in Kubernetes and its related tools.</p><h4><a href="https://greatexpectations.io/">Great Expectations</a></h4><p><a href="https://github.com/great-expectations/great_expectations">GitHub</a> . <a href="https://docs.greatexpectations.io/docs/">Docs</a> . <a href="https://greatexpectations.io/slack">Community</a></p><p><em>Current Deployments: GitHub, Calm, Agero, Civis, etc.</em></p><p>Data isn’t always how we expect it to be. Data validation is crucial all the time to ensure that data looks the way we want it to. Great Expectations helps achieve the exact same thing—ensure data quality before the actual process begins, be it modeling or analysis.</p><p><a href="https://greatexpectations.io/">Great Expectations</a> does this through “<em>expectations</em>”, which are the assertations for data. These are the suites where we have to describe what we expect from the data. Alternatively, expectations can be generated using a “<em>data profiler</em>”. I haven’t tested this feature thoroughly, but it sure looks promising!</p><p>After defining the required configuration and expectations, Great Expectations does data validation and throws off the results. To spruce up the data validation results, Great Expectations offers “<em>data docs</em>”—a user-friendly UI to view what has gone wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GRqo_8t_QCMPmErv.png" /><figcaption>Data docs screengrab from <a href="https://docs.greatexpectations.io/docs/tutorials/getting_started/tutorial_validate_data">docs.greatexpectations.io</a></figcaption></figure><p><strong>How to deploy? </strong><a href="https://docs.greatexpectations.io/docs/deployment_patterns/how_to_instantiate_a_data_context_hosted_environments#">Deployment guides</a> are included in the docs. It has some integrations up and running; however, standalone deployment shouldn’t be too hard.</p><h4><a href="https://feast.dev/">Feast</a></h4><p><a href="https://github.com/feast-dev/feast">GitHub</a> . <a href="https://docs.feast.dev/">Docs</a> . <a href="https://slack.feast.dev/">Community</a></p><p><em>Current Deployments: GoJek, Shopify, Salesforce, IBM, etc.</em></p><p><a href="https://feast.dev/">Feast</a> is a <em>fea</em>ture <em>st</em>ore. In ML, there can be lots of features if the dataset is huge. A feature store is helpful to manage and productionize features, achieve consistency between training and serving data and track the feature versions and lineage. Feast published a comprehensive article titled <a href="https://feast.dev/blog/what-is-a-feature-store/">What is a Feature Store?</a> if you’d like to dive deep into the concept of feature store.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*19yeR8GFV0-owKtF" /><figcaption>Feast architecture (<a href="https://docs.feast.dev/">Source</a>)</figcaption></figure><p>One interacts with the feature store primarily during training and inference. First, the feature definitions need to be defined and registered, training data can be generated using the feature store, features need to be materialized from an offline to an online store, and lastly, the latest features from the online store can be used for inference.</p><p><strong>How to deploy? </strong>There’s<strong> </strong>a<strong> </strong><a href="https://docs.feast.dev/how-to-guides/running-feast-in-production">detailed guide</a> to deploy Feast in production. The deployment requires automation to be put in place to simplify usage down the line.</p><h4><a href="https://horovod.ai/">Horovod</a></h4><p><a href="https://github.com/horovod/horovod">GitHub</a> . <a href="https://horovod.readthedocs.io/en/stable/">Docs</a></p><p>If deep learning (DL) models do not perform well during training with a single GPU, we may need many of them to distribute workload in order to speed up the process. <a href="https://horovod.ai/">Horovod</a> is a distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. It makes it easy to scale to thousands of GPUs even when the code is single-GPU compatible (of course, the code has to use the Horovod APIs).</p><p>The main advantage of Horovod is it works with multiple DL libraries—choose the DL framework, call the right APIs, and voila! results should be produced in far less time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xbw8qond13P4smLK.png" /><figcaption>Horovod benchmarks (<a href="https://horovod.readthedocs.io/en/stable/summary_include.html">Source</a>)</figcaption></figure><p><strong>How to deploy?</strong> The <a href="https://horovod.readthedocs.io/en/stable/summary_include.html#id7">installation guide</a> should help with setting up the environment. If there’s a Dockerfile, insert all the installation instructions carefully, be it MPI, NCCL, or the relevant environment variables.</p><h4><a href="https://developer.nvidia.com/nvidia-triton-inference-server">Triton Inference Server</a></h4><p><a href="https://github.com/triton-inference-server/server">GitHub</a> . <a href="https://github.com/triton-inference-server/server/tree/r22.01#documentation">Docs</a></p><p>Inferencing is a significant aspect of ML. Since it’d be on the user-facing side, we cannot comprise on the speed—it has to be fast and accurate. When you open Netflix, you get movie recommendations on the fly. Imagine the speed with which the predictions have to be generated and the number of users who might be accessing Netflix at the same time! That could be a pretty big number, isn’t it?</p><p>NVIDIA’s <a href="https://developer.nvidia.com/nvidia-triton-inference-server">Triton</a> “inference” server can simplify the deployment of a wide range of models belonging to various frameworks, and scale with ease. It can run concurrently on multiple GPUs and offers a host of features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/0*vbozLjL9bJ7KQoQA.jpg" /><figcaption>What’s Triton made of? (<a href="https://developer.nvidia.com/nvidia-triton-inference-server">Source</a>)</figcaption></figure><p>To run Triton, pull a pre-existing Docker image, and start the container which in turn, spins up the server.</p><p><strong>How to deploy? </strong>To deploy Triton using Kubernetes and Helm, there are <a href="https://github.com/triton-inference-server/server/blob/main/deploy/gcp/README.md">GCP</a>, <a href="https://github.com/triton-inference-server/server/blob/main/deploy/aws/README.md">AWS</a>, and <a href="https://github.com/triton-inference-server/server/blob/main/deploy/fleetcommand/README.md">NVIDIA FleetCommand</a> guides. An open-source project called <a href="https://els-rd.github.io/transformer-deploy/">transformer-deploy</a> provides an off-the-shelf CPU/GPU inference server based on NVIDIA Triton for Hugging Face transformer models to simplify the deployment process.</p><p><a href="https://els-rd.github.io/transformer-deploy/">transformer-deploy by Lefebvre Dalloz</a></p><p>Thanks for reading! Let me know the tools you use to build ML pipelines.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=46eb5d4c3488" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/5-open-source-tools-that-can-help-you-build-ml-pipelines-with-ease-46eb5d4c3488">5 Open-Source Tools That Can Help You Build ML Pipelines With Ease</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data-Parallel Distributed Training With Horovod and Flyte]]></title>
            <link>https://medium.com/union-ai/data-parallel-distributed-training-with-horovod-and-flyte-a92b48919b32?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/a92b48919b32</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[distributed-training]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 13:31:45 GMT</pubDate>
            <atom:updated>2022-07-28T05:05:52.803Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understand how Horovod and Flyte, along with Spark and MPIOperator, simplify building robust distributed data pipelines</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3oTtShwl4HP1Tw3zA_ZVw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@martinsanchez">Martin Sanchez</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p><em>This blog is with reference to a talk titled “Efficient Data Parallel Distributed Training with Flyte, Spark &amp; Horovod”, presented by </em><a href="https://medium.com/u/1835b5d5b5d0"><em>Katrina Rogan</em></a><em> and </em><a href="https://medium.com/u/f3ae7ced4945"><em>Ketan Umare</em></a><em> at OSPOCon 2021, Seattle.</em></p><p>To get the ball rolling, let’s understand what the title implies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q8AQ385dD6Zv_tjXsIwl9g.png" /></figure><ol><li><strong>Distributed Training</strong>: <a href="https://neptune.ai/blog/distributed-training">Distributed training</a> is the process of distributing the machine learning workload to multiple worker processes. It is helpful in scenarios where we have time- and compute-intensive deep learning workloads to run.</li><li><strong>Data-Parallel</strong>: Distributed training can be two types: model-parallel and data-parallel. Model-parallel training parallelizes the model where the global parameters are usually shared amongst the workers. Data-parallel training parallelizes the data, requiring sharing weights after one batch of training data. <em>Our focus in this blog would be on data-parallel distributed training.</em></li><li><strong>Horovod</strong>: Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. The goal of Horovod is to make distributed deep learning fast and easy to use. It uses the <a href="https://mpitutorial.com/tutorials/mpi-reduce-and-allreduce/">all-reduce</a> algorithm for fast distributed training rather than a parameter server approach (<a href="https://www.run.ai/guides/gpu-deep-learning/distributed-training/#Deep">all-reduce vs. parameter server</a>). It builds on top of low-level frameworks like MPI and NCCL and provides optimized algorithms for sharing data between parallel training processes.</li><li><strong>Flyte</strong>: Flyte is a workflow orchestration engine that seamlessly builds machine learning and data pipelines.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6DeoFeokyOpjEcJmZugugw.png" /><figcaption><em>Figure 1. Data-parallel distributed training (Image by Author)</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aLaHmQVxCWtOW6gmdCGm6A.png" /><figcaption><em>Figure 2. Parameter server vs. ring all-reduce (Image by Author)</em></figcaption></figure><p>With Horovod and Flyte, we can build robust distributed data pipelines. The two possible use cases that we shall explore in this blog integrate with:</p><ul><li>Spark</li><li>MPI Operator</li></ul><h3>Spark</h3><p>Spark is a data processing and analytics engine to deal with large-scale data. Here’s how Horovod-Spark-Flyte can be beneficial: Horovod provides the distributed framework, Spark enables extracting, preprocessing, and partitioning data, Flyte can stitch the former two pieces together, e.g., by connecting the data output of a Spark transform to a training system using Horovod while ensuring high utilization of GPUs!</p><p>In the next section, let’s understand how the integration took shape.</p><p>Here’s an interesting fact about Spark integration:</p><p><strong>Spark integrates with both Horovod and Flyte</strong></p><h4>Horovod and Spark</h4><p>Horovod implicitly supports Spark. It facilitates running distributed jobs in Spark clusters. “In situations where training data originates from Spark, Horovod on Spark enables a tight model design loop in which data processing, model training, and model evaluation are all done in Spark”[1].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KR7I8Gjm7pSei9XinszYoQ.png" /><figcaption><em>Figure 3. Pre-process, train, and evaluate in the same environment (ref: </em><a href="https://eng.uber.com/horovod-pyspark-apache-mxnet-support/"><em>Horovod Adds Support for PySpark and Apache MXNet and Additional Features for Faster Training </em></a><em>) (Image by Author)</em></figcaption></figure><p>In our example, to activate Horovod on Spark, we use an <em>Estimator API</em>. An estimator API abstracts the data processing, model training and checkpointing, and distributed training, making it easy to integrate and run our example code. There is also a low-level <em>Run API</em> available, which offers a more fine-grained control.</p><p>Since we use the Keras deep learning library, here’s how we install the relevant Horovod packages:</p><pre>HOROVOD_WITH_MPI=1 HOROVOD_WITH_TENSORFLOW=1 pip install --no-cache-dir horovod[spark,tensorflow]==0.22.1</pre><p>The installation includes enabling <a href="https://horovod.readthedocs.io/en/stable/mpi_include.html">MPI</a> and TensorFlow environments.</p><h4>Flyte and Spark</h4><p>Flyte can execute Spark jobs natively on a Kubernetes Cluster, which manages a virtual cluster’s lifecycle, spin-up, and tear down. It leverages the open-sourced <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator">Spark On K8s Operator</a> and can be enabled without signing up for any service. This is like running a transient spark cluster — a type of cluster spun up for a specific Spark job and torn down after completion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yD-zwo6E0KrZSbEY3lOgzg.png" /><figcaption><em>Figure 4. Flyte-Spark integration (Image by Author)</em></figcaption></figure><p>To install the <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html">Spark plugin on Flyte</a>, we use the following command:</p><pre>pip install flytekitplugins-spark</pre><h4>Summing-up: Horovod, Spark, and Flyte</h4><p>The problem statement we are looking at is forecasting sales using <a href="https://www.kaggle.com/c/rossmann-store-sales">Rossmann store sales</a> data. Our example is an adaptation of the <a href="https://github.com/horovod/horovod/blob/master/examples/spark/keras/keras_spark_rossmann_estimator.py">Horovod-Spark</a> example. Here’s how the code is streamlined:</p><p>In a Spark cluster that is set up using the Flyte-Spark plugin:</p><ul><li>Fetch the Rossmann sales data</li><li>Perform complicated data pre-processing on Spark DataFrame</li><li>Define a Keras model and perform distributed training using Horovod on Spark</li><li>Generate predictions and store them in a submission file</li></ul><h4>Flyte-Spark Plugin</h4><p>Flyte-Spark plugin can be activated by applying Spark cluster configuration to a Flyte task.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252540task%2528%25250A%252520%252520%252520%252520task_config%25253DSpark%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252523%252520this%252520configuration%252520is%252520applied%252520to%252520the%252520spark%252520cluster%25250A%252520%252520%252520%252520%252520%252520%252520%252520spark_conf%25253D%25257B%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.driver.memory%252522%25253A%252520%2525222000M%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.executor.memory%252522%25253A%252520%2525222000M%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.executor.cores%252522%25253A%252520%2525221%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.executor.instances%252522%25253A%252520%2525222%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.driver.cores%252522%25253A%252520%2525221%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.sql.shuffle.partitions%252522%25253A%252520%25252216%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252520%252522spark.worker.timeout%252522%25253A%252520%252522120%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%25257D%25250A%252520%252520%252520%252520%2529%25252C%25250A%252520%252520%252520%252520cache%25253DTrue%25252C%25250A%252520%252520%252520%252520cache_version%25253D%2525220.2%252522%25252C%25250A%252520%252520%252520%252520requests%25253DResources%2528mem%25253D%2525221Gi%252522%2529%25252C%25250A%252520%252520%252520%252520limits%25253DResources%2528mem%25253D%2525221Gi%252522%2529%25252C%25250A%2529%25250Adef%252520horovod_spark_task%2528%25250A%252520%252520%252520data_dir%25253A%252520FlyteDirectory%25252C%252520hp%25253A%252520Hyperparameters%25252C%252520work_dir%25253A%252520FlyteDirectory%25250A%2529%252520-%25253E%252520FlyteDirectory%25253A%25250A%252520%252520%252520%252520...&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252540task%252528%2525250A%25252520%25252520%25252520%25252520task_config%2525253DSpark%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252523%25252520this%25252520configuration%25252520is%25252520applied%25252520to%25252520the%25252520spark%25252520cluster%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520spark_conf%2525253D%2525257B%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.driver.memory%25252522%2525253A%25252520%252525222000M%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.executor.memory%25252522%2525253A%25252520%252525222000M%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.executor.cores%25252522%2525253A%25252520%252525221%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.executor.instances%25252522%2525253A%25252520%252525222%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.driver.cores%25252522%2525253A%25252520%252525221%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.sql.shuffle.partitions%25252522%2525253A%25252520%2525252216%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252522spark.worker.timeout%25252522%2525253A%25252520%25252522120%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%2525257D%2525250A%25252520%25252520%25252520%25252520%252529%2525252C%2525250A%25252520%25252520%25252520%25252520cache%2525253DTrue%2525252C%2525250A%25252520%25252520%25252520%25252520cache_version%2525253D%252525220.2%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520requests%2525253DResources%252528mem%2525253D%252525221Gi%25252522%252529%2525252C%2525250A%25252520%25252520%25252520%25252520limits%2525253DResources%252528mem%2525253D%252525221Gi%25252522%252529%2525252C%2525250A%252529%2525250Adef%25252520horovod_spark_task%252528%2525250A%25252520%25252520%25252520data_dir%2525253A%25252520FlyteDirectory%2525252C%25252520hp%2525253A%25252520Hyperparameters%2525252C%25252520work_dir%2525253A%25252520FlyteDirectory%2525250A%252529%25252520-%2525253E%25252520FlyteDirectory%2525253A%2525250A%25252520%25252520%25252520%25252520...&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/53a0d410f2e5cfe697f551a565842f01/href">https://medium.com/media/53a0d410f2e5cfe697f551a565842f01/href</a></iframe><p>On applying Spark config to horovod_spark_task, the task behaves like a <strong>PySpark task</strong>. All attributes in the spark_conf correspond to the pre-existing Spark attributes.</p><p>Keep in mind that the horovod_spark_task encapsulates data pre-processing, training, and evaluation.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252540task%2528...%2529%25250Adef%252520horovod_spark_task%2528%25250A%252520%252520%252520data_dir%25253A%252520FlyteDirectory%25252C%252520hp%25253A%252520Hyperparameters%25252C%252520work_dir%25253A%252520FlyteDirectory%25250A%2529%252520-%25253E%252520FlyteDirectory%25253A%25250A%25250A%252520%252520%252520max_sales%25252C%252520vocab%25252C%252520train_df%25252C%252520test_df%252520%25253D%252520data_preparation%2528data_dir%25252C%252520hp%2529%25250A%25250A%252520%252520%252520%252523%252520working%252520directory%252520will%252520have%252520the%252520model%252520and%252520predictions%252520as%252520separate%252520files%25250A%252520%252520%252520working_dir%252520%25253D%252520flytekit.current_context%2528%2529.working_directory%25250A%25250A%252520%252520%252520keras_model%252520%25253D%252520train%2528%25250A%252520%252520%252520%252520%252520%252520%252520max_sales%25252C%25250A%252520%252520%252520%252520%252520%252520%252520vocab%25252C%25250A%252520%252520%252520%252520%252520%252520%252520hp%25252C%25250A%252520%252520%252520%252520%252520%252520%252520work_dir%25252C%25250A%252520%252520%252520%252520%252520%252520%252520train_df%25252C%25250A%252520%252520%252520%252520%252520%252520%252520working_dir%25252C%25250A%252520%252520%252520%2529%25250A%252520%252520%252520return%252520test%2528keras_model%25252C%252520working_dir%25252C%252520test_df%25252C%252520hp%2529&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252540task%252528...%252529%2525250Adef%25252520horovod_spark_task%252528%2525250A%25252520%25252520%25252520data_dir%2525253A%25252520FlyteDirectory%2525252C%25252520hp%2525253A%25252520Hyperparameters%2525252C%25252520work_dir%2525253A%25252520FlyteDirectory%2525250A%252529%25252520-%2525253E%25252520FlyteDirectory%2525253A%2525250A%2525250A%25252520%25252520%25252520max_sales%2525252C%25252520vocab%2525252C%25252520train_df%2525252C%25252520test_df%25252520%2525253D%25252520data_preparation%252528data_dir%2525252C%25252520hp%252529%2525250A%2525250A%25252520%25252520%25252520%25252523%25252520working%25252520directory%25252520will%25252520have%25252520the%25252520model%25252520and%25252520predictions%25252520as%25252520separate%25252520files%2525250A%25252520%25252520%25252520working_dir%25252520%2525253D%25252520flytekit.current_context%252528%252529.working_directory%2525250A%2525250A%25252520%25252520%25252520keras_model%25252520%2525253D%25252520train%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520max_sales%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520vocab%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520hp%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520work_dir%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520train_df%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520working_dir%2525252C%2525250A%25252520%25252520%25252520%252529%2525250A%25252520%25252520%25252520return%25252520test%252528keras_model%2525252C%25252520working_dir%2525252C%25252520test_df%2525252C%25252520hp%252529&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0584c76ef65a941f5aa0e68dfb416954/href">https://medium.com/media/0584c76ef65a941f5aa0e68dfb416954/href</a></iframe><p><em>Note: You can have two separate Spark tasks for data processing and training in Flyte. Training can also be done through Flyte’s MPIOperator, which shall be seen in the MPIOperator section.</em></p><p>In the train function, we define our Keras model and perform distributed training on Spark data using Horovod’s Estimator API.</p><h4>Horovod Estimator API</h4><p>The Keras estimator in Horovod can be used to train a model on an existing Spark DataFrame, leveraging Horovod’s ability to scale across multiple workers without any specialized code for distributed training.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%252520horovod.spark.keras%252520as%252520hvd%25250Afrom%252520horovod.spark.common.backend%252520import%252520SparkBackend%25250Afrom%252520horovod.spark.common.store%252520import%252520Store%25250A%25250Adef%252520train%2528%25250A%252520%252520%252520max_sales%25253A%252520float%25252C%25250A%252520%252520%252520vocab%25253A%252520Dict%25255Bstr%25252C%252520List%25255BAny%25255D%25255D%25252C%25250A%252520%252520%252520hp%25253A%252520Hyperparameters%25252C%25250A%252520%252520%252520work_dir%25253A%252520FlyteDirectory%25252C%25250A%252520%252520%252520train_df%25253A%252520pyspark.sql.DataFrame%25252C%25250A%252520%252520%252520working_dir%25253A%252520FlyteDirectory%25252C%25250A%2529%25253A%25250A%252520%252520%252520...%25250A%25250A%252520%252520%252520%252523%252520Horovod%25253A%252520run%252520training%25250A%252520%252520%252520store%252520%25253D%252520Store.create%2528work_dir.remote_source%2529%25250A%252520%252520%252520backend%252520%25253D%252520SparkBackend%2528%25250A%252520%252520%252520%252520%252520%252520%252520num_proc%25253Dhp.num_proc%25252C%25250A%252520%252520%252520%252520%252520%252520%252520stdout%25253Dsys.stdout%25252C%25250A%252520%252520%252520%252520%252520%252520%252520stderr%25253Dsys.stderr%25252C%25250A%252520%252520%252520%252520%252520%252520%252520prefix_output_with_timestamp%25253DTrue%25252C%25250A%252520%252520%252520%2529%25250A%252520%252520%252520keras_estimator%252520%25253D%252520hvd.KerasEstimator%2528%25250A%252520%252520%252520%252520%252520%252520%252520backend%25253Dbackend%25252C%25250A%252520%252520%252520%252520%252520%252520%252520store%25253Dstore%25252C%25250A%252520%252520%252520%252520%252520%252520%252520model%25253Dmodel%25252C%25250A%252520%252520%252520%252520%252520%252520%252520optimizer%25253Dopt%25252C%25250A%252520%252520%252520%252520%252520%252520%252520loss%25253D%252522mae%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520metrics%25253D%25255Bexp_rmspe%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520%252520custom_objects%25253DCUSTOM_OBJECTS%25252C%25250A%252520%252520%252520%252520%252520%252520%252520feature_cols%25253Dall_cols%25252C%25250A%252520%252520%252520%252520%252520%252520%252520label_cols%25253D%25255B%252522Sales%252522%25255D%25252C%25250A%252520%252520%252520%252520%252520%252520%252520validation%25253D%252522Validation%252522%25252C%25250A%252520%252520%252520%252520%252520%252520%252520batch_size%25253Dhp.batch_size%25252C%25250A%252520%252520%252520%252520%252520%252520%252520epochs%25253Dhp.epochs%25252C%25250A%252520%252520%252520%252520%252520%252520%252520verbose%25253D2%25252C%25250A%252520%252520%252520%252520%252520%252520%252520checkpoint_callback%25253Dckpt_callback%25252C%25250A%252520%252520%252520%2529%25250A%25250A%252520%252520%252520keras_model%252520%25253D%252520keras_estimator.fit%2528train_df%2529.setOutputCols%2528%25255B%252522Sales_output%252522%25255D%2529%25250A%25250A%252520%252520%252520...%252520%252520%252520%25250A%25250A%252520%252520%252520return%252520keras_model&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3Dimport%25252520horovod.spark.keras%25252520as%25252520hvd%2525250Afrom%25252520horovod.spark.common.backend%25252520import%25252520SparkBackend%2525250Afrom%25252520horovod.spark.common.store%25252520import%25252520Store%2525250A%2525250Adef%25252520train%252528%2525250A%25252520%25252520%25252520max_sales%2525253A%25252520float%2525252C%2525250A%25252520%25252520%25252520vocab%2525253A%25252520Dict%2525255Bstr%2525252C%25252520List%2525255BAny%2525255D%2525255D%2525252C%2525250A%25252520%25252520%25252520hp%2525253A%25252520Hyperparameters%2525252C%2525250A%25252520%25252520%25252520work_dir%2525253A%25252520FlyteDirectory%2525252C%2525250A%25252520%25252520%25252520train_df%2525253A%25252520pyspark.sql.DataFrame%2525252C%2525250A%25252520%25252520%25252520working_dir%2525253A%25252520FlyteDirectory%2525252C%2525250A%252529%2525253A%2525250A%25252520%25252520%25252520...%2525250A%2525250A%25252520%25252520%25252520%25252523%25252520Horovod%2525253A%25252520run%25252520training%2525250A%25252520%25252520%25252520store%25252520%2525253D%25252520Store.create%252528work_dir.remote_source%252529%2525250A%25252520%25252520%25252520backend%25252520%2525253D%25252520SparkBackend%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520num_proc%2525253Dhp.num_proc%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520stdout%2525253Dsys.stdout%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520stderr%2525253Dsys.stderr%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520prefix_output_with_timestamp%2525253DTrue%2525252C%2525250A%25252520%25252520%25252520%252529%2525250A%25252520%25252520%25252520keras_estimator%25252520%2525253D%25252520hvd.KerasEstimator%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520backend%2525253Dbackend%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520store%2525253Dstore%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520model%2525253Dmodel%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520optimizer%2525253Dopt%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520loss%2525253D%25252522mae%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520metrics%2525253D%2525255Bexp_rmspe%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520custom_objects%2525253DCUSTOM_OBJECTS%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520feature_cols%2525253Dall_cols%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520label_cols%2525253D%2525255B%25252522Sales%25252522%2525255D%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520validation%2525253D%25252522Validation%25252522%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520batch_size%2525253Dhp.batch_size%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520epochs%2525253Dhp.epochs%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520verbose%2525253D2%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520checkpoint_callback%2525253Dckpt_callback%2525252C%2525250A%25252520%25252520%25252520%252529%2525250A%2525250A%25252520%25252520%25252520keras_model%25252520%2525253D%25252520keras_estimator.fit%252528train_df%252529.setOutputCols%252528%2525255B%25252522Sales_output%25252522%2525255D%252529%2525250A%2525250A%25252520%25252520%25252520...%25252520%25252520%25252520%2525250A%2525250A%25252520%25252520%25252520return%25252520keras_model&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/32b2979a02e7de6b4940ad6eb71e83eb/href">https://medium.com/media/32b2979a02e7de6b4940ad6eb71e83eb/href</a></iframe><p>After training, the Estimator returns a Transformer representation of the trained model. “The model transformer can be used like any Spark ML transformer to make predictions on an input DataFrame, writing them as new columns in the output DataFrame”[1].</p><pre>pred_df = keras_model.transform(test_df)</pre><p>The end-to-end code is available in the Flyte documentation in the <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/ml_training/spark_horovod/keras_spark_rossmann_estimator.html#sphx-glr-auto-case-studies-ml-training-spark-horovod-keras-spark-rossmann-estimator-py">Forecast Sales Using Rossmann Store Sales Data With Horovod and Spark section</a>.</p><h4>Pitfalls of Using Horovod on Spark</h4><ul><li>Lack of Spark expertise in your organization: data-scientists do not already use Spark</li><li>A hard boundary between data processing on the RDD and training on a Parquet file, which introduces overhead and operational complexity (where will the Parquet file live, how long will we persist it, etc.).</li><li>Lack of visibility: Horovod processes run within Spark executors. However, Horovod processes do not run as tasks within the Spark task graph because of which failures may be hard to track.</li><li>Your data-processing is on unstructured data, and simple map operations work better</li></ul><p>MPI Operator has also been developed as part of an effort to integrate Horovod.</p><h3>MPI Operator</h3><p>The MPI operator plugin within Flyte uses the <a href="https://github.com/kubeflow/mpi-operator">Kubeflow MPI Operator</a>, which makes it easy to run an all reduce-style distributed training on Kubernetes. It provides an extremely simplified interface for executing distributed training using MPI.</p><p>“MPI and Horovod together can be leveraged to simplify the process of distributed training. The MPI Operator provides a convenient wrapper to run the Horovod scripts”[2].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8Qg1LClwTi0KwFjtOqTkA.png" /><figcaption><em>Figure 5. Use Flyte’s MPIOperator to perform distributed training through Horovod (Image by Author)</em></figcaption></figure><p>To use the Flytekit MPI Operator plugin, we run the following command:</p><pre>pip install flytekitplugins-kfmpi</pre><p>Here’s an example of an MPI-enabled Flyte task:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%3Fbg%3Drgba%2528171%252C184%252C195%252C0%2529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%2525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%252540task%2528%25250A%252520%252520%252520%252520task_config%25253DMPIJob%2528%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252523%252520number%252520of%252520workers%252520to%252520be%252520spawned%252520in%252520the%252520cluster%252520for%252520this%252520job%25250A%252520%252520%252520%252520%252520%252520%252520%252520num_workers%25253D2%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252523%252520number%252520of%252520launcher%252520replicas%252520to%252520be%252520spawned%252520in%252520the%252520cluster%252520for%252520this%252520job%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252523%252520the%252520launcher%252520pod%252520invokes%252520mpirun%252520and%252520communicates%252520with%252520worker%252520pods%252520through%252520MPI%25250A%252520%252520%252520%252520%252520%252520%252520%252520num_launcher_replicas%25253D1%25252C%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252523%252520number%252520of%252520slots%252520per%252520worker%252520used%252520in%252520the%252520hostfile%25250A%252520%252520%252520%252520%252520%252520%252520%252520%252523%252520the%252520available%252520slots%252520%2528GPUs%2529%252520in%252520each%252520pod%25250A%252520%252520%252520%252520%252520%252520%252520%252520slots%25253D1%25252C%25250A%252520%252520%252520%252520%2529%25252C%25250A%252520%252520%252520%252520requests%25253DResources%2528cpu%25253D%25271%2527%25252C%252520mem%25253D%2525223000Mi%252522%2529%25252C%25250A%252520%252520%252520%252520limits%25253DResources%2528cpu%25253D%25272%2527%25252C%252520mem%25253D%2525226000Mi%252522%2529%25252C%25250A%252520%252520%252520%252520retries%25253D3%25252C%25250A%252520%252520%252520%252520cache%25253DTrue%25252C%25250A%252520%252520%252520%252520cache_version%25253D%2525220.5%252522%25252C%25250A%2529%25250Adef%252520mpi_task%2528...%2529%25253A%25250A%252520%252520%252520%252520%252523%252520do%252520some%252520work%25250A%252520%252520%252520%252520pass&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F%3Fbg%3Drgba%252528171%25252C184%25252C195%25252C0%252529%26t%3Dnight-owl%26wt%3Dsharp%26l%3Dpython%26ds%3Dfalse%26dsyoff%3D20px%26dsblur%3D68px%26wc%3Dtrue%26wa%3Dfalse%26pv%3D0px%26ph%3D0px%26ln%3Dfalse%26fl%3D0%26fm%3DHack%26fs%3D14px%26lh%3D133%252525%26si%3Dfalse%26es%3D2x%26wm%3Dfalse%26code%3D%25252540task%252528%2525250A%25252520%25252520%25252520%25252520task_config%2525253DMPIJob%252528%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252523%25252520number%25252520of%25252520workers%25252520to%25252520be%25252520spawned%25252520in%25252520the%25252520cluster%25252520for%25252520this%25252520job%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520num_workers%2525253D2%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252523%25252520number%25252520of%25252520launcher%25252520replicas%25252520to%25252520be%25252520spawned%25252520in%25252520the%25252520cluster%25252520for%25252520this%25252520job%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252523%25252520the%25252520launcher%25252520pod%25252520invokes%25252520mpirun%25252520and%25252520communicates%25252520with%25252520worker%25252520pods%25252520through%25252520MPI%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520num_launcher_replicas%2525253D1%2525252C%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252523%25252520number%25252520of%25252520slots%25252520per%25252520worker%25252520used%25252520in%25252520the%25252520hostfile%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252523%25252520the%25252520available%25252520slots%25252520%252528GPUs%252529%25252520in%25252520each%25252520pod%2525250A%25252520%25252520%25252520%25252520%25252520%25252520%25252520%25252520slots%2525253D1%2525252C%2525250A%25252520%25252520%25252520%25252520%252529%2525252C%2525250A%25252520%25252520%25252520%25252520requests%2525253DResources%252528cpu%2525253D%2525271%252527%2525252C%25252520mem%2525253D%252525223000Mi%25252522%252529%2525252C%2525250A%25252520%25252520%25252520%25252520limits%2525253DResources%252528cpu%2525253D%2525272%252527%2525252C%25252520mem%2525253D%252525226000Mi%25252522%252529%2525252C%2525250A%25252520%25252520%25252520%25252520retries%2525253D3%2525252C%2525250A%25252520%25252520%25252520%25252520cache%2525253DTrue%2525252C%2525250A%25252520%25252520%25252520%25252520cache_version%2525253D%252525220.5%25252522%2525252C%2525250A%252529%2525250Adef%25252520mpi_task%252528...%252529%2525253A%2525250A%25252520%25252520%25252520%25252520%25252523%25252520do%25252520some%25252520work%2525250A%25252520%25252520%25252520%25252520pass&amp;image=https%3A%2F%2Fcarbon.now.sh%2Fstatic%2Fbrand%2Fbanner.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d3193bbbc91ac7e78fa9c3f3894c602/href">https://medium.com/media/6d3193bbbc91ac7e78fa9c3f3894c602/href</a></iframe><p>An end-to-end <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/kfmpi/mpi_mnist.html#sphx-glr-auto-integrations-kubernetes-kfmpi-mpi-mnist-py">MPIJob example with Horovod</a> can be found in the Flyte documentation.</p><p>Horovod, Spark/MPI Operator, and Flyte together could help run distributed deep learning workflows much faster! If you happen to utilize either of the two integrations, do let us know what you think of it.</p><p><strong>Future work: use Streaming TypeTransformers to improve GPU Utilization.</strong></p><p>At scale, one of the challenges of training is the utilization of GPUs. In the case of downloading data, utilization of GPUs drops as the data is downloaded first.</p><p>Streaming TypeTransformers can help stream data as it is prepared into the training phase without sitting idle waiting for all the data to be processed. The Flyte community is working on allowing data to be streamed to training instances to improve the utilization of the GPUs.</p><h4>References</h4><p>[1] <a href="https://horovod.readthedocs.io/en/stable/spark_include.html">Horovod documentation — Spark</a></p><p>[2] <a href="https://horovod.readthedocs.io/en/stable/spark_include.html">Flyte documentation — MPI Operator</a></p><p><em>Flyte is a workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others, and is an incubating project at </em><a href="https://lfaidata.foundation/"><em>LF AI &amp; Data</em></a><em>.</em></p><p><em>We invite you to check out Flyte at </em><a href="https://flyte.org/"><em>flyte.org</em></a><em> and join our </em><a href="https://docs.google.com/forms/d/e/1FAIpQLScWPJZncyL-bBCWbuQ9HWuMzA8-r2RJc__CSWJoAYUQADN-BQ/viewform"><em>community</em></a><em> on Google Groups.</em></p><p><em>To start using Flyte, refer to the </em><a href="https://flyte.readthedocs.io/en/latest/"><em>documentation</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a92b48919b32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/union-ai/data-parallel-distributed-training-with-horovod-and-flyte-a92b48919b32">Data-Parallel Distributed Training With Horovod and Flyte</a> was originally published in <a href="https://medium.com/union-ai">Union-ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bring ML Close to Data Using Feast and Flyte]]></title>
            <link>https://medium.com/union-ai/bring-ml-close-to-data-using-feast-and-flyte-bd0cb5608678?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd0cb5608678</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Thu, 11 Nov 2021 13:40:37 GMT</pubDate>
            <atom:updated>2022-05-24T17:49:06.063Z</atom:updated>
            <content:encoded><![CDATA[<h4>And handle feature-engineered data effectively in an ML pipeline</h4><figure><img alt="Lego characters riding a bike" src="https://cdn-images-1.medium.com/max/1000/0*HUrAh7vujKUAcwPX" /><figcaption>Photo by <a href="https://unsplash.com/@tolga__">Tolga Ulkan</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption></figure><p><em>This blog is with reference to a talk titled “Self-serve Feature Engineering Platform Using Flyte and Feast,” which was presented by Ketan Umare and Felix Wang at OSPOCon 2021, Seattle.</em></p><p>Feature engineering is one of the greatest challenges in applied machine learning. It is the process of transforming raw data into <strong>features</strong> that better represent the underlying problem, resulting in improved model accuracy on unseen data. When machine learning models are deployed to production, effectively operationalizing data for model training and online inference is critical. For example, building feature pipelines is one of the most time-consuming activities in ML, but also one of the most important.</p><p>There are many different challenges around operationalizing data for machine learning:</p><p>1. <strong>Building feature pipelines is hard!</strong></p><ul><li>Each individual project often requires the development of unique pipelines, which adds up to a lot of work.</li><li>Data can come from many sources: data lakes, data warehouses, stream sources.</li><li>We are often bound by performance requirements for our use case, so we have to do a bunch of extra optimizations to meet those requirements.</li></ul><p>2.<strong> Consistent data access</strong></p><ul><li>Redevelopment of pipelines leads to inconsistencies in data</li><li>Training-serving skew degrades the model performance</li><li>Models need a point-in-time correct view of data to avoid label leakage</li></ul><p>3.<strong> Duplication of effort</strong></p><ul><li>Siloed development leads to no means of collaboration</li><li>Lack of governance and standardization</li></ul><p>“Increasingly data science and data engineering teams are turning towards <strong>feature stores</strong> to manage the data sets and data pipelines needed to productionize their ML applications.”¹ <a href="https://feast.dev/">Feast</a> is one such feature store that can act as a storage and serving layer to manage your features. It is typically placed after the transformation/data pipeline that produces the feature values.</p><p>Let’s now look at how Feast can help us solve the challenges mentioned above.</p><p>1. <strong>Building feature pipelines is hard!</strong></p><ul><li>Feast supports easy feature creation</li><li>Every time we query for the on-demand feature view, Feast will automatically execute the transform we specified</li></ul><p>2.<strong> Consistent data access</strong></p><ul><li>Feast provides a unified serving layer through which to retrieve features, which ensures that data is accessed consistently across the offline and online use cases</li></ul><p>3.<strong> Duplication of effort</strong></p><ul><li>Feast supports cataloging and discovery</li><li>Provides a standard interface through which teams can write and publish features</li></ul><h3>Flyte Meets Feast!</h3><p>“Flyte provides a way to train models and perform feature engineering as a single pipeline. But it provides no way to serve these features to production when the model matures and is ready to be served in production.”²</p><p>This is where Feast comes into the picture.</p><p>Feast provides the feature registry, an online feature serving system, and Flyte can engineer the features. Flyte can also help ensure incremental development of features and enables us to turn on the sync to online stores only when we are confident about the features.</p><p>With Feast and Flyte, we can assert control over our features! The following are advantages we can claim with Feast in Flyte:</p><ul><li>Reduce skew between offline and online features -&gt; improved model accuracy</li><li>Simple retrieval of features -&gt; lesser effort in managing data</li><li>Consistent data access -&gt; consistent data pipelines</li><li>No more duplication of effort -&gt; increased focus on the actual business logic</li><li>Feature cataloging and discovery -&gt; bridging siloed development</li></ul><p>-&gt; : leads to</p><h3>Integration</h3><p>TL;DR: Refer to the fully working example in the <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/feature_engineering/feast_integration/index.html">docs</a>.</p><p>The goal is to train a model on the horse colic dataset to determine if a lesion is surgical or not. We first retrieve the SQLite dataset, perform mean-median imputation, build a feature store, store and load offline features, perform univariate selection, train a Naive Bayes model, store online features, retrieve random online data points, and finally, generate prediction against the data point.</p><p>When Flyte and Feast are utilized together to implement the integration, Flyte turns the feature engineering pipeline into a reproducible workflow and Feast retrieves features for training the model and online inference.</p><p>To access Feast methods — apply(), get_historical_features(), materialize(), and get_online_features() using a feature store abstraction within a Flyte workflow and tasks, a <a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/feature_engineering/feast_integration/feast_dataobjects.html">feature store data class</a> has been defined. Moreover, a <a href="https://github.com/flyteorg/flytesnacks/blob/master/cookbook/case_studies/feature_engineering/feast_integration/custom_provider/provider.py">custom provider</a> has been defined to access remote Feast manifests within Flyte, which helps Flyte tasks residing in different Kubernetes pods to communicate seamlessly with each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2n0qvIpg1NmvIDNe" /><figcaption><em>Figure 1. Interoperability of Flyte and Feast (Image by Author)</em></figcaption></figure><h3>Next Steps</h3><p>Feast uses a centralized registry to handle features and their metadata. In such a case, concurrent writes by the Flyte workflows to the registry aren’t allowed as it mutates the same registry. This is an anti-pattern as this impedes Flyte’s memoization.</p><p>To resolve this issue, we could make the Feast registry (R) immutable and have every application result in a new registry: (R’, R’’, R’’’). Users can then access the new registry from the Flyte API as the output of a specific pipeline. This may seem disadvantageous because there would be no single, centralized registry. On the flip side, an immutable registry supports independent feature generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*heuGrVSQkgwxOZtQ" /><figcaption><em>Figure 2. Limitations and Strengths of Flyte + Feast integration (Image by Author)</em></figcaption></figure><h3><strong>References</strong></h3><ol><li><a href="https://feast.dev/blog/what-is-a-feature-store/">What is a Feature Store?</a></li><li><a href="https://docs.flyte.org/projects/cookbook/en/latest/auto/case_studies/feature_engineering/feast_integration/index.html">Flyte &lt;&gt; Feast Integration</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd0cb5608678" width="1" height="1" alt=""><hr><p><a href="https://medium.com/union-ai/bring-ml-close-to-data-using-feast-and-flyte-bd0cb5608678">Bring ML Close to Data Using Feast and Flyte</a> was originally published in <a href="https://medium.com/union-ai">Union-ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet Flyte at Hacktoberfest 2021]]></title>
            <link>https://medium.com/union-ai/meet-flyte-at-hacktoberfest-2021-ba5624ccceef?source=rss-7e65fabdea53------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba5624ccceef</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[hacktoberfest]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Samhita Alla]]></dc:creator>
            <pubDate>Thu, 30 Sep 2021 06:58:31 GMT</pubDate>
            <atom:updated>2022-05-24T17:48:18.887Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w3yHVpODAXggXMxZmgpFbg.png" /></figure><p>Open-source is growing fast, so are its contributors. At Flyte, we have had meaningful contributions from many contributors who have helped us improve Flyte’s codebase and made us think through unseen perspectives. We have always wanted to have more such contributions to expand our horizons. We have been thinking about ways to get more people involved, and at the same time, give back to our community.</p><p>We know that October is the month of the year when open-source is the most alive. Revealing <a href="https://hacktoberfest.digitalocean.com/">Hacktoberfest 2021</a>!</p><h3>About Hacktoberfest</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*puuQ50FV-6-NywID" /></figure><p>Hacktoberfest is the month-long event held in October to celebrate open-source software, which is when you can contribute to open-source software irrespective of what you do. It indeed is an excellent opportunity to develop your skills, and most importantly, win limited edition swag 🎁.</p><p>This year, we’re beyond thrilled to be a part of it. ✨</p><h3>About Flyte</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QfJnaQMaGkZLSzmI" /></figure><p><a href="https://flyte.org/">Flyte</a> is an open-source, container-native, structured programming and distributed processing platform. It enables highly concurrent, scalable, and maintainable workflows for machine learning and data processing.</p><p>We at Flyte are constantly striving to make MLOps better by providing off-the-shelf orchestration mechanisms to automate the way you build pipelines. <strong>Ours is a Kubernetes-native and type-safe platform</strong> — which we are super proud of!</p><blockquote><em>Though what Flyte encompasses is enormous, we made sure your onboarding process is easy, so fret not! And with the tremendous growth rate of MLOps with its increased adoption, who doesn’t want to be part of a vibrant community and have some contributions flowing along the MLOps space?</em></blockquote><h3>Quick Start</h3><p><strong>So here’s the deal 🤝. Solve a challenge and win cool swag!</strong></p><p>To get started:</p><ol><li>Sign up to Hacktoberfest <a href="https://hacktoberfest.digitalocean.com/register">here</a></li><li>Find all the relevant <a href="https://github.com/flyteorg/flyte/issues?q=is%3Aissue+is%3Aopen+label%3AHacktoberfest+">issues</a></li><li>Start creating pull requests</li><li>Fill out this <a href="https://tally.so/r/3qPO7m">form</a></li><li><a href="https://blog.flyte.org/meet-flyte-at-hacktoberfest-2021#grab-your-swag">Grab Your Swag</a> (<strong>there’s a special swag, too! 🎉</strong>)</li></ol><h3>Topics</h3><p>We have a couple of repos under Flyte that you can contribute to.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fairtable.com%2Fembed%2FshrirZb8gzTbZ4Bje&amp;display_name=Airtable&amp;url=https%3A%2F%2Fairtable.com%2FshrirZb8gzTbZ4Bje&amp;image=https%3A%2F%2Fstatic.airtable.com%2Fimages%2Foembed%2Fairtable.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=airtable" width="800" height="533" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d6fe1795bd334532a50f1d6035b0163/href">https://medium.com/media/6d6fe1795bd334532a50f1d6035b0163/href</a></iframe><p>Additionally, you can help us with*:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fairtable.com%2Fembed%2FshruavmLikvNorUzU&amp;display_name=Airtable&amp;url=https%3A%2F%2Fairtable.com%2FshruavmLikvNorUzU&amp;image=https%3A%2F%2Fstatic.airtable.com%2Fimages%2Foembed%2Fairtable.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=airtable" width="800" height="533" frameborder="0" scrolling="no"><a href="https://medium.com/media/891fbafa699a1220353f08def89474dd/href">https://medium.com/media/891fbafa699a1220353f08def89474dd/href</a></iframe><p>* You wouldn’t be receiving any Hacktoberfest swag for the additional topics. However, Flyte will recognize your contribution and give away some super cool gifts!</p><p><strong>Btw, feel free to create new issues and submit the concerning PRs. We admire creative minds! Make sure to add </strong><strong>Hacktoberfest label to your issue and PR title.</strong></p><h3>Grab Your Swag 🎁</h3><p>Every contribution is valued. <em>How?</em> We are going to give a shout-out to every contributor! 📣</p><p>And… let’s not forget the give-aways 🤣:</p><ul><li>Mugs</li><li>T-shirts</li><li>Vouchers</li><li>✨ <strong>The North Face Jackets</strong> ✨ (for real!)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QsYnMb5a7bRYv7dD" /></figure><h3>What do you have to do to claim swag?</h3><ul><li>2 merged/accepted pull requests to receive a mug</li><li>3+ merged/accepted pull requests or a video to receive a Flyte t-shirt</li><li>Full-fledged flytekit integration or a blog to receive a voucher or “The North Face” jacket.</li></ul><p><strong>The first 15 contributors will get a mug irrespective of the number of merged pull requests.</strong></p><p>Don’t forget to fill out <a href="https://tally.so/r/3qPO7m">this form</a> after you are done with either of the challenges, on or before October 31st, 2021.</p><p><strong>If your contribution adds value to Flyte heavily, we will customize your swag and add you as a Flyte contributor 🥳 !</strong></p><blockquote><em>Let the contributions be coming!</em></blockquote><h3>Rules</h3><ul><li>Refer to the <a href="https://docs.flyte.org/en/latest/community/contribute.html">Flyte contribution guidelines</a> before creating any pull request</li><li>Refer to the <a href="https://hacktoberfest.digitalocean.com/resources/participation">Hacktoberfest guidelines</a></li><li><a href="https://hacktoberfest.digitalocean.com/resources/qualitystandards">Quality is key</a></li><li>Mention the issue link in the PR you would be working on. Also, add Hacktoberfest to the title of your PR</li></ul><p>If you have any questions, don’t hesitate to chat with us on <a href="https://docs.google.com/forms/d/e/1FAIpQLScWPJZncyL-bBCWbuQ9HWuMzA8-r2RJc__CSWJoAYUQADN-BQ/viewform">Slack</a>. We have a <a href="https://flyte-org.slack.com/messages/C02FU3HJQE5">#hacktoberfest2021</a> channel that you can hop into!</p><p>Explore our <a href="https://github.com/flyteorg">GitHub repos</a> and give us a star ⭐️ <a href="https://github.com/flyteorg/flyte">here</a> if you like what you see 👀.</p><p>If you want to know what’s happening with Flyte, check out our <a href="https://twitter.com/flyteorg">Twitter</a> or Slack.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba5624ccceef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/union-ai/meet-flyte-at-hacktoberfest-2021-ba5624ccceef">Meet Flyte at Hacktoberfest 2021</a> was originally published in <a href="https://medium.com/union-ai">Union-ai</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>