<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[CodeFlare - Medium]]></title>
        <description><![CDATA[CodeFlare Simplifying the integration, scaling and acceleration of complex multi-step analytics and machine learning pipelines on the hybrid multi-cloud - Medium]]></description>
        <link>https://medium.com/codeflare?source=rss----830b19c217e0---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>CodeFlare - Medium</title>
            <link>https://medium.com/codeflare?source=rss----830b19c217e0---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:56:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/codeflare" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[InstaScale: Aggregate node scaler for guaranteed execution of AI workloads]]></title>
            <link>https://medium.com/codeflare/instascale-aggregate-node-scaler-for-guaranteed-execution-of-ai-workloads-9a065cf76627?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/9a065cf76627</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[openshift]]></category>
            <dc:creator><![CDATA[Abhishek Malvankar]]></dc:creator>
            <pubDate>Mon, 01 May 2023 22:01:03 GMT</pubDate>
            <atom:updated>2023-05-01T22:01:03.784Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>InstaScale: Aggregate node scaler for guaranteed execution of AI workloads</strong></p><p><em>Authors: Abhishek Malvankar, Alaa Youssef, Diana Arroyo, and Olivier Tardieu</em></p><p><strong>Introduction</strong></p><p><a href="https://thenewstack.io/kubernetes-evolution-from-microservices-to-batch-processing-powerhouse/">Kubernetes</a> continues to evolve to be the powerhouse for training (batch) workloads. Thanks to significant resource demands added by workloads such as Foundation models, many users turn to the cloud to train such resource-hungry models. Resource (node/pod) scaling becomes a critical aspect when moving to the cloud for patterns where workloads acquire (many) resources when needed and release them as soon as the workload is completed.</p><p><a href="https://research.ibm.com/blog/what-are-foundation-models">Foundation model</a> workloads typically need a group of nodes on which frameworks clusters like Spark, Ray, and Pytorch are spawned. The only way such framework cluster workloads can make progress is, if all the pod&#39;s resources (i.e. resources for head node and N worker nodes) needed are available, let&#39;s call it acquiring <em>gang (all-at-once) of resources</em>.</p><p>When capacity is not enough for running framework clusters additional resources (nodes) need to be acquired in a gang (all-at-once) fashion. Such functionality is needed to avoid starting partial jobs i.e. framework clusters with either only worker nodes or head nodes. This causes resource wastage leading to high cloud costs.</p><p>In this blog, we introduce a new controller called <a href="https://github.com/project-codeflare/instascale">InstaScale</a> which works with our no-code/low-code queuing system <a href="https://github.com/project-codeflare/multi-cluster-app-dispatcher">Multi-Cluster App dispatcher (MCAD)</a> to acquire aggregated resources for the target framework cluster (user job).</p><p><strong>Problem</strong></p><p>Training large AI models like the foundation model is experiment-driven where users submit multiple jobs to train the model and finally adopt the model to several downstream tasks. Infrastructure engineers or platform teams use a shared environment with bursting capability in the cloud where multiple users are onboarded and compete to run such experiments.</p><p>Assuming the shared environment is <a href="https://kubernetes.io/">Kubernetes</a>, it operates on pods. Kubernetes takes workloads to desired state only if a pod is spawned. A similar mechanism is used for scaling the cluster, the cluster autoscaler will add nodes if pods are in a pending state, and such pending pods are placed on the newly added nodes by the Kubernetes scheduler. Competing user&#39;s workloads can start partially in such a scenario, lets us understand this from the example below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DaP_C9vdwHIrWR5OXmTinQ.png" /><figcaption>Partial framework clusters (Jobs) launched for competing users</figcaption></figure><p>User#1 (blue) submits a workload that has 1 head node and 1 worker node and user# 2 (green) also submits another workload that has 1 head node and 1 worker node. Initially, all 4 pods will remain pending, this will cause cluster autoscaler to add nodes sequentially. As nodes become active/available in the Kubernetes cluster, the scheduler places pods on each of the nodes. The result is that both workloads have individual pods but none of them can make progress which leads to the hogging of resources and wastage.</p><p><strong>Solution</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RRPCXxyhOG1JuPHA" /><figcaption>InstaScale: Aggregate node scaler for guaranteed execution of AI workloads</figcaption></figure><p>InstaScale is a controller under <a href="https://github.com/project-codeflare">project-codeflare</a> that addresses the issue of partial job launch by bringing in aggregated resources needed for the entire target job on <a href="https://www.redhat.com/en/technologies/cloud-computing/openshift">OpenShift</a>. It provides a guarantee for running target user jobs. The acquired resources are tainted and labeled for a target workload by the InstaScale controller which allows us to hint scheduler to place the desired user workload pods.</p><p>For the same scenario of Two users (blue and green) mentioned in the <strong>Problem</strong> section, with deployed InstaScale you will observe the below behavior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NhX2bnSS39SdFah5SULTA.png" /><figcaption>InstaScale avoids partial frameworks cluster (jobs) submitted by competing users</figcaption></figure><p>All pods belonging to User#1 (blue) are allocated while the User#2 pods are not created since no resources are available. It was possible to schedule User#1 pods to the resources acquired with the help of labels and tainting. User#2 pods will be created as soon as User#1 pods are removed or run to completion. Scaling is done for users with quota limitations at the cluster level. Thus InstaScale will avoid resource wastage when both users compete for resources, reduce cloud, and scale within cluster quota limits.</p><p><em>Below are some of the key features:</em></p><ul><li>Proactive scaling before pending pod(s) is created.</li><li>Scaling for the entire Job.</li><li>Provides reuse of acquired resources for the next queued job.</li><li>Provides aggregated scale-down (<em>scale-down to zero</em>) of resources when no job needs previously acquired resources.</li><li>Provides quota limits per cluster for node scaling.</li></ul><p><strong>Summary</strong></p><p>We understood some limitations of the existing scaling mechanism and introduce InstaScale, a new controller that works with OpenShift to acquire aggregated nodes for guaranteed workload execution and avoid resource wastage causing high cloud costs. Please reach out to us with any questions on the InstaScale GitHub <a href="https://github.com/project-codeflare/instascale">repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a065cf76627" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/instascale-aggregate-node-scaler-for-guaranteed-execution-of-ai-workloads-9a065cf76627">InstaScale: Aggregate node scaler for guaranteed execution of AI workloads</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Packing workloads on AI supercomputer in the cloud]]></title>
            <link>https://medium.com/codeflare/packing-workloads-on-ai-supercomputer-in-the-cloud-b4c3d9a3e570?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/b4c3d9a3e570</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[openshift]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Abhishek Malvankar]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 13:21:41 GMT</pubDate>
            <atom:updated>2023-03-15T17:02:23.017Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Authors: Abhishek Malvankar, Alaa Youssef, Asser Tantawi, Olivier Tardieu, and Carlos Costa</em></p><p><strong>Introduction</strong></p><p>To cope with the massive compute requirements for training foundation models, IBM has built its first cloud-native supercomputer, <a href="https://research.ibm.com/blog/AI-supercomputer-Vela-GPU-cluster">Vela</a>. Housed within IBM Cloud, it is currently being used mainly by the IBM Research community. Vela is built using a cloud-native software stack and runs on the <a href="https://www.redhat.com/en/technologies/cloud-computing/openshift">Red Hat OpenShift</a> platform, leveraging Kubernetes capabilities for container orchestration.</p><p>Fragmentation is a phenomenon where one or more (possibly expensive) resources are used inefficiently (across nodes) in a cluster. AI workloads typically need expensive accelerators like GPU(s) to run the computation. Wastage of such resources will lead to low cluster utilization and high costs.</p><p>In this article, we describe an issue we faced with the fragmentation of resources in the system, explain Kubernetes scheduler configurations that were made for large-scale GPU-based OpenShift cluster Vela to drive utilization with help of packing (at GPU dimension) that can result in a speedup of workload execution and potential cost savings in the cloud environment.</p><p>In future articles will describe more advanced scheduling techniques to address such issues in a more efficient way.</p><p><strong>Configuration</strong></p><p>The <a href="https://en.wikipedia.org/wiki/Bin_packing_problem">bin packing</a> problem is an optimization problem, in which items of different sizes must be packed into a finite number of bins or containers, each of a fixed given capacity, in a way that minimizes the number of bins used.</p><p>Kubernetes scheduler is a highly configurable system that provides various extension points to configure scheduling behavior. The default policy is “LeastAllocated” for the plugin “NodeResourcesFit”, use case for such a policy would be to enable reliability and HA configuration of microservices as it ensures a single node failure cannot takedown every replica at once. For AI workloads the “LeastAllocated” policy does not work, especially for training workloads. Spreading the workload instead of packing increases latency to execute workloads in the communication phase.</p><p>Let’s consider the below case where spread policy can cause underutilization of nodes, Cluster has 2 nodes with 8 GPUs each. Each node has 4 GPUs used, if the next job that requests 8 GPUs on a single node arrives then none of the nodes have 8 free GPUs available due to fragmentation as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*koadtO3YAfeTvBUHhOqlOQ.png" /><figcaption>8 GPU workload remains pending as none of the nodes have 8 GPUs available as an aggregate</figcaption></figure><p>Kubernetes scheduler provides several <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/resource-bin-packing/">scoring functions</a>. The scoring function is configurable and users can define the score function corresponding to different utilization rates to determine the decision-making process of scheduling.</p><p>Users (admins) at Vela had to update the scheduler configmap as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*XtA_e1NkHbJiBqiKn0FrYg.png" /><figcaption>Figure 1: ConfigMap configuration</figcaption></figure><p>The above policy would mean that the scheduler performs packing at the GPU dimension with a linear scoring function. Nodes with higher resource utilization get higher scores. If the utilization rate is 0 then the score is 0 and if the resource utilization rate is 100 then the score is 10. Shown below is the plotted policy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*sSWFcthhUbzePcSdMcwoCQ.png" /><figcaption>Figure 2: Packing on GPU dimension</figcaption></figure><p><strong>Deployment</strong></p><ul><li>Backup existing configmap using the command:</li></ul><pre>oc get configmap scheduler-config -n scheduler-plugins -o yaml &gt; prev-cm-coschd.yaml</pre><ul><li>Admin needs to edit/patch scheduler config using the configuration shown in Figure 1 and execute the command:</li></ul><pre>oc apply -f scheduler-configmap-packing.yaml</pre><ul><li>Verify the updates applied to the configmap in “.data” section using the command:</li></ul><pre>oc get configmap scheduler-config -n scheduler-plugins -o yaml</pre><ul><li>Scale down the deployment of the co-scheduler to zero using the command:</li></ul><pre>oc scale deployment scheduler-plugins-scheduler -n scheduler-plugins –replicas=0</pre><ul><li>After the scale-down completes scale up the scheduler deployment using the command:</li></ul><pre>oc scale deployment scheduler-plugins-scheduler -n scheduler-plugins –replicas=1</pre><p><strong>Testing</strong></p><p>The Scheduler Plugins <a href="https://github.com/kubernetes-sigs/scheduler-plugins">repository</a> provides various extensions of the default scheduler. We use the co-scheduler from the same repository to spawn multiple pods in a group called “podgroups”. This testing can also be done with the default scheduler but we feel the co-scheduler plugin is more suitable for AI workloads as it provides gang scheduling i.e., groups of pods tied to the workload can be scheduled together in the system.</p><p>Below are the steps executed in the test environment to enable packing:</p><ul><li>Create podgroup with name packing-test-1-pg with minMember as 2 and batchv1/Job that spawns 3 jobs (completions == parallelism ==3)</li><li>Submit the below test yaml using the command:</li></ul><pre>oc apply -f packing-test-1.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*jLPlphYMe1WCEVPIYwIxpQ.png" /><figcaption>Figure 3: packing-test-1.yaml</figcaption></figure><ul><li>Submit the second job using the command:</li></ul><pre>oc apply -f packing-test-2.yaml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*d8D6SUyzMTxu3sF_kXtZLA.png" /><figcaption>Figure 4: packing-test-2.yaml</figcaption></figure><p>Binpacking will cause both jobs to be landed on the same node:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/467/1*c6-vKtu8nh0mVws7gRASrw.png" /><figcaption>Both test jobs on the same node named ip-10–0–134–124</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4c3d9a3e570" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/packing-workloads-on-ai-supercomputer-in-the-cloud-b4c3d9a3e570">Packing workloads on AI supercomputer in the cloud</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless stream processing with Rayvens on Ray.io]]></title>
            <link>https://medium.com/codeflare/serverless-stream-processing-with-rayvens-on-ray-io-a8d804661109?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8d804661109</guid>
            <category><![CDATA[event-streaming]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Gheorghe-Teodor Bercea]]></dc:creator>
            <pubDate>Thu, 09 Dec 2021 21:12:36 GMT</pubDate>
            <atom:updated>2021-12-09T21:12:35.959Z</atom:updated>
            <content:encoded><![CDATA[<p>Rayvens seamlessly integrates <a href="http://ray.io/">Ray.io</a>, <a href="https://camel.apache.org/">Apache Camel</a>, and <a href="https://kafka.apache.org/">Kafka</a><br>to empower data scientists to build scalable stream processing applications on the cloud with easy access to hundreds of data/event sources and sinks.</p><p>In previous posts we introduced <a href="https://medium.com/codeflare/rayvens-event-sources-sinks-and-event-driven-programming-with-ray-b229794dc41f">Rayvens eventing</a> for Ray.io, its <a href="https://medium.com/codeflare/rayvens-event-streams-da0511b23e9a">programming model</a>, and its <a href="https://medium.com/codeflare/accessing-hundreds-of-event-sources-and-sinks-with-rayvens-17f3ff45d92">catalog of data sources and sinks</a>.</p><p>In this post we show how Rayvens can stream events using Kafka as the event transport while leveraging Ray’s scaling capabilities to enable the efficient and scalable parallel processing of large event quantities.</p><p>Rayvens generalizes the steps described in Anyscale “<a href="https://www.anyscale.com/blog/serverless-kafka-stream-processing-with-ray">Serverless Kafka Stream Processing with Ray</a>” blog to the point that the source and sink in Anyscale’s example code can be easily replaced with any of the hundreds of sources and sinks provided by Camel. Our example application below receives real-time cryptocurrency data from <a href="https://www.binance.com/en">Binance</a>. It processes the data producing messages that are pushed to a <a href="https://slack.com/">Slack</a> channel.</p><h3>The event flow</h3><p>We want to process a <strong>large number of cryptocurrency price events</strong>. The user is responsible for providing a list of cryptocurrencies to be monitored, for this example we will use a list of two coins: [&quot;BTC&quot;, &quot;ETH&quot;]. The real-time prices will be provided by <strong>Binance</strong>. The prices will be formatted and forwarded to a <strong>Slack</strong> channel for user inspection if above a certain threshold.</p><p>Kafka will be used to stream the events between:</p><ul><li><strong>Binance</strong> and the processing tasks</li><li>the tasks and <strong>Slack</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jVEvpd1BYmpkJeVCkEKvqg.png" /><figcaption>Event processing flow</figcaption></figure><p>The Rayvens programming model can readily express the event flow above using its concepts of:</p><ul><li><strong>event source</strong> for receiving events from Binance</li><li><strong>event sink</strong> for outputting events to Slack</li><li><strong>event processor</strong> for processing the events</li><li><strong>event transport</strong> for the Kafka communication to/from the main task, automatic creation of Kafka topics and provisioning of Kafka event Consumers and Producers</li></ul><p>Rayvens employs the concept of <strong>event streams</strong> upon which all Rayvens events sources, sinks, and processors operate. A detailed description of event streams can be found <a href="https://medium.com/codeflare/rayvens-event-streams-da0511b23e9a">here</a> but for the purposes of this example a single stream object is needed:</p><pre>stream = rayvens.Stream(&#39;kafka-eventing&#39;)</pre><h4>Creating the Binance event source</h4><p>With the stream in place we can <strong>configure</strong> a source and then <strong>add</strong> it to the stream:</p><pre>source_config = dict(<br>      kind=&#39;binance-source&#39;,<br>      coin=[&quot;BTC&quot;, &quot;ETH&quot;]<br>)</pre><pre>stream.add_source(source_config)</pre><h4><strong>Creating the Slack event sink</strong></h4><p>Configuring a sink and adding it to the stream is very simple:</p><pre>sink_config = dict(<br>      kind=&#39;slack-sink&#39;,<br>      channel=slack_channel,<br>      webhookUrl=slack_webhook<br>)</pre><pre>stream.add_sink(sink_config)</pre><p>For setting up a <strong>channel</strong> and <strong>webhook URL</strong> for Slack please consult the <a href="https://slack.com/intl/en-ro/help/articles/115005265063-Incoming-webhooks-for-Slack">Slack API</a>.</p><h4>Creating the event processor</h4><p>We can use a plain Python function to express the event processor:</p><pre>def process_currency_price(event):<br>    parsed_event = json.loads(event)<br>    currency = parsed_event[&#39;instrument&#39;].split(&quot;/&quot;)[0]<br>    price = parsed_event[&#39;last&#39;]<br>    return f&quot;{currency} : {price}&quot;</pre><pre>stream.add_operator(process_currency_price)</pre><p>Note: The value returned by the processor represents the processed event content and will be put back into the event stream and forwarded to all sinks attached to the event stream.</p><h4>Event transport</h4><p>Rayvens, by default, uses HTTP for propagating events. To use Kafka, use the transport=”kafka” option explicitly when initializing Rayvens:</p><pre>rayvens.init(transport=&quot;kafka&quot;)</pre><p>Rayvens will automatically create:</p><ul><li>the two topics for the source and sink communication</li><li>the Kafka Consumer object for ingesting incoming events from the Kafka topic responsible for handling incoming Binance events</li><li>the Kafka Producer object for publishing outgoing events to the Kafka topic used for communication with Slack</li></ul><p>With these elements in place the event flow is ready:</p><pre># Initialize Rayvens:<br>rayvens.init(transport=&quot;kafka&quot;)</pre><pre># Create an event stream:<br>stream = rayvens.Stream(&#39;kafka-eventing&#39;)</pre><pre># Add sink:<br>sink_config = dict(<br>      kind=&#39;slack-sink&#39;,<br>      channel=slack_channel,<br>      webhookUrl=slack_webhook<br>)<br>stream.add_sink(sink_config)<br></pre><pre># Add event processor:<br>def process_currency_price(event):<br>    parsed_event = json.loads(event)<br>    currency = parsed_event[&#39;instrument&#39;].split(&quot;/&quot;)[0]<br>    price = parsed_event[&#39;last&#39;]<br>    return f&quot;{currency} : {price}&quot;</pre><pre>stream.add_operator(process_currency_price)</pre><pre># Add event source:<br>source_config = dict(<br>      kind=&#39;binance-source&#39;,<br>      coin=[“BTC”, “ETH”]<br>)<br>stream.add_source(source_config)<br></pre><p>But we are not done yet, what about scaling?</p><h3>Scaling the event flow using Ray</h3><p>As written above the event flow will not scale which might be fine for testing purposes but not so much for a real application with a <strong>compute intensive processing step</strong>.</p><p>We want the processing step to be distributed and scale and Rayvens can do this with <strong>two</strong> simple changes.</p><p>Rayvens is based on <strong>Ray</strong> and supports event processors that are <strong>Ray tasks</strong>, <strong>Ray actors</strong> and <strong>Ray actor methods</strong>.</p><p><strong>Change 1:</strong> Since price events are handled independently the best representation for our processor is a <strong>Ray task. </strong>This can be achieved by annotating the plain Python function we used as processor before with @ray.remote:</p><pre>@ray.remote<br>def process_currency_price(event):<br>    ...</pre><p>Simply using a task is not enough.</p><p><strong>Remaining issues:</strong> As written, the flow only has <strong>one</strong> Kafka Consumer. Moreover, the Kafka topic that is used for incoming Binance events, by default, is not partitioned which means that <strong>only one Kafka event queue </strong>is being created.</p><p>What we would like is to have <strong>several event queues</strong> feeding into <strong>several Kafka Consumers</strong> forwarding events to <strong>several Ray Tasks:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YpyofPcLco1bOFHmySfcrg.png" /><figcaption>Scaling event processing flow</figcaption></figure><p><strong>Change 2: </strong>Rayvens supports <strong>automatic partitioning of Kafka topics</strong> for event sources via the source kafka_transport_partitions configuration option:</p><pre>source_config = dict(<br>      ...<br>      kafka_transport_partitions=3<br>)</pre><p><strong>Behind the scenes</strong>: Rayvens automatically wraps the Kafka Consumer into a Ray Actor with low CPU usage num_cpus=0.05 and creates a <strong>pool</strong> of these Kafka Actor Consumers equal to the number of topic partitions. Each Kafka Actor Consumer will then call the user-provided processor (in this case a Ray Task) to process the event. The outcome will be sent to Slack via a single Kafka Producer.</p><p>And we’re done!</p><p><a href="https://github.com/project-codeflare/rayvens/blob/main/examples/blogs/binance_to_slack.py">Link to the full code.</a></p><h3>Using other sources and sinks</h3><p>Rayvens allows the types of sources and sinks used in the example above to be replaced. Simply replace the configurations of the Binance source or Slack sink with the configuration for other other source/sink types.</p><p>Cloud Object Storage source:</p><pre>source_config = dict(kind=&#39;cloud-object-storage-source&#39;,<br>                     bucket_name=&lt;bucket_name&gt;,<br>                     access_key_id=&lt;...&gt;,<br>                     secret_access_key=&lt;...&gt;,<br>                     endpoint=&lt;...&gt;)</pre><p>Cloud Object Storage sink:</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   bucket_name=&lt;bucket_name&gt;,<br>                   access_key_id=&lt;...&gt;,<br>                   secret_access_key=&lt;...&gt;,<br>                   endpoint=&lt;...&gt;,<br>                   file_name=&lt;file_name&gt;)</pre><p>Actually any other sources and sinks Rayvens supports (and there are hundreds) can be used to replace the sources and sinks used in the example code. See the Rayvens sources and sinks blog linked below.</p><h3>Related blog posts</h3><ul><li><a href="https://medium.com/codeflare/accessing-hundreds-of-event-sources-and-sinks-with-rayvens-17f3ff45d92">Rayvens sources and sinks blog</a>.</li><li>Anyscale’s <a href="https://www.anyscale.com/blog/serverless-kafka-stream-processing-with-ray">“</a><a href="https://www.anyscale.com/blog/serverless-kafka-stream-processing-with-ray">Serverless Kafka Stream Processing with Ray</a>” blog. Rayvens automates the process described in this blog and generalizes it to the point that any Camel source/sink can be used interchangeably.</li><li><a href="https://medium.com/codeflare/event-driven-data-processing-with-ibm-cloud-object-storage-and-ray-io-7838a344f4b6">Cloud object storage sources and sinks blog</a>.</li></ul><p>Authors: <a href="https://medium.com/u/b606edb74928">Gheorghe-Teodor Bercea</a> and <a href="https://medium.com/u/4ae0218b5e8">Olivier Tardieu</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNWKKPqxV3rkwjsoVAk0PQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8d804661109" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/serverless-stream-processing-with-rayvens-on-ray-io-a8d804661109">Serverless stream processing with Rayvens on Ray.io</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-driven data processing with IBM Cloud Object Storage and Ray.io]]></title>
            <link>https://medium.com/codeflare/event-driven-data-processing-with-ibm-cloud-object-storage-and-ray-io-7838a344f4b6?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/7838a344f4b6</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[event-sources-and-sinks]]></category>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[distributed-computing]]></category>
            <category><![CDATA[apache-camel]]></category>
            <dc:creator><![CDATA[Gheorghe-Teodor Bercea]]></dc:creator>
            <pubDate>Tue, 12 Oct 2021 14:41:04 GMT</pubDate>
            <atom:updated>2021-10-12T14:41:04.590Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Jointly written by </em><a href="https://medium.com/u/b606edb74928"><em>Gheorghe-Teodor Bercea</em></a><em> and </em><a href="https://medium.com/u/4ae0218b5e8"><em>Olivier Tardieu</em></a><em>.</em></p><p>In a <a href="https://medium.com/codeflare/rayvens-event-sources-sinks-and-event-driven-programming-with-ray-b229794dc41f">previous blog</a> we introduced <a href="https://github.com/project-codeflare/rayvens">Rayvens</a>, a library for bridging <a href="https://www.ray.io/">Ray.io</a> with the vast collection of event and data services supported by <a href="https://camel.apache.org/">Apache Camel</a>. We showcased Rayvens <a href="https://medium.com/codeflare/rayvens-event-streams-da0511b23e9a">event streams</a>, and configured simple <a href="https://medium.com/codeflare/accessing-hundreds-of-event-sources-and-sinks-with-rayvens-17f3ff45d92">event sources and sinks</a>.</p><p>In this post we leverage Rayvens to produce and consume data (and metadata) hosted in <a href="https://www.ibm.com/cloud/object-storage">IBM Cloud Object Storage</a> or <a href="https://aws.amazon.com/s3/">AWS S3</a> buckets.</p><h3>Getting started</h3><p>First, install Rayvens following the instructions provided <a href="https://github.com/project-codeflare/rayvens">here</a>.</p><p>To access an IBM Cloud Object Storage or AWS S3 bucket, Rayvens requires:</p><ul><li>the bucket name,</li><li>the access key id,</li><li>the secret access key,</li><li>the public or private endpoint.</li></ul><h3>Cloud Object Storage Source</h3><p>The Cloud Object Storage source monitors a single bucket. It is configured as follows:</p><pre>source_config = dict(kind=&#39;cloud-object-storage-source&#39;,<br>                     bucket_name=&lt;bucket_name&gt;,<br>                     access_key_id=&lt;...&gt;,<br>                     secret_access_key=&lt;...&gt;,<br>                     endpoint=&lt;...&gt;)</pre><p>The bucket_name, access_key_id, secret_access_key, and endpoint are required configuration parameters. The source configuration supports additional optional configuration parameters discussed below.</p><p>When a file is uploaded to the bucket, the source produces an event on the Rayvens event stream attached to the source. This event has the following JSON structure:</p><pre>{<br>   &quot;filename&quot;: &lt;filename_value&gt;,<br>   &quot;body&quot;: &lt;body_value&gt;<br>}</pre><p>The file is automatically deleted from the bucket upon firing the event.</p><h4>region</h4><p>The region of the Cloud Storage instance if not specified defaults tous-east. To specify a different region use:</p><pre>source_config = dict(kind=&#39;cloud-object-storage-source&#39;,<br>                     ...<br>                     region=&#39;eu-de&#39;)</pre><h4>move_after_read</h4><p>The move_after_read option makes it possible to preserve the files triggering the events by moving them to an alternate bucket instead of deleting them.</p><pre>source_config = dict(kind=&#39;cloud-object-storage-source&#39;,<br>                     ...<br>                     move_after_read=&lt;alternate_bucket_name&gt;)</pre><p>The bucket needs to be created in advance in Cloud Object Storage.</p><h4>meta_event_only</h4><p>To only propagate the file name use the meta_event_only option:</p><pre>source_config = dict(kind=&#39;cloud-object-storage-source&#39;,<br>                     ...<br>                     meta_event_only=True)</pre><p>The JSON structure of the produced event is:</p><pre>{<br>   &quot;filename&quot;: &lt;filename_value&gt;<br>}</pre><p>Combine the meta_event_only and move_after_read options to be notified of new files but pull file contents on demand.</p><h3>Cloud Object Storage Sink</h3><p>The Cloud Object Storage sink makes it possible to upload one or several files to a bucket. When a file is uploaded to a Cloud Object Storage bucket, a new file entry will be created unless a file with the same name already exists in the bucket in which case the existing file will be overwritten with the new file.</p><p>All instances of the sink require the following configuration parameters:</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   bucket_name=&lt;bucket_name&gt;,<br>                   access_key_id=&lt;...&gt;,<br>                   secret_access_key=&lt;...&gt;,<br>                   endpoint=&lt;...&gt;,<br>                   ...)</pre><p>Other parameters define what, when, and how to upload the data to Cloud Object Storage.</p><h4>Uploading a file from an in-memory buffer</h4><p>The sink configuration may specify a file name.</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   ...<br>                   file_name=&lt;file_name&gt;)</pre><p>In this configuration, the content of the file is the event payload itself.</p><p>Add the sink to an existing event stream:</p><pre>stream.add_sink(sink_config)</pre><p>Dump the file contents onto the stream:</p><pre>stream &lt;&lt; &#39;File contents.&#39;</pre><p>This works well for small to medium files.</p><h4>Uploading one or several files from the file system</h4><p>For larger files, multi-part uploads are recommended. Pass the upload_type=&#39;multi-part&#39; option to the sink configuration and optionally specify the part size by adding part_size=&lt;number_of_bytes&gt; to the configuration. By default the part size is 10MB.</p><p>For example:</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   ...<br>                   upload_type=&#39;multi-part&#39;,<br>                   part_size=2 * 1024 * 1024)</pre><p>This configuration will upload a file in 2MB chunks.</p><p>In the case of a multi-part file upload, there are several ways to specify files to be uploaded.</p><p><strong>Method 1:</strong> A file and its content can be specified from within the application using file_name=&lt;file_name&gt; and stream &lt;&lt; &#39;File contents&#39; method.</p><p><strong>Method 2: </strong>Files can also be specified by adding a Cloud Object Storage sink to an event stream and then passing the following to the stream:</p><pre>stream &lt;&lt; Path(&#39;test_files/test.txt&#39;)</pre><p>This requires Path to be imported: from pathlib import Path.</p><p>The event will be treated as a path to a file rather than the file contents. The file will be uploaded using its original name unless the file_name option is provided to override it.</p><p><strong>Method 3: </strong>Files can be uploaded using the from_file option. For example:</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   ...<br>                   upload_type=&#39;multi-part&#39;,<br>                   part_size=2 * 1024 * 1024,<br>                   from_file=&#39;test_files/test3.txt&#39;)</pre><p>Every time a file named test3.txt is available in the test_files directory, the file will be uploaded to Cloud Object Storage.</p><p><strong>Method 4: </strong>Files can be uploaded using thefrom_directory option:</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   ...<br>                   upload_type=&#39;multi-part&#39;,<br>                   part_size=2 * 1024 * 1024,<br>                   from_directory=&#39;test_files/&#39;)</pre><p>Any file added to the test_files directory will be uploaded to Cloud Object Storage under its own name. Existing files will be silently overwritten.</p><p>For Methods 3 and 4, the keep_file option ensures the files are kept on the file system but their location is changed. After upload the files are moved to an hidden subdirectory of the directory they were uploaded from.</p><h3>Conclusion</h3><p>This post showcases the practical ways in which Rayvens can bridge Cloud Object Storage and Ray.io. The Cloud Object Storage source monitors a bucket and produces an event whenever a file is added to a bucket. The event includes the file name and if desired the file content. The Cloud Object Storage sink supports various methods for uploading files to a bucket. At this time only multi-part uploads are supported with streaming uploads coming soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNWKKPqxV3rkwjsoVAk0PQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7838a344f4b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/event-driven-data-processing-with-ibm-cloud-object-storage-and-ray-io-7838a344f4b6">Event-driven data processing with IBM Cloud Object Storage and Ray.io</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AIX360 meets CodeFlare: How to scale explainability using CodeFlare pipelines]]></title>
            <link>https://medium.com/codeflare/aix360-meets-codeflare-how-to-scale-explainability-using-codeflare-pipelines-c6bd8a7b0090?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/c6bd8a7b0090</guid>
            <category><![CDATA[hybrid-cloud]]></category>
            <category><![CDATA[ray-project]]></category>
            <category><![CDATA[codeflare]]></category>
            <category><![CDATA[aix360]]></category>
            <category><![CDATA[scaling-ai]]></category>
            <dc:creator><![CDATA[Raghu K Ganti]]></dc:creator>
            <pubDate>Wed, 29 Sep 2021 21:21:53 GMT</pubDate>
            <atom:updated>2021-09-29T21:21:53.162Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Authors: Carlos Costa, Amit Dhurandhar, Raghu Ganti, Mudhakar Srivatsa, Kush R. Varshney</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uO_yyB7JaM_mqxrSeux-Aw.jpeg" /><figcaption>AIX360 meets CodeFlare (credit: @bfigas Unsplash, Jun 2019)</figcaption></figure><p>Explainability of AI models helps users gain insight into a machine learning model’s decision-making process. Such understanding provides key business insights and is essential to fostering trust and confidence in AI systems. <a href="https://github.com/Trusted-AI/AIX360">AI Explainability 360</a> (AIX360) is a comprehensive open-source toolkit of state-of-the-art algorithms that support interpretability and explainability of machine learning models donated by IBM to the Linux Foundation AI &amp; Data. As explainability adoption increases and amount of data on which these explanations need to be provided increases, the explanation algorithms face scaling challenges.</p><p><a href="https://github.com/project-codeflare/codeflare">CodeFlare</a> is IBM’s open-source toolkit that targets the reduction of time to setup, run, and scale machine-learning tasks. It is built on top of <a href="https://github.com/ray-project/ray">Ray</a>, an emerging open-source distributed computing framework for machine learning applications. CodeFlare extends the capabilities of Ray by adding specific elements to make scaling workflows easier.</p><p>With explainability, the bottleneck step typically is the ability to scale out multiple instance explanations, which is an embarrassingly parallel workload and the lowest hanging fruit for us to target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*KWheXlGqtqegkfs5JbN5bg.jpeg" /><figcaption>Low hanging fruit (credit: @yirage Unsplash, July 6, 2018)</figcaption></figure><p>CodeFlare pipelines (v0.1.2) released a new feature enabling the “splitting” of a numpy array or pandas dataframe into multiple objects (very similar to what Dask on Ray provides for partitioning). This enables the scaling of compute intensive workloads to be spread across multiple processes. With this feature and the ability to wrap the local explanation into a simple “Estimator” or OR node of CF pipelines, we can achieve this scaling.</p><p>Below, we demonstrate the code snippet that wraps the <em>LocalBBExplainer</em> (from AIX360) into an <em>Estimator</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4edcdd73feeb5446064e35f6ab0888e2/href">https://medium.com/media/4edcdd73feeb5446064e35f6ab0888e2/href</a></iframe><p>The data is split 4-way as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bcb647c7817d9a8146325b48a020b516/href">https://medium.com/media/bcb647c7817d9a8146325b48a020b516/href</a></iframe><p>With the input being 4 objects to a simple one-node pipeline, we are able to scale it up to 4 ways using CodeFlare pipelines. We observe a 4x linear speedup when using this approach on the standard example from AIX360. The ability to scale AIX360 explanations with a simple wrapper and using the <em>splitter</em> makes it easy for the data scientist to provide explanations for large tabular data and scale it to clusters with simple code changes.</p><p>The notebook in its full is available <a href="https://github.com/project-codeflare/codeflare/blob/develop/notebooks/AIX360%20meets%20CodeFlare.ipynb">here</a>. The reader is invited to try out <a href="https://github.com/project-codeflare/codeflare">CodeFlare</a> and <a href="https://github.com/Trusted-AI/AIX360">AIX360</a>. Happy coding and scaling!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6bd8a7b0090" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/aix360-meets-codeflare-how-to-scale-explainability-using-codeflare-pipelines-c6bd8a7b0090">AIX360 meets CodeFlare: How to scale explainability using CodeFlare pipelines</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accessing hundreds of event sources and sinks with Rayvens]]></title>
            <link>https://medium.com/codeflare/accessing-hundreds-of-event-sources-and-sinks-with-rayvens-17f3ff45d92?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/17f3ff45d92</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[events]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[apache-camel]]></category>
            <dc:creator><![CDATA[Gheorghe-Teodor Bercea]]></dc:creator>
            <pubDate>Mon, 23 Aug 2021 15:10:22 GMT</pubDate>
            <atom:updated>2021-08-23T15:10:22.604Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Jointly written by </em><a href="https://medium.com/u/b606edb74928"><em>Gheorghe-Teodor Bercea</em></a><em> and </em><a href="https://medium.com/u/4ae0218b5e8"><em>Olivier Tardieu</em></a><em>.</em></p><p>In a <a href="https://medium.com/codeflare/rayvens-event-sources-sinks-and-event-driven-programming-with-ray-b229794dc41f">previous blog</a> we introduced <a href="https://github.com/project-codeflare/rayvens">Rayvens</a>. Rayvens combines event-driven programming (<a href="https://camel.apache.org/">Apache Camel</a>) and AI (<a href="https://ray.io/">Ray.io</a>). We followed with a more detailed description of Rayvens <a href="https://medium.com/codeflare/rayvens-event-streams-da0511b23e9a">event streams</a>. In this blog post we dive into the details of the Rayvens event sources and sinks. We will cover:</p><ol><li>General principles: configuring, attaching, and detaching sources and sinks to/from event streams</li><li>Built-in catalog of sources and sinks</li><li>Programming more sources and sinks using Camel’s YAML dialect</li></ol><p>Sources and sinks are configured by means of Python dictionaries, for example:</p><pre>source_config = dict(kind=&#39;http-source&#39;,<br>                     url=&lt;source_url&gt;,<br>                     period=3000)</pre><p>Each configuration requires a kind and, depending on this kind, other fields will be mandatory or optional. For example, the http-source periodically fetches data from a REST API. The configuration of the url to get data from is required whereas the period is optional.</p><p>Typical sources and sinks are configured by means of simple values (numbers, strings, Boolean flags). In contrast, the generic-source and generic-sink make it possible to use Camel’s YAML dialect to program sophisticated sources and sinks. They accept a single spec field that specifies the program for the source or sink to run.</p><h3>Attaching a source or sink to a stream</h3><p>A source or sink is attached to an event stream using add_source or add_sink respectively:</p><pre>source = stream.add_source(source_sonfig)<br>sink = stream.add_sink(sink_config)</pre><p>An event source produces events and appends these events to the stream the source is added to. An event sink is subscribed to the stream it is attached to. It accepts all events emitted by the stream once it is attached to the stream and forwards these events (typically in order) to an external data service.</p><p>Keep in mind that sources and sinks typically take a few seconds to initialize and start producing or accepting events after the invocation of add_source or add_sink. Multiple sinks attached to the same stream eventually accept the same events.</p><p>As explained in prior blogs, a stream by default emits all the events appended to it in order, hence forwards all events from its sources to its sinks (after initialization). Stream operators can be used to filter or transform events between sources and sinks. A stream can forward events to multiple streams for instance to partition events across multiple sinks.</p><h3>Detaching a source or sink from a stream</h3><p>The return value of add_source or add_sink makes it possible to detach a source or sink from the stream it is attached to:</p><pre>stream.disconnect_source(source)<br>stream.disconnect_sink(sink)</pre><p>To detach all sources and sinks from a stream invoke:</p><pre>stream.disconnect_all()</pre><p>Disconnecting a source or sink from a stream does not preempt events already queued for processing on the stream.</p><h3>Rayvens built-in sources</h3><p>Rayvens offers built-in sources and sinks for popular services. A comprehensive list can be found <a href="https://github.com/project-codeflare/rayvens/blob/main/docs/connectors.md">here</a>. These sources and sinks expose a curated set of options that the user can configure. A summary of built-in sources follows.</p><h4>HTTP source</h4><p>This source makes an HTTP GET request to url every period milliseconds producing for each request an event consisting of the response body:</p><pre>source_config = dict(kind=&#39;http-source&#39;,<br>                     url=&lt;url_value&gt;,<br>                     period=&lt;optional_period_value&gt;)</pre><h4>Kafka source</h4><p>This source receives messages from a Kafka topic. It requires the topic name as well as a comma-separated list of Kafka brokers. See the <a href="https://github.com/project-codeflare/rayvens/blob/main/docs/connectors.md#kindkafka-source">full documentation</a> for additional parameters such as Kafka credentials.</p><pre>source_config = dict(kind=&#39;kafka-source&#39;,<br>                     topic=&lt;topic&gt;,<br>                     brokers=&lt;kafka_brokers&gt;)</pre><h4>Telegram source</h4><p>This source receives all the incoming messages from a Telegram Bot.</p><pre>source_config = dict(kind=&#39;telegram-source&#39;,      <br>                     authorization_token=&lt;token&gt;)</pre><h4>Binance source</h4><p>This source receives cryptocurrency status events from the Binance API every period milliseconds. This source allows the monitoring of several cryptocurrencies.</p><pre>source_config = dict(kind=&#39;binance-source&#39;,<br>                     coin=&lt;list_of_coin_IDs&gt;,<br>                     period=&lt;optional_period_value&gt;)</pre><p>Example list of coin IDs:</p><pre>[&#39;BTC&#39;, &#39;ETH&#39;]</pre><h4>Cloud Object Storage source</h4><p>This source receives objects from <a href="https://www.ibm.com/cloud/object-storage">IBM Cloud Object Storage</a> or <a href="https://aws.amazon.com/s3/">AWS S3</a>. For example, it can be configured to produce an event for each object initially present or later added to a bucket:</p><pre>source_config = dict(kind=&#39;cloud-object-storage-source&#39;,<br>                     bucket_name=&lt;bucket&gt;,<br>                     access_key_id=&lt;access_key_id&gt;,<br>                     secret_access_key=&lt;secret_access_key&gt;,<br>                     endpoint=endpoint)</pre><p>Several other configuration options exist and will be covered in a separate blog post.</p><p>The events received by the stream from this source have the following JSON format:</p><pre>{ &quot;filename&quot;: &lt;filename&gt;, &quot;body&quot;: &lt;body&gt; }</pre><h4>File source</h4><p>This source reads a file from the file system when the file becomes available.</p><pre>source_config = dict(kind=&#39;file-source&#39;,<br>                     path=&#39;path/to/file&#39;,<br>                     keep_files=True)</pre><p>When keep_files is true the file will be moved into a hidden subfolder and not deleted after being read which is the default behavior.</p><h4>File watch source</h4><p>This source monitors a directory for file-related events: file modification (MODIFY), file deletion (DELETE), and file creation (CREATE). A subset of these event types to monitor can be specified. By default all events are monitored.</p><pre>source_config = dict(kind=&#39;file-watch-source&#39;,<br>                     path=&#39;path/to/directory&#39;,<br>                     events=&#39;DELETE,CREATE&#39;)</pre><p>The event received by the stream from this source has the following JSON format:</p><pre>{ &quot;filename&quot;: &lt;filename&gt;, &quot;event_type&quot;: &lt;event_type&gt; }</pre><h3>Rayvens built-in sinks</h3><h4>Slack sink</h4><p>Output a message to Slack.</p><pre>sink_config = dict(kind=&#39;slack-sink&#39;,<br>                   channel=&lt;slack_channel_name&gt;,<br>                   webhook_url=&lt;slack_webhook&gt;)</pre><h4>Kafka sink</h4><p>Publish events to a Kafka topic.</p><pre>sink_config = dict(kind=&#39;kafka-sink&#39;,<br>                   topic=&lt;topic&gt;,<br>                   brokers=&lt;list_of_brokers&gt;)</pre><h4>Telegram sink</h4><p>Send message to a Telegram bot.</p><pre>sink_config = dict(kind=&#39;telegram-sink&#39;,<br>                   authorization_token=&lt;authorization_token&gt;,<br>                   chat_id=&lt;chat_id&gt;)</pre><h4>Cloud Object Storage sink</h4><p>Upload objects to IBM Cloud Object Storage or AWS S3.</p><pre>sink_config = dict(kind=&#39;cloud-object-storage-sink&#39;,<br>                   bucket_name=&lt;bucket_name&gt;,<br>                   access_key_id=&lt;access_key_id&gt;,<br>                   secret_access_key=&lt;secret_access_key&gt;,<br>                   endpoint=&lt;endpoint&gt;,<br>                   file_name=&lt;name_of_uploaded_file&gt;)</pre><p>Several other options for this sink will be covered in a separate blog post.</p><h3><strong>Generic sources and sinks</strong></h3><p>Rayvens leverages the breadth of components supported by the <a href="https://camel.apache.org/camel-k/latest/">Apache Camel-K project</a> (253+ components). Rayvens includes a growing set of built-in sources and sinks to mediate the interaction between Ray and popular services such as Twitter, Telegram, Cloud Object Storage, etc. Items in this catalog comprise compositions of elementary Camel components for instance a periodic ticker with an HTTP client to offer simple and intuitive sources and sinks such as the http-source. However this catalog today barely scratches the surface of what Camel is capable of.</p><p>To allow power users to program more sophisticated sources and sinks or unlock the entiere catalog of Camel components, Rayvens offer a generic-source and a generic-sink. They makes it possible to instantiate a source or sink running a custom <a href="https://camel.apache.org/camel-k/latest/languages/yaml.html">Camel YAML</a> script. For instance an http-source-like source can be programmed as follows:</p><pre>source_config = dict(kind=&#39;generic-source&#39;,<br>                     spec=&quot;&quot;&quot;<br>- from:<br>  uri: timer:tick?period=3000<br>  steps:<br>    - to: <a href="https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL">https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL</a><br>&quot;&quot;&quot;)</pre><p>Simply set the kind to either generic-source or generic-sink and paste the appropriate YAML script as a properly indented string under the spec field. See the <a href="https://github.com/project-codeflare/rayvens/blob/main/docs/connectors.md#generic-sources">full documentation</a> for details.</p><h4><strong>Conclusion</strong></h4><p>Rayvens makes it easy to connect to external event sources and sinks by means of the vast <a href="https://camel.apache.org/components/latest/index.html">component catalog</a> of Apache Camel. To flatten the learning curve, Rayvens offers easy-to-configure, built-in sources and sinks. Generic sources and sinks enable power users to access the full catalog of Camel components and program sophisticated sources and sinks.</p><p>Rayvens catalog of built-in sources and sinks is rapidly growing. Going forward we plan to enrich this catalog with the recently introduced catalog of <a href="https://github.com/apache/camel-kamelets">Apache Camel Kamelets</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNWKKPqxV3rkwjsoVAk0PQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17f3ff45d92" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/accessing-hundreds-of-event-sources-and-sinks-with-rayvens-17f3ff45d92">Accessing hundreds of event sources and sinks with Rayvens</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accelerating object storage processing for Ray framework]]></title>
            <link>https://medium.com/codeflare/accelerating-object-storage-processing-for-ray-framework-f581863c7662?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/f581863c7662</guid>
            <dc:creator><![CDATA[Gil Vernik]]></dc:creator>
            <pubDate>Thu, 15 Jul 2021 13:52:13 GMT</pubDate>
            <atom:updated>2021-07-15T13:52:13.868Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="http://ray.io/">Ray</a> is a distributed execution framework that makes it easy to scale computing applications. Originally created at UC Berkley, Ray was designed to address deep learning and machine learning workloads, while providing a “serverless” user experience. Ray clusters can be started locally, over virtual machines, Kubernetes, or on cloud platforms. For example, you can have Ray over the <a href="https://www.ibm.com/cloud/blog/ray-on-ibm-cloud-code-engine">IBM Cloud Code Engine</a>.</p><p>A growing ecosystem of libraries on top of Ray allows users and developers to use SQL or data frames over data that is stored in object storage (e.g., IBM Cloud Object Storage, Amazon S3, etc.) For example, Modin, Dask, and even Apache Spark now support integration with Ray and their data pre-processing capabilities can be exploited before the core Ray computations. However, <a href="https://www.ibm.com/cloud/blog/decoding-dark-molecular-matter-in-spatial-metabolomics-with-ibm-cloud-functions">in certain cases</a>, users may have proprietary code, packages, or software that doesn’t naturally fit into Dask, Modin or Apache Spark execution model. As more workloads start to leverage and benefit from Ray, those requirements need to be addressed as well. In this blog, we discuss the challenges in applying Ray computation on object storage data and demonstrate how the <a href="http://lithops.cloud/">Lithops</a> framework can used within a Ray application to simplify data handling. All the code used in this blog and additional documentation can be found under the <a href="https://github.com/project-codeflare/data-integration">CodeFlare</a> project.</p><h3>Ray and object storage­</h3><p>Let’s start with an example of images stored in the IBM Cloud Object Storage and assume we want to extract the colors of each image, along with its dominant color range. Having the colors will enable us to run various deep learning algorithms, like the one described in the “<a href="https://www.youtube.com/watch?v=8GTd8Y_JGTQ&amp;list=PLzTswPQNepXlr2L75tZt7l8tyVZkO-RVt">Movie recommendation system</a>” keynote talk at the Ray Summit 2020.</p><p>We initially write ­­­a helper function that reads a single image from the object storage and extracts its colors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*xRtVsZxiGqXgTrJP_C3-kw.png" /></figure><p>Next, we write a Ray task that, once invoked, will invoke extract colors function, identify the dominant color range, and return the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*hwGsS53SQm7UUK4stR5qAQ.png" /></figure><p>Now let’s implement the main method that lists the images in the bucket and submit our Ray task for a single image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*WgFZTls5DYZwWNDQVGCwRg.png" /></figure><h3>Deep dive into the example</h3><p>While the code above is simple enough, it demonstrates some challenges for the developer. First of all, the developer needs to be familiar with object storage semantics. For example, we need to use paginator to list objects in a bucket, since a single response is usually limited to 1000 entries. Moreover, each Ray task will initiate an object storage connection and perform authentication. Thus, if there are 100K images, there will be 100K tasks: each will set up a new object storage connection and perform the authentication. In such a scenario, a cloud vendor is more likely to throttle (ignore) requests due to a large number of concurrent authentication requests.</p><p>To address the authentication issue, developers can potentially implement a Ray actor and share a token between tasks. But, this involves much more effort when it comes to writing boiler plate code. The “boiler plate” code will have to list and read objects, and will grow even larger if data is stored in other storage locations, like Amazon S3, Azure, Google, CEPH, etc. This essentially forces users to be aware of the different storage backend implementations and APIs, and write more code.</p><p>Clearly, an efficient method for object storage access requires significant effort by the developer. This is exactly where the <a href="http://lithops.cloud/">Lithops</a> framework can be leveraged within Ray to significantly simplify data access and handling for Ray applications.</p><h3>Data adapters for Ray</h3><p>Lithops framework is designed to address two main challenges: deploy and execute user code on a massive scale against any serverless backend, and enable user applications to process large volumes of data, especially when data is persisted in the object storage. As part of CodeFlare effort, we focus on enabling and leveraging Lithops&#39; data loading and processing features to simplify data handling within a Ray context. Lithops has been recently extended to provide native support for Ray, where Lithops uses a Ray cluster as its compute backend. It currently supports IBM Cloud Object Storage, Amazon S3, Google Cloud Storage, Azure Blob Storage, Ceph, and others.</p><figure><img alt="Namibia" src="https://cdn-images-1.medium.com/max/1024/1*QwD0zzz3f2NHXGlKmofPtw.png" /></figure><h3>Leveraging Lithops within Ray</h3><p>Now let’s see how <a href="http://lithops.cloud/">Lithops</a> can help in our example. We need to modify the extract_color function by removing all the object storage semantics. Instead of using boto3 directly, we let Lithops inject a reference to the data object, hiding all boto3 object storage semantics from the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*I4aDEiSRxaJpW1CihorO3g.png" /></figure><p>At runtime, Lithops will fill in the value of the `obj` with a reference to the actual data. This reference has two properties: ‘obj.key’ is the data object name and ‘obj.data_stream’ is a data stream to the actual data in the object storage bucket. More information on the ‘obj’ can be found <a href="https://github.com/lithops-cloud/lithops/blob/master/docs/data_processing.md#processing-data-from-a-cloud-object-storage-service">here</a>.</p><p>We make a small modification to the Ray task. Instead of a direct call to extract_color, Lithops is being used behind the scenes (through the data object) to call it only at the right moment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*_3pU4EClePwagPE6oUNMEw.png" /></figure><p>Let’s tie it all together with a main method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*pLRexhRfusIuy46aTb7gIA.png" /></figure><p>By enabling Lithops within Ray, we are able to remove all the boiler plate code required to list data from the object storage. It also inspects the data source by using the internal Lithops data partitioner and creates a lazy execution plan, where each entry maps an “extract_color” function to a single image. Moreover, Lithops creates a single authentication token that is used by all the tasks, instead of letting each task perform authentication. The parallelism is controlled by Ray and once Ray task is executed, it will call Lithops to execute the extract_color function directly in the context of the calling task. Thus, by using Lithops, we can allow code to access object storage data, without requiring additional coding effort from the user.</p><h3>Summary</h3><p>By enabling Lithops within Ray, we just saw how we can hide all storage semantics and enable Ray users to focus on the business logic, rather than on the boiler plate code required to access underlying storage. Since Lithops supports various storage backends, a simple URI scheme change will allow it to access other storage vendors, such as Amazon S3, which uses “s3://”, and so on. <a href="https://github.com/lithops-cloud/lithops/blob/master/config/README.md#compute-and-storage-backends">Here</a> you can see the supported storage backends and how to configure them. Lithops also supports the chunking of data that resides in big CSV files without breaking lines in the middle. This enables Ray tasks to process chunks of data, instead of loading an entire CSV file into a single task or actor.</p><h3>Next steps</h3><p>We plan to publish more blogs and tutorials showing how Lithops can use Ray as its backend compute engine. We will also explain how to use Ray actors, how to reduce the number of Ray tasks, and even demonstrate how Lithops can be leveraged to reduce overload on the entire Ray cluster, while helping to enable a hybrid cloud experience for Ray use cases.</p><p>Stay tuned!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f581863c7662" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/accelerating-object-storage-processing-for-ray-framework-f581863c7662">Accelerating object storage processing for Ray framework</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rayvens event streams]]></title>
            <link>https://medium.com/codeflare/rayvens-event-streams-da0511b23e9a?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/da0511b23e9a</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[events]]></category>
            <category><![CDATA[actors]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Olivier Tardieu]]></dc:creator>
            <pubDate>Tue, 06 Jul 2021 21:49:15 GMT</pubDate>
            <atom:updated>2021-07-06T22:04:07.110Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Jointly written by </em><a href="https://medium.com/u/b606edb74928"><em>Gheorghe-Teodor Bercea</em></a><em> and </em><a href="https://medium.com/u/4ae0218b5e8"><em>Olivier Tardieu</em></a><em>.</em></p><p>In a <a href="https://medium.com/codeflare/rayvens-event-sources-sinks-and-event-driven-programming-with-ray-b229794dc41f">previous blog</a> we introduced <a href="https://pypi.org/project/rayvens/">Rayvens</a>. With Rayvens, <a href="https://ray.io/">Ray</a> programs can subscribe to event streams, process, and produce events. In this post we dive into the details of the Rayvens streams API. We will cover:</p><ol><li>Rayvens initialization</li><li>Stream construction</li><li>Producing and consuming events</li><li>Stream subscriber types</li><li>Stream operators</li></ol><h3>Rayvens initialization</h3><p>Rayvens must be initialized before use:</p><pre>import rayvens</pre><pre>rayvens.init()</pre><p>The init method takes optional configuration parameters we will cover in a separate post.</p><h3>Stream construction</h3><p>The Rayvens API is built around streams. A stream is an abstraction of an ordered, potentially infinite sequence of events. Events are just items of data with no prescribed schema or semantics. Streams bridge producers and consumers of events in a Ray program with external event sources and sinks. Streams also have the ability to process events. They can filter and/or transform events.</p><p>An event stream is created by instantiating the Rayvens Stream class:</p><pre>stream = rayvens.Stream(&#39;stream_name&#39;)</pre><p>A stream name must be provided.</p><h3>Producing and consuming events</h3><p>Events can be appended to a stream one by one with the &lt;&lt; operator:</p><pre>stream &lt;&lt; &quot;hello&quot; &lt;&lt; &quot;world&quot;</pre><p>This statement appends two events to the stream stream in order, starting with &quot;hello&quot; and followed by &quot;world&quot;.</p><p>Subscribers can be added to a stream to consume its events using the &gt;&gt; operator. For instance, events can be logged to the console using:</p><pre>stream &gt;&gt; print</pre><p>As seen in the previous <a href="https://medium.com/codeflare/rayvens-event-sources-sinks-and-event-driven-programming-with-ray-b229794dc41f">blog post</a>, a custom message can be constructed using a lambda function:</p><pre>stream &gt;&gt; (lambda event: print(&#39;LOG:&#39;, event))</pre><p>The &lt;&lt; and &gt;&gt; operator are not symmetrical. Each &lt;&lt; invocation appends a <em>single event</em> to the stream. In contrast, the right-hand side of the &gt;&gt; invocation is invoked for <em>every event</em> appended to the stream.</p><p>Events are ephemeral. A stream does not remember past events. A subscriber receives all events appended to the stream <em>from the time of the subscription onward</em>.</p><h3>Stream subscriber types</h3><p>A stream subscriber may be a Python callable, a Ray task, a Ray actor method, or a Rayvens stream. A subscriber is invoked for each event appended to the stream with the event as the invocation sole argument. We now demonstrate these different subscriber types and explain ordering guarantees.</p><h4>Python callable</h4><pre>def handler(event):<br>    print(&#39;received&#39;, event)</pre><pre>stream &gt;&gt; handler</pre><p>The handler function is invoked for each event in order in a blocking manner. Each invocation has to complete before the next one can execute.</p><p>If multiple functions are subscribed to a stream, Rayvens guarantees that the executions of these functions do not overlap. Moreover, the processing of one event has to finish for all functions before the next event is considered.</p><h4>Ray tasks</h4><pre>@ray.remote<br>def task(event):<br>    print(&#39;received&#39;, event)</pre><pre>stream &gt;&gt; task</pre><p>Ray tasks subscribed to a stream execute in arbitrary order, irrespective of the order of events. Moreover task executions may overlap.</p><h4>Ray actors</h4><pre>@ray.remote<br>class SimpleActor:<br>    def method(self, event):<br>        print(&#39;received&#39;, event)</pre><pre>actor = SimpleActor.remote()</pre><pre>stream &gt;&gt; actor.method</pre><p>Events are delivered in order to each actor subscribed to the stream. However, if multiple actors are subscribed to the same stream, the order or delivery across actors is unspecified. In particular, one actor may be ahead of another actor by multiple events.</p><p>An actor with a method named append may be subscribed with:</p><pre>stream &gt;&gt; actor</pre><p>This is simply a shorthand for:</p><pre>stream &gt;&gt; actor.append</pre><h4>Rayvens streams</h4><pre>source_stream = rayvens.Stream(&#39;source_stream_name&#39;)<br>sink_stream = rayvens.Stream(&#39;sink_stream_name&#39;)</pre><pre>source_stream &gt;&gt; sink_stream</pre><p>All source_stream events will be appended in order to sink_stream (after processing if an operator is added to source_stream).</p><h3>Stream operators</h3><p>A stream can specify a stream operator. A stream operator can be a Python callable, a Ray task, or a Ray actor method. It is invoked on each event before any stream subscriber. Each operator invocation may either drop the event by returning None or transform the event by returning any other value. Transformed events are then forwarded in order to the stream subscribers.</p><p>For example, an operator may be used to decorate events with a timestamp:</p><pre>@ray.remote<br>class Decorator:<br>    def append(self, event):<br>        return { &#39;event&#39;: event, &#39;time&#39;: time.time() }</pre><pre>decorator = Decorator.remote()</pre><pre>stream.add_operator(decorator)</pre><p>While it is possible to use Ray tasks as stream operators, keep in mind that these tasks may execute in arbitrary order even if the task outputs are then forwarded to the stream subscribers in the order of the incoming events. So Ray tasks should not be used for non-commutative event processing.</p><h3>Conclusion</h3><p>Rayvens unleashes event-driven programming in Ray. In addition to producing and consuming events with the &lt;&lt; and &gt;&gt; operators showcased in the previous sections, Rayvens leverages <a href="https://camel.apache.org/">Apache Camel</a> to accept events from external sources and emit events to external sinks. We will discuss these sources and sinks in a separate blog post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kNWKKPqxV3rkwjsoVAk0PQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da0511b23e9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/rayvens-event-streams-da0511b23e9a">Rayvens event streams</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fit and score multiple classifiers with CodeFlare Pipeline]]></title>
            <link>https://medium.com/codeflare/fit-and-score-multiple-classifiers-with-codeflare-pipeline-6e58fa5b2c52?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/6e58fa5b2c52</guid>
            <category><![CDATA[sklearn-pipeline]]></category>
            <category><![CDATA[codeflare]]></category>
            <dc:creator><![CDATA[Yuan-Chi Chang]]></dc:creator>
            <pubDate>Tue, 22 Jun 2021 03:02:22 GMT</pubDate>
            <atom:updated>2021-06-22T03:02:21.999Z</atom:updated>
            <content:encoded><![CDATA[<p>We use a sklearn pipeline example <a href="https://scikit-learn.org/stable/auto_examples/neighbors/plot_nca_classification.html#sphx-glr-auto-examples-neighbors-plot-nca-classification-py"><strong>Comparing Nearest Neighbors with and without Neighborhood Components Analysis</strong></a><strong> </strong>to demonstrate how to define, fit and score multiple classifiers with CodeFlare (CF) pipeline. The sklearn and CF pipeline notebook is published here <a href="https://github.com/project-codeflare/codeflare/blob/main/notebooks/plot_nca_classification.ipynb">https://github.com/project-codeflare/codeflare/blob/main/notebooks/plot_nca_classification.ipynb</a></p><p>This example plots the class decision boundaries given by a Nearest Neighbors classifier when using the Euclidean distance on the original features, versus using the Euclidean distance after the transformation learned by Neighborhood Components Analysis. Its output is pictorially illustrated with colored decision boundaries like the pictures below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*picmjviDacrLSyEkPivVEA.jpeg" /><figcaption>Classification score and boundaries of KNN with k=1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*1palugNek5hINRZ-JoAzPw.jpeg" /><figcaption>Classification score and boundaries of KNN with Neighborhood Component Analysis</figcaption></figure><p>In the original sklearn pipeline definition, the two KNN classifiers, with and without NCA, are defined with a list of two Pipeline objects as follows:</p><pre>classifiers = [Pipeline([(&#39;scaler&#39;, StandardScaler()),<br>                         (&#39;knn&#39;, KNeighborsClassifier(n_neighbors=n_neighbors))<br>                         ]),<br>               Pipeline([(&#39;scaler&#39;, StandardScaler()),<br>                         (&#39;nca&#39;, NeighborhoodComponentsAnalysis()),<br>                         (&#39;knn&#39;, KNeighborsClassifier(n_neighbors=n_neighbors))<br>                         ])<br>               ]</pre><p>Recognizing both Pipelines start with StandardScalar, we can express it with the same EstimatorNode in a CF pipeline to save redundant calculation, as follows:</p><pre>pipeline = dm.Pipeline()<br>node_scalar = dm.EstimatorNode(&#39;scaler&#39;, StandardScaler())<br>node_knn = dm.EstimatorNode(&#39;knn&#39;, KNeighborsClassifier(n_neighbors=n_neighbors))<br>node_nca = dm.EstimatorNode(&#39;nca&#39;, NeighborhoodComponentsAnalysis())<br>node_knn_post_nca = dm.EstimatorNode(&#39;knn_post_nca&#39;, KNeighborsClassifier(n_neighbors=n_neighbors))</pre><p>In the above CF pipeline, four EstimatorNodes are initiated to define a StandardScalar, an NCA and two instances of KNN classifiers. The EstimatorNodes host respective transformers and models with data input and output ports connected by edges. As shown below, the add_edge() method with binary input connects EstimatorNodes in the upstream-to-downstream order.</p><pre>pipeline.add_edge(node_scalar, node_knn)<br>pipeline.add_edge(node_scalar, node_nca)<br>pipeline.add_edge(node_nca, node_knn_post_nca)</pre><p>A pipeline’s input data is constructed by creating the PipelineInput object. The add_xy_arg() method specifies the input node of the CF pipeline where training data and labels X_train and y_train are added. CF runtime rt executes the pipeline with the declared pipeline, ExecutionType and PipelineInput as input arguments. In our two-branch pipeline example, ExecutionType.FIT fits both pipelines in a single invocation.</p><pre>train_input = dm.PipelineInput()<br>train_input.add_xy_arg(node_scalar, dm.Xy(X_train, y_train))<br>pipeline_fitted = rt.execute_pipeline(pipeline, ExecutionType.FIT, train_input)</pre><p>To get classification scores and prediction output, we first use the select_pipeline() method to identify a fitted pipeline by its end EstimatorNode. For example, the first KNN classifier without NCA is declared by node_knn. select_pipeline() returns the branch ending at node_knn. Subsequently, we use the execute_pipeline() method to invoke the scoring method by specifying ExecutionType.SCORE. Similarly, prediction output is returned via ExecutionType.PREDICT.</p><pre>knn_pipeline = rt.select_pipeline(pipeline_fitted, pipeline_fitted.get_xyrefs(node_knn)[0])</pre><pre>knn_score = ray.get(rt.execute_pipeline(knn_pipeline, ExecutionType.SCORE, test_input).get_xyrefs(node_knn)[0].get_Xref())</pre><pre>Z = ray.get(rt.execute_pipeline(knn_pipeline, ExecutionType.PREDICT, predict_input).get_xyrefs(node_knn)[0].get_Xref())</pre><p>The same steps apply to get the score and predictions of the branch ending at node_knn_post_nca as shown below.</p><pre>nca_pipeline = rt.select_pipeline(pipeline_fitted, pipeline_fitted.get_xyrefs(node_knn_post_nca)[0])</pre><pre>nca_score = ray.get(rt.execute_pipeline(nca_pipeline, ExecutionType.SCORE, test_input).get_xyrefs(node_knn_post_nca)[0].get_Xref())<br><br>Z = ray.get(rt.execute_pipeline(nca_pipeline, ExecutionType.PREDICT, predict_input).get_xyrefs(node_knn_post_nca)[0].get_Xref())</pre><p>Lastly, the fitted, scored and predicted results of the CF pipeline fully replicated those of the original sklearn pipeline, with the same plots shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*picmjviDacrLSyEkPivVEA.jpeg" /><figcaption>Classification score and boundaries of KNN with k=1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*1palugNek5hINRZ-JoAzPw.jpeg" /><figcaption>Classification score and boundaries of KNN with Neighborhood Component Analysis</figcaption></figure><p>The Jupyter notebook of this example is available at <a href="https://github.com/project-codeflare/codeflare/blob/main/notebooks/plot_nca_classification.ipynb">https://github.com/project-codeflare/codeflare/blob/main/notebooks/plot_nca_classification.ipynb</a> to demonstrate how one might translate sklearn pipelines to Codeflare pipelines that take advantage of Ray’s distributed processing. Please try it out and let us know what you think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e58fa5b2c52" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/fit-and-score-multiple-classifiers-with-codeflare-pipeline-6e58fa5b2c52">Fit and score multiple classifiers with CodeFlare Pipeline</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning hyper-parameters with CodeFlare Pipelines]]></title>
            <link>https://medium.com/codeflare/tuning-hyper-parameters-with-codeflare-pipelines-13b884c8336f?source=rss----830b19c217e0---4</link>
            <guid isPermaLink="false">https://medium.com/p/13b884c8336f</guid>
            <category><![CDATA[codeflare-pipelines]]></category>
            <category><![CDATA[codeflare]]></category>
            <category><![CDATA[gridsearchcv]]></category>
            <dc:creator><![CDATA[Kun-Lung Wu]]></dc:creator>
            <pubDate>Tue, 22 Jun 2021 03:02:00 GMT</pubDate>
            <atom:updated>2021-06-22T03:01:59.976Z</atom:updated>
            <content:encoded><![CDATA[<p>GridSearchCV() is often used for hyper-parameter turning for a model constructed via sklearn pipelines. It does an exhaustive search over specified parameter values for a pipeline. It implements a fit() method and a score() method. The parameters of the pipeline used to apply these methods are optimized by cross-validated grid-search over a parameter grid.</p><p>Here we show how to convert an example of using GridSearchCV() to tune the hyper-parameters of an sklearn pipeline into one that uses Codeflare (CF) pipelines grid_search_cv(). We use the “<a href="https://scikit-learn.org/stable/auto_examples/compose/plot_digits_pipe.html#sphx-glr-auto-examples-compose-plot-digits-pipe-py"><strong>Pipelining: chaining a PCA and a logistic regression</strong></a>” from sklearn pipelines as an example.</p><p>In this sklearn example, a pipeline is chained together with a PCA and a LogisticRegression. The n_components parameter of the PCA and the C parameter of the LogisticRegression are defined in a param_grid: with n_components in [5, 15, 30, 45, 64] and C defined by <a href="https://numpy.org/doc/stable/reference/generated/numpy.logspace.html#numpy.logspace">np.logspace</a>(-4, 4, 4). A total of 20 combinations of n_components and C parameter values will be explored by GridSearchCV() to find the best one with the highest mean_test_score.</p><pre>pca = <a href="https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA">PCA</a>()<br>logistic = <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression">LogisticRegression</a>(max_iter=10000, tol=0.1)<br>pipe = <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline">Pipeline</a>(steps=[(&#39;pca&#39;, pca), (&#39;logistic&#39;, logistic)])<br><br>X_digits, y_digits = <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits">datasets.load_digits</a>(return_X_y=True)<br><br>param_grid = {<br>    &#39;pca__n_components&#39;: [5, 15, 30, 45, 64],<br>    &#39;logistic__C&#39;: <a href="https://numpy.org/doc/stable/reference/generated/numpy.logspace.html#numpy.logspace">np.logspace</a>(-4, 4, 4),<br>}<br>search = <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV">GridSearchCV</a>(pipe, param_grid, n_jobs=-1)<br>search.fit(X_digits, y_digits)<br>print(&quot;Best parameter (CV score=%0.3f):&quot; % search.best_score_)<br>print(search.best_params_)</pre><p>After running GridSearchCV().fit(), the best parameters of PCA__n_components and LogisticRegression__C, together with the cross-validated mean_test scores are printed out as follows. In this example, the best n_components chosen is 45 for the PCA.</p><pre>Best parameter (CV score=0.920):<br>{&#39;logistic__C&#39;: 0.046415888336127774, &#39;pca__n_components&#39;: 45}</pre><p>The PCA explained variance ratio and the best n_components chosen are plotted in the top chart. The classification accuracy and its std_test_score are plotted in the bottom chart. The best n_components can be obtained by calling best_estimator_.named_step[‘pca’].n_components from the returned object of GridSearchCV().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*XZAEz1toi42SqAB0pmGi9w.png" /></figure><h3>Converting to CF pipelines grid_search_cv()</h3><p>We next describe the step-by-step conversion of this example to one that uses Codeflare pipelines.</p><h4><strong>Step 1: importing codeflare.pipelines packages and ray</strong></h4><p>We need to first import various codeflare.pipelines packages, including Datamodel and runtime, as well as ray and call ray.shutdwon() and ray.init(). Note that, in order to run this codeflare example notebook, you need to have a running ray instance.</p><pre>import codeflare.pipelines.Datamodel as dm<br>import codeflare.pipelines.Runtime as rt<br>from codeflare.pipelines.Datamodel import Xy<br>from codeflare.pipelines.Datamodel import XYRef<br>from codeflare.pipelines.Runtime import ExecutionType</pre><pre>import ray<br>ray.shutdown()<br>ray.init()</pre><h4>Step 2: defining and setting up a codeflare pipeline</h4><p>A codeflare pipeline is defined by EstimatorNodes and edges connecting two EstimatorNodes. In this case, we define node_pca and node_logistic and we connect these two nodes with pipeline.add_edge(). Before we can execute fit() on a pipeline, we need to set up the proper input to the pipeline.</p><pre>pca = PCA()<br># set the tolerance to a large value to make the example faster<br>logistic = LogisticRegression(max_iter=10000, tol=0.1)<br><br>pipeline = dm.Pipeline()<br>node_pca = dm.EstimatorNode(&#39;pca&#39;, pca)<br>node_logistic = dm.EstimatorNode(&#39;logistic&#39;, logistic)<br><br>pipeline.add_edge(node_pca, node_logistic)<br><br># input to pipeline<br>pipeline_input = dm.PipelineInput()<br>pipeline_input.add_xy_arg(node_pca, dm.Xy(X_digits, y_digits))</pre><h4>Step 3: defining pipeline param grid and executing Codeflare pipelines grid_search_cv()</h4><p>Codeflare pipelines runtime converts an sklearn param_grid into a codeflare pipelines param grid. We also specify the default KFold parameter for running the cross-validation. Finally, Codeflare pipelines runtime executes the grid_search_cv().</p><pre># param_grid<br>param_grid = {<br>        &#39;pca__n_components&#39;: [5, 15, 30, 45, 64],<br>        &#39;logistic__C&#39;: np.logspace(-4, 4, 4),<br>    }<br><br>pipeline_param = dm.PipelineParam.from_param_grid(param_grid)<br><br># default KFold for grid search<br>k = 5<br>kf = KFold(k)<br><br># execute CF pipeplie grid_search_cv<br>result = rt.grid_search_cv(kf, pipeline, pipeline_input, pipeline_param)</pre><h4>Step 4: parsing the returned result from grid_search_cv()</h4><p>As the Codeflare pipelines project is still actively under development, APIs to access some attributes of the explored pipelines in the grid_search_cv() are not yet available. As a result, a slightly more verbose code is used to get the best pipeline, its associated parameter values and other statistics from the returned object of grid_search_cv(). For example, we need to loop through all the 20 explored pipelines to get the best pipeline. And, to get the n_component of an explored pipeline, we first use .get_nodes() on the returned cross-validated pipeline and then use .get_estimator() and then finally use .get_params().</p><pre>import statistics<br><br># pick the best mean&#39; and best pipeline<br>best_pipeline = None<br>best_mean_scores = 0.0<br>best_n_components = 0<br><br>df = pd.DataFrame(columns =(&#39;n_components&#39;, &#39;mean_test_score&#39;, &#39;std_test_score&#39;))<br><br>for cv_pipeline, scores in result.items():<br>    mean = statistics.mean(scores)<br>    std = statistics.stdev(scores)<br>    n_components = 0<br>    params = {}<br>    # get the &#39;n_components&#39; value of the PCA in this cv_pipeline<br>    for node_name, node in cv_pipeline.get_nodes().items():<br>        params[node_name] = node.get_estimator().get_params()<br>        if &#39;n_components&#39; in params[node_name]:<br>            n_components = params[node_name][&#39;n_components&#39;]<br>    assert(n_components &gt; 0)<br>    df = df.append({&#39;n_components&#39; : n_components, &#39;mean_test_score&#39; : mean, &#39;std_test_score&#39; : std}, ignore_index=True)<br>    if mean &gt; 0.92:<br>        print(mean)<br>        print(str(params))<br>        <br>    if mean &gt; best_mean_scores:<br>        best_pipeline = cv_pipeline<br>        best_mean_scores = mean<br>        best_n_components = n_components</pre><p>Due to the differences in split, the Codeflare pipelines grid_search_cv() produces the best pipeline with n_components = 64 for the PAC, and the 2nd best with n_components = 45. We print out the parameters of the 2nd best and the best pipelines as follows.</p><pre>0.9226679046734757<br>{&#39;pca__3&#39;: {&#39;copy&#39;: True, &#39;iterated_power&#39;: &#39;auto&#39;, &#39;n_components&#39;: 45, &#39;random_state&#39;: None, &#39;svd_solver&#39;: &#39;auto&#39;, &#39;tol&#39;: 0.0, &#39;whiten&#39;: False}, &#39;logistic__1&#39;: {&#39;C&#39;: 0.046415888336127774, &#39;class_weight&#39;: None, &#39;dual&#39;: False, &#39;fit_intercept&#39;: True, &#39;intercept_scaling&#39;: 1, &#39;l1_ratio&#39;: None, &#39;max_iter&#39;: 10000, &#39;multi_class&#39;: &#39;auto&#39;, &#39;n_jobs&#39;: None, &#39;penalty&#39;: &#39;l2&#39;, &#39;random_state&#39;: None, &#39;solver&#39;: &#39;lbfgs&#39;, &#39;tol&#39;: 0.1, &#39;verbose&#39;: 0, &#39;warm_start&#39;: False}}<br>0.9260058805323429<br>{&#39;pca__4&#39;: {&#39;copy&#39;: True, &#39;iterated_power&#39;: &#39;auto&#39;, &#39;n_components&#39;: 64, &#39;random_state&#39;: None, &#39;svd_solver&#39;: &#39;auto&#39;, &#39;tol&#39;: 0.0, &#39;whiten&#39;: False}, &#39;logistic__1&#39;: {&#39;C&#39;: 0.046415888336127774, &#39;class_weight&#39;: None, &#39;dual&#39;: False, &#39;fit_intercept&#39;: True, &#39;intercept_scaling&#39;: 1, &#39;l1_ratio&#39;: None, &#39;max_iter&#39;: 10000, &#39;multi_class&#39;: &#39;auto&#39;, &#39;n_jobs&#39;: None, &#39;penalty&#39;: &#39;l2&#39;, &#39;random_state&#39;: None, &#39;solver&#39;: &#39;lbfgs&#39;, &#39;tol&#39;: 0.1, &#39;verbose&#39;: 0, &#39;warm_start&#39;: False}}</pre><p>The corresponding plots are similar to those from the sklearn GridSearchCV(), except that the n_components chosen is 64 for the best score for the Codeflare pipelines grid_search_cv().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*JwPu1HvFitOwfESbHIGutw.png" /></figure><p>The Jupyter notebook of this example is available <a href="https://github.com/project-codeflare/codeflare/blob/main/notebooks/plot_digits_pipe.ipynb">here</a>. Please download it and try it out to understand how you might convert an sklearn example to one that uses Codeflare pipelines. And please let us know what you think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13b884c8336f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codeflare/tuning-hyper-parameters-with-codeflare-pipelines-13b884c8336f">Tuning hyper-parameters with CodeFlare Pipelines</a> was originally published in <a href="https://medium.com/codeflare">CodeFlare</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>