<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gautam Karmakar on Medium]]></title>
        <description><![CDATA[Stories by Gautam Karmakar on Medium]]></description>
        <link>https://medium.com/@gautam.karmakar?source=rss-3f842eaeb840------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DC2CtkSKH7_I06sdfImRAA@2x.jpeg</url>
            <title>Stories by Gautam Karmakar on Medium</title>
            <link>https://medium.com/@gautam.karmakar?source=rss-3f842eaeb840------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:24:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gautam.karmakar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LLM Training and Computation Challenges]]></title>
            <link>https://medium.com/@gautam.karmakar/llm-training-and-computation-challenges-23acda393488?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/23acda393488</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computation]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Sun, 07 Apr 2024 16:32:01 GMT</pubDate>
            <atom:updated>2024-04-07T16:32:01.488Z</atom:updated>
            <content:encoded><![CDATA[<p>The parameter size of a modern LLM is at the magnitude of hundreds of billions, which exceeds the GPU memory of a single device or host. For example, an OPT-175B model requires GPU memory of 350 GB just to accommodate the model parameters—not to mention the GPU memory needed for gradients and optimizer states during training, which can push memory requirements beyond 1 TB.</p><p>To run training and inference for LLMs efficiently, developers need to partition the model across its computation graph, parameters, and optimizer states, such that each partition fits within the memory limit of a single GPU host. Based on the GPU cluster available, ML researchers must adhere to a strategy that optimizes across different parallelization dimensions to enable efficient training.</p><p>Currently, however, optimizing training across different parallelization dimensions (data, model, and pipeline) is a difficult, manual process. Existing dimensional partition strategies of an LLM include the following categories:</p><p><strong>Interoperator parallelism</strong>: Partition the full computation graph to disjoint subgraphs. Each device computes its assigned subgraph and communicates with other devices upon finishing.</p><p><strong>Intraoperator parallelism</strong>: Partition matrices participate in the operator to submatrices. Each device computes its assigned submatrices and communicates with other devices when multiplication or addition takes place.</p><p><strong>Combined</strong>: Both strategies can be applied to the same computation graph.</p><p>The optimization between interoperator and intraoperator is a challenging combinatorial problem with trade-offs. With reasonable computation graph partitioning of interoperator parallelism, the communication cost can be small between subgraphs, but introduces data dependency. Although pipelining can help alleviate the problem, device idle time is still inevitable. Data parallelism frameworks such as <a href="http://alpa.ai/">alpa.ai</a> and <a href="http://ray.io/">Ray.io</a> enhance developer productivity and scale models effectively. Alpa pipeline parallelism can distribute computation across multiple GPU devices in multiple machines and can scale 1000s of machines training 175 Billion parameter LLM models. Ray offers a distributed computing framework that enables simplified scaling and management of resources across multiple machines.</p><p>On the other hand, intraoperator parallelism can parallelize the operator computation among multiple GPU devices with less idle time when the next operator cannot preserve matrix partition from the previous one. This approach comes with a higher communication cost.</p><p>In addition to partitioning matrices and computation graphs, the ability to map partitions to GPU devices with awareness of the heterogeneous network topology is needed. GPU connections inside a node (NVIDIA <a href="https://www.nvidia.com/en-us/data-center/nvlink/">NVLink</a>) are orders of magnitude faster than interhost networking (<a href="https://en.wikipedia.org/wiki/InfiniBand">InfiniBand</a>, EFA, Ethernet), and will lead to significant performance differences among different partition strategies.</p><p>The optimization between interoperator and intraoperator is a challenging combinatorial problem with trade-offs. With reasonable computation graph partitioning of interoperator parallelism, the communication cost can be small between subgraphs, but introduces data dependency. Although pipelining can help alleviate the problem, device idle time is still inevitable. Data parallelism frameworks such as <a href="http://alpa.ai/">alpa.ai</a> and <a href="http://ray.io/">Ray.io</a> enhance developer productivity and scale models effectively. Alpa pipeline parallelism can distribute computation across multiple GPU devices in multiple machines and can scale 1000s of machines training 175 Billion parameter LLM models. Ray offers a distributed computing framework that enables simplified scaling and management of resources across multiple machines.</p><p>On the other hand, intraoperator parallelism can parallelize the operator computation among multiple GPU devices with less idle time when the next operator cannot preserve matrix partition from the previous one. This approach comes with a higher communication cost.</p><p>In addition to partitioning matrices and computation graphs, the ability to map partitions to GPU devices with awareness of the heterogeneous network topology is needed. GPU connections inside a node (NVIDIA <a href="https://www.nvidia.com/en-us/data-center/nvlink/">NVLink</a>) are orders of magnitude faster than interhost networking (<a href="https://en.wikipedia.org/wiki/InfiniBand">InfiniBand</a>, EFA, Ethernet), and will lead to significant performance differences among different partition strategies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RH7WHP6wTnoXPgU_" /></figure><p>Combining Alpa and Ray OSS frameworks, developers can efficiently scale LLM training across a large cluster on JAX. Use Alpa to automatically compile your network architecture, and use Ray to orchestrate and run your tasks across a cluster of machine<strong>s.</strong></p><p><strong>LLM Inference Performance</strong></p><p>LLM inference is another piece of the puzzle that requires special attention to computation complexity and optimization. <a href="https://www.nvidia.com/en-us/data-center/resources/mlperf-benchmarks/?ncid=so-link-516563-vt11">MLPerf</a> Inference is a benchmark suite that measures inference performance across several popular deep-learning use cases. The latest version of the benchmark suite–MLPerf Inference v4.0–adds two new workloads that represent popular and modern generative AI use cases. The first is an LLM benchmark based on the largest of the Meta Llama 2 family of large language models (LLMs), <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/llama2-70b/overview">Llama 2 70B</a>. The second is a text-to-image test based on <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/sdxl">Stable Diffusion XL</a>.</p><p>The NVIDIA accelerated computing platform set performance records on both the new workloads using the <a href="https://www.nvidia.com/en-us/data-center/h200/">NVIDIA H200 Tensor Core GPU</a>. And, using <a href="https://developer.nvidia.com/blog/nvidia-tensorrt-llm-supercharges-large-language-model-inference-on-nvidia-h100-gpus/">NVIDIA TensorRT-LLM</a> software, the <a href="https://www.nvidia.com/en-us/data-center/h100/">NVIDIA H100 Tensor Core GPU</a> nearly tripled performance on the GPT-J LLM test.</p><p>LLM based use cases such as chatbot applications demand faster response to user queries cost effectively. Production LLMs must be able to serve requests at the lowest latency and highest throughput possible.</p><p>NVDA TensorRT-LLM is a high-performance, open-source software library providing state-of-the-art performance when running the latest LLMs on NVIDIA GPUs.</p><p>MLPerf Inference v4.0 includes two LLM tests. The first is GPT-J, which was introduced in the prior round of MLPerf, and the second is the newly added Llama 2 70B benchmark. H100 Tensor Core GPUs using TensorRT-LLM achieved speedups on GPT-J of 2.4x and 2.9x in the offline and server scenarios, respectively. Compared to submissions in the prior round. TensorRT-LLM was also central to the NVIDIA platform’s exceptional performance on the Llama 2 70B test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2CWwK80m6qoLTB-Y" /><figcaption><em>Image: TensorRT-LLM enabled large performance increases on the GPT-J test in this round of MLPerf Inference</em></figcaption></figure><p>The following are some of the key features of TensorRT-LLM that enabled these great performance results:</p><p><a href="https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/gpt_attention.md#inflight-batching"><strong>Inflight sequence batching</strong></a> increases GPU use during LLM inference by better interleaving inference requests and evicting requests in a batch as soon as they’ve completed processing and inserting new requests in their place.</p><p><a href="https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/gpt_attention.md#paged-kv-cache"><strong>Paged KV cache</strong></a> improves memory consumption and usage by partitioning and storing the KV cache into non-contiguous memory blocks, allocating and evicting blocks on demand, and accessing the blocks dynamically during attention computation.</p><p><a href="https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/"><strong>Tensor parallelism</strong></a> supports splitting weights across GPUs and nodes using NCCL for communication, enabling efficient inference at scale.</p><p><a href="https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/quantization-in-TRT-LLM.md"><strong>Quantization</strong></a> supports FP8 quantization, which uses the fourth-generation Tensor Cores in the NVIDIA Hopper architecture to reduce model size and increase performance.</p><p><a href="https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/XQA-kernel.md"><strong>XQA kernel</strong></a><strong> </strong>high-performance implementation of attention that supports MHA, MQA, and GQA, along with beam search, significantly increasing throughput within a given latency budget.</p><p>The NVIDIA GPUs with TensorRT software delivered the highest performance in the MLPerf Inference v4.0 text-to-image test.</p><p>An 8-GPU NVIDIA HGX H200 system with GPUs configured to a 700W TDP, achieved performance of 13.8 queries/second and 13.7 samples/second in the server and offline scenarios, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O_dTYET40wucqAvN" /><figcaption>Image: The NVIDIA INT8-based recipe achieves similar image quality to the FP16 baseline while improving performance.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23acda393488" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangChain for Building Real World GenAI Application]]></title>
            <link>https://medium.com/@gautam.karmakar/langchain-for-building-real-world-genai-application-a9af8f40f0ff?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9af8f40f0ff</guid>
            <category><![CDATA[gen-ai-tools]]></category>
            <category><![CDATA[aritificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Sat, 16 Mar 2024 17:15:26 GMT</pubDate>
            <atom:updated>2024-03-16T17:15:26.254Z</atom:updated>
            <content:encoded><![CDATA[<p>Langchain provides necessary instructions over LLMs and leverages company’s private data and uses API to interact with different utilities for real world context aware applications and reasoning.</p><p>It connects sources such as prompt, few shot examples and content to ground LLMs response to provide necessary context and rely on LLMs to reason based on the context. LLMs don’t retain states and for contextual conversation or task completion maintaining state is important to produce correct result. Langchain can be run in two ways: plan and execute or reason and act. In reason to act mode it performs a series of context based operations using LLMs to solve a problem. Plan and execute works little differently where it needs to execute some steps ahead of time and run operations sequentially.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XacrK1YJdEV3qCkU" /><figcaption>Image Courtesy: <a href="https://www.langchain.com/">https://www.langchain.com/</a></figcaption></figure><h3>Agents Based Orchestration</h3><p>Agents need access to an external data source, and they need to connect to your existing APIs. Developers can use the AWS Bedrock console or the AWS SDK to upload the API schema. The agents will orchestrate tasks with the help of FMs and perform API calls using AWS Lambda functions. Therefore, agents remove the need to manage system integration and infrastructure provisioning. Following a process flow of using LangChain using AWS Bedrock LLMs for ReACT application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oKZEM8ySqNzcVcGc" /><figcaption>Image Courtesy: <a href="https://aws.amazon.com/bedrock/">https://aws.amazon.com/bedrock/</a></figcaption></figure><p>LangChain is open source and is currently available in three programming languages: Python, TypeScript, and JavaScript. LangChain consists of components such as schema, models, prompts, indexes, memory, chains, and agents. Developers can use agents for Amazon Bedrock to create an orchestration plan without any manual coding. For example, an agent-powered generative AI restaurant application can provide a basic response to the question, “Do we have sufficient amounts of dough to sustain this week?” However, it can also help you with the task of updating your inventory. LangChain integrations for models, prompt templates, indexes, memory, chains, and agents and can be used with these components in a RAG (Retrieval Augmented Generation) application.</p><p>LangChain interacts with LLMs provided in AWS BedRock and can access Amazon Titan Text, AI21 Labs Jurassic, Anthropic Claude, and Cohere Command. Following is example python code.</p><pre>from langchain.llms import Bedrock<br>inference_modifiers = {&quot;temperature&quot;: 0.3, &quot;maxTokenCount&quot;: 512}<br>llm = Bedrock(<br>     client = boto3_bedrock,<br>     model_id=&quot;amazon.titan-tg1-large&quot;<br>     model_kwargs =inference_modifiers<br>)<br>response = llm.predict(&quot;What is the largest city in Vermont?&quot;)<br>print(response) <br>The following example demonstrates how you can get a response from an LLM by passing a user request to the LLM.<br>from langchain.chat_models.bedrock import BedrockChat<br>from langchain.schema import HumanMessage<br>chat = BedrockChat(model_id=&quot;anthropic.claude-v2&quot;, model_kwargs={&quot;temperature&quot;:0.1})<br>messages = [<br>     HumanMessage(<br>          content=&quot;I would like to try Indian food. Name three Indian vegetarian dishes for a meal.&quot;<br>     )<br>]<br>chat(messages)<br>Response:<br>AIMessage(content=&quot; Here is a list of Indian vegetarian dishes for a meal: aloo gobi, chana masala, and dal tadka.&quot;, additional_kwargs={}, example=False)</pre><p>LangChain provides predefined prompt templates in the form of text strings that can take a set of parameters from the user and generate a prompt. Prompt templates make prompt engineering more efficient and make it possible to reuse prompts.</p><p>The following example demonstrates the creation of a prompt template, pass input variables, and templates as arguments.</p><pre>from langchain import PromptTemplate<br># Create a prompt template that has multiple input variables<br>multi_var_prompt = PromptTemplate(<br>     input_variables=[&quot;customerName&quot;, &quot;feedbackFromCustomer&quot;],<br>     template=&quot;&quot;&quot;<br>     Human: Create an email to {customerName} in response to the following customer service feedback that was received from the customer: <br>     &lt;customer_feedback&gt; <br>          {feedbackFromCustomer}<br>     &lt;/customer_feedback&gt;<br>     Assistant:&quot;&quot;&quot;<br>)<br># Pass in values to the input variables<br>prompt = multi_var_prompt.format(customerName=&quot;John Doe&quot;,<br>          feedbackFromCustomer=&quot;&quot;&quot;Hello AnyCompany, <br>     I am very pleased with the recent experience I had when I called your customer support.<br>      I got an immediate call back, and the representative was very knowledgeable in fixing the problem. <br>     We are very happy with the response provided and will consider recommending it to other businesses.<br>     &quot;&quot;&quot;<br>)</pre><p>LangChain provides the document loaders component, which is responsible for loading documents from various sources. Sources can include a database, an online store, or a local store. You can index and use information from these sources for information retrieval. You can use the document loaders to load different types of documents, such as HTML, PDF, and code.</p><pre>from langchain.retrievers import AmazonKendraRetriever<br>from langchain.chains import ConversationalRetrievalChain<br>from langchain.prompts import PromptTemplate<br>from langchain.llms.bedrock import Bedrock<br>llm = Bedrock(<br>     credentials_profile_name=credentials_profile_name,<br>     region_name = region,<br>     model_kwargs={&quot;max_tokens_to_sample&quot;:300,&quot;temperature&quot;:1,&quot;top_k&quot;:250,&quot;top_p&quot;:0.999,&quot;anthropic_version&quot;:&quot;bedrock-2023-05-31&quot;},<br>     model_id=&quot;anthropic.claude-v2&quot;<br>)<br>retriever = AmazonKendraRetriever(index_id=kendra_index_id,top_k=5,region_name=region)<br>prompt_template = &quot;&quot;&quot; Human: This is a friendly conversation between a human and an AI. <br>     The AI is talkative and provides specific details from its context but limits it to 240 tokens.<br>     If the AI does not know the answer to a question, it truthfully says it does not know.<br><br>     Assistant: OK, got it, I&#39;ll be a talkative truthful AI assistant.<br>     Human: Here are a few documents in &lt;documents&gt; tags:<br>     &lt;documents&gt;<br>     {context}<br>&lt;/documents&gt;<br>Based on the above documents, provide a detailed answer for, {question} <br>     Answer &quot;do not know&quot; if not present in the document. <br>      Assistant:<br>     &quot;&quot;&quot;<br>PROMPT = PromptTemplate(<br>     template=prompt_template, input_variables=[&quot;context&quot;, &quot;question&quot;]<br>     )<br>response = ConversationalRetrievalChain.from_llm(<br>     llm=llm, <br>     retriever=retriever, <br>     return_source_documents=True, <br>     combine_docs_chain_kwargs={&quot;prompt&quot;:PROMPT},<br>     verbose=True)</pre><p>The LLMs are pretrained on a vast amount of text data and learn from a wide range of patterns. Hence, they can generate effective text responses. However, they lack the capability when it comes to solving rule-based mathematical, logic, and reasoning types of requests.</p><p>LangChain agents can interact with external sources, such as search engines, calculators, APIs, or databases. The agents can also run code to perform actions to assist the LLMs in generating accurate responses. LangChain provides the chain interface to sequence multiple components to build an application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9af8f40f0ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reference Architecture of GenAI Applications]]></title>
            <link>https://medium.com/@gautam.karmakar/reference-architecture-of-genai-applications-0b959402dcac?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b959402dcac</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gen-ai-for-business]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Sun, 10 Mar 2024 18:04:29 GMT</pubDate>
            <atom:updated>2024-03-10T18:13:07.233Z</atom:updated>
            <content:encoded><![CDATA[<p>The first components for generative AI is a user interface where users can interact with generative AI models and other components for a variety of use cases such as text summarization, question and answering, text generation and agents.</p><h3>Interface</h3><p>Users enter prompts in the user interface that can be deployed using Azure App service Web Apps which can reduce infrastructure management overhead and provide required scale and high availability of the application. The interface apps can communicate to backend cloud services using private endpoints for secured reliable connection. The interface primarily makes API calls to different services including machine learning models and GenAI models to send requests and get responses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/709/1*LyXCOIuUFzH2sPFbsUeH1A.png" /><figcaption>Image Courtesy: <a href="https://explore.skillbuilder.aws/learn/lp/2068/generative-ai-learning-plan-for-developers">https://explore.skillbuilder.aws/learn/lp/2068/generative-ai-learning-plan-for-developers</a></figcaption></figure><h3>Inference Parameters:</h3><p>Along with supplying prompts, using effective inference parameters can strongly influence the output from a foundation model. You can pass the parameters along with the prompts to the foundation model interface APIs. LLMs operate on tokens, which can be words, letters, or partial word units. The LLM takes a sequence of input tokens and predicts the next token. The inference parameters help provide guidance to the LLM to produce the output or a sequence of tokens that are relevant for your use case. Some of the common parameters are top P or nucleus sampling. This technique controls choosing from the smallest possibility of tokens where the combined probability exceeds the parameter Top P. A higher value of Top P, such as 0.9, means the output is more deterministic and predictable because it chooses the most probable next tokens. Lower values can increase diversity, but output can become incoherent. It’s essential that you balance creativity and coherence. Whereas Top P works based on probabilities, Top K reduces the sample size to the next k probable tokens. Typical k values are from 10 to 100. A k value of 1 is called a greedy strategy because the most probable token is always chosen. Whereas Top P and Top K control which tokens are chosen based on the model’s output, the temperature parameter affects the model’s output directly. The higher the temperature, the flatter the probability distribution, which means it will be uniform across tokens. The generated tokens will be more creative and random. Lower temperature will polarize the distribution, which makes deterministic outputs possible.</p><h3>Enterprise Datasets</h3><p>GenAI foundation model performance can be further enhanced and influenced using the company’s knowledge base using its own enterprise datasets.Typically Retrieval Augmented Generation technique is used to optimize GenAI foundation model’s output. It provides a better price performance alternative to fine tune foundation model by training using enterprise data. The way it works is that raw enterprise data is passed through a ML model to generate dense vector representation called embeddings that then is stored and retrieved to influence output from foundation models.</p><p><strong>Vector Embeddings</strong></p><p>Embeddings creates n-dimensional vectors to understand the contextual meaning of the natural language. Enterprise datasets, such as documents, images and audio, are passed to ML models as tokens and are vectorized. These vectors in an n-dimensional space, along with the metadata about them, are stored in purpose-built vector databases for faster retrieval.</p><p>These vectors are stored in specialized vector databases such as Oracle Database 23c which comes with AI vector search using OpenSearch and MySQL HeatWave vector store services. The complete RAG solution comes with an embedding model, a search solution and a vector datastore. Amazon Web Services (AWS) offers Amazon OpenSearch Service,<strong><em> pgvector</em></strong> extension of Amazon RDS, Marketplace solution for Pinecone and Facebook’s FAISS, Chroma and many more. Enterprise can leverage their datasets to develop knowledge base and optimize foundation GenAI model’s output using a RAG.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hVdNpuI6csAUqg8U" /><figcaption>Image Courtesy: <a href="https://explore.skillbuilder.aws/learn/course/17256/planning-a-generative-ai-project;lp=2068">https://explore.skillbuilder.aws/learn/course/17256/planning-a-generative-ai-project;lp=2068</a></figcaption></figure><p>The architecture diagram describes the components used in the RAG architecture and depicts the sequence of events.</p><h3>Prompt Engineering</h3><p>RAG provides a solution that is a low cost alternative to fine tune pretrained foundation GenAI models. However, foundation models also can be further trained using a machine learning model to adapt to enterprise data and encode knowledge of specific domains. In the latter case foundation models are modified with generation of new weights. One of such techniques to fine tune models are prompt based learning where a smaller labeled dataset is provided to fine tune foundation models, usually involving lightweight training and few epochs. The input is carefully crafted in the format of question and response pairs and phrased as instructions. Crafting prompts to fine tune LLMs are no trivial tasks and give rise to new skills of prompt engineering. The fine tuning is often done for a specific use case and not likely to be generalized for every type of use case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GaBAtHZM5Yunx2tl" /></figure><h3>Orchestrator</h3><p>An orchestrator app plays a significant role to coordinate different tasks in the process flow. A common way of implementing this orchestrator is using prompt flow. A prompt flow allows you to build, evaluate and deploy tasks such as linking user’s input to RAG service, integrate the response from the RAG system and handle request and response from foundation models, finally presenting the result to the user interface app. Prompt flow has built in capability to understand natural language in the form of instructions and calls different services such as embedding model, vector stores and foundation model based on natural language understanding. Orchestrator calls prompt history store to retrieve more context and state information for contextual long conversation. A prompt history store is another essential component of the architecture that helps with contextually aware conversations that are both relevant and coherent.The prompt flow implements the RAG pattern to extract the appropriate query from the prompt, query AI Search, and use the results as grounding data for the LLMs.</p><h3>Networking</h3><p>Network flow and a secured implementation of networks are essential for a reliable architecture.The interface is exposed using a single secure endpoint using firewall to filter traffic and a DDoS service to prevent potential cyberattack and malicious attacks. Network traffic should be encrypted using TLS 2.0 end to end. Data exfiltration should be prevented by using private endpoints to bypass the internet. A prompt flow author can connect to different machine learning services using bastion host and https REST calls. All application servers and data stores should be launched in a private network protected by network security groups and routing policies. All PaaS services should be accessed using private endpoints and accessing outbound internet should be done using NAT gateway.</p><h3>Governance &amp; Security</h3><p>Manage and audit GenAI application who has access to model, data stores, APIs is central for an enterprise grade deployment. Principle of least privileges should be maintained via identity and access management and using RBAC. Resources should be tagged and grouped by usage and access requirements. Application and data access should be monitored at the granular level to detect and mitigate any risks. Log request and responses to foundation models, update prompt history stores and periodically audit request and response for compliance, bias or ethical implications. Document various facets of the applications and keep them up to date. Use CI/CD and automation for build, test and deplyments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b959402dcac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training Large Language Models in AWS SageMaker]]></title>
            <link>https://medium.com/@gautam.karmakar/training-large-language-models-in-aws-sagemaker-2e5e38526382?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e5e38526382</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 17:44:49 GMT</pubDate>
            <atom:updated>2023-11-19T17:44:49.845Z</atom:updated>
            <content:encoded><![CDATA[<p>Large language models based on transformer model and many variaton of transformer models are making tremendous buzz and mostly for a right reasons.</p><p>Large language models are large because of billions and trillions of parameters. It also process huge amount of data with billions of tokens. Large data size and models pose great challenge for these models to train as its difficult to load all the data and parameters in memory to process. This demands specialized hardware infrastructure for both compute and networks as well as data processing.</p><p>Single GPU device is not sufficient to train large language models and it needs multiple GPU devices in multiple instances. But distributed cmputing for large language models comes with challenges. We need to manage data parallelism, model parallelism and orchestrate everything such that expensive devices are optimally used to save cost and time.</p><p>AWS SageMaker comes with framework and libraries that makes it easier to train LLMs using multiple instances and GPUs. AWS SageMaker is a managed batch compute that lets users to train and deploy machine learning models at scale without need to manage any infrastructure. It comes with advanced features such as metadata management, data and model version control, deploy for inference in your choice of compute, debugging and tracking model performance.</p><p>In this article we will dive deep into challenges in training large language models and how SageMaker can help to achieve best performance and scale to train large language models.</p><h3>The Model</h3><p>Transformer was designed to avoid pitfalls of RNN by relying on self-attention mechanisms to draw global dependencies between input and output. Transformer architecture allows higher degree of parallelism and higher performance to finish the training in a shorter period of time. Building a transformer original architecture BRT (Bi-Directional Transformer Representation For Language Understanding) was introduced to have bi-directional pre training for language. BERT was pre-trained using masked language modeling (MLM) where model is to recover original words of randomly generated masked tokens. The BERT model is also pre-trained on the next sentence prediction (NSP) task to predict if tow sentences are in correct order. Since 2018 variations of BERT have been popularly used.</p><p>The model create two embedding layers; one for tokens and another for token’s positional indexes.</p><pre>def __init__(self, maxlen, vocab_size, embed_dim):<br>        super(TokenAndPositionEmbedding, self).__init__()<br>        self.token_emb = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim)<br>        self.pos_emb = tf.keras.layers.Embedding(input_dim=maxlen, output_dim=embed_dim)<br><br>    def call(self, x):<br>        maxlen = tf.shape(x)[-1]<br><br>        # positions are represented by a token&#39;s index<br>        positions = tf.range(start=0, limit=maxlen, delta=1)<br>        positions = self.pos_emb(positions)<br><br>        # token embedding<br>        x = self.token_emb(x)<br><br>        # return sum as input <br>        return x + positions</pre><p>Then we define transformer decoder block with two sub layers; a multi head self attention layer and a fully connected feed forward network followed by a layer normalization and dropout for regulariation.</p><pre>class TransformerBlock(tf.keras.layers.Layer):<br>    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):<br>        # self attention layer<br>        super(TransformerBlock, self).__init__()<br>        self.att = tf.keras.layers.MultiHeadAttention(<br>            num_heads=num_heads, key_dim=embed_dim)<br>        <br>        # feed forward layer<br>        self.ffn = [tf.keras.layers.Dense(ff_dim, activation=&quot;relu&quot;), tf.keras.layers.Dense(embed_dim)]<br><br>        # layer normalization <br>        self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)<br>        self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)<br><br>        # dropout <br>        self.dropout1 = tf.keras.layers.Dropout(rate)<br>        self.dropout2 = tf.keras.layers.Dropout(rate)<br><br>    def call(self, inputs):<br>        # getting batch size and seq len from input shape<br>        input_shape = tf.shape(inputs)<br>        batch_size = input_shape[0]<br>        seq_len = input_shape[1]<br><br>        # decoder casual mask<br>        casual_mask = casual_attention_mask(batch_size, seq_len, seq_len, tf.bool)<br><br>        # self attention forward pass<br>        attention_output = self.att(inputs, inputs, attention_mask=causal_mask)<br><br>        # dense layers, dropout and normalization<br>        attention_output = self.dropout1(attention_output)<br>        ffn_output = self.ffn[0](out1)<br>        ffn_output = self.ffn[1](ffn_output)<br>        out2 = self.dropout2(ffn_output)<br>        <br>        return self.layernorm2(out1 + out2)<br><br>Finally we create our language model with the preceding embedding layer and transformation blocks.<br>class MyModel(tf.keras.Model):<br>    def __init__(self, maxlen, vocab_size, embed_dim, num_heads, feed_forward_dim, num_layers, learning_rate):<br>        super(MyModel, self).__init__(maxlen, vocab_size, embed_dim, num_heads, feed_forward_dim, num_layers, learning_rate)<br><br>        # embedding layer<br>        self.embedding_layer = TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim)<br><br>        # transformer blocks<br>        self.transformer_blocks = [<br>            TransformerBlock(embed_dim, num_heads, feed_forward_dim)<br>            for i in range(num_layers)<br>        ]<br><br>        # last dense layer<br>        self.dense = tf.keras.layers.Dense(vocab_size)<br>        <br>    def call(self, inputs, training=None):<br>        x_emb = self.embedding_layer(inputs)<br>        x = x_emb        <br>        for transformer_block in self.transformer_blocks:<br>            x = transformer_block(x)<br>        outputs = self.dense(x)<br>        return [outputs, x_emb]<br><br><br>def init_train_settings(maxlen, vocab_size, embed_dim, num_heads, feed_forward_dim, num_layers, learning_rate):<br>    &quot;&quot;&quot;<br>    Creates model, optimizer and loss function <br>    &quot;&quot;&quot;<br>    model = MyModel(maxlen, vocab_size, embed_dim, num_heads, feed_forward_dim, num_layers, learning_rate) <br>    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)<br>    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)<br>    return model, optimizer, loss_fn</pre><p>As there are limit on loading model parameters into a single device’s memory there are framework like Horovod that can be used to distribute models into multiple devices for distributed training. SageMaker also comes with its own built-in framework for model parallelism and we will look into both.</p><h3>The Dataset:</h3><p>The models are trained typically using multi-terabytes of dataset. for example Pile dataset is high quality language data sourced from various domain of texts such as GitHub repositories; webpages; chat logs; and medical, physics, math, computer science, and philosophy paper and over 4 TiB preprocessed and uncomptressed. The data needs to be broken into i GiB chunks to send it to multiple devices for parallel training and updated parameters and loss function value needs to be aggregated. Tensoflow dataset comes with high performance low latency and esy to use data parallelism that let you create pipeline to integarates with high performance file storage like AWS EFS that scales automatically with bursting I/O throughput. For data distribution mini batches of data sent to different devices and trained using SGD. Then gradients are averaged using a technique called <strong><em>AllReduce, </em></strong>which becomes complex as more and more devices are used.</p><p><strong>Using Horovod:</strong></p><ol><li>Initialize running hvd.init()</li><li>Associate each machine with a single process, the first process or first worker is associated with the first device etc.</li><li>Adjust learning rate based on number of devices.</li><li>Wrap optimizer rate based using hvd.DistributedOptimizer.</li><li>Broadcast the initial variable from the first worker with rank 0 to all other processes. This is to ensre consistent initialization of all workers when training is atrater with random weights or restored from a checkpoint.</li><li>Make sure that only device 0 can save checkpoints to prevent other workers from corrupting them.</li></ol><p>Training script:</p><pre>import horovod.tensorflow as hvd<br># Initialize Horovod<br>hvd.init()<br><br># Pin GPU to be used to process local rank (one GPU per process)<br>gpus = tf.config.experimental.list_physical_devices(&#39;GPU&#39;)<br>for gpu in gpus:<br>    tf.config.experimental.set_memory_growth(gpu, True)<br>if gpus:<br>    tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], &#39;GPU&#39;)<br><br># Build model<br>...<br><br>@tf.function<br>def training_step(texts, labels, first_batch):<br>    with tf.GradientTape() as tape:<br>        predictions = model(texts, training=True)<br>        loss = loss_fn(labels, predictions[0])<br><br>    # Horovod: add Horovod Distributed GradientTape.<br>    tape = hvd.DistributedGradientTape(tape)<br><br>    grads = tape.gradient(loss, model.trainable_variables)<br>    opt.apply_gradients(zip(grads, model.trainable_variables))<br><br>    # Horovod: broadcast initial variable states from rank 0 to all other processes.<br>    # This is necessary to ensure consistent initialization of all workers when<br>    # training is started with random weights or restored from a checkpoint.<br>    #<br>    # Note: broadcast should be done after the first gradient step to ensure optimizer<br>    # initialization.<br>    if first_batch:<br>        hvd.broadcast_variables(model.variables, root_rank=0)<br>        hvd.broadcast_variables(opt.variables(), root_rank=0)<br><br>    return loss<br><br># Horovod: adjust number of steps based on number of GPUs.<br>for batch, (texts, labels) in enumerate(dataset.take(10000 // hvd.size())):<br>    loss = training_step(texts, labels, batch == 0)<br><br>    if batch % 10 == 0 and hvd.local_rank() == 0:<br>        print(&#39;Step #%d\tLoss: %.6f&#39; % (batch, loss))<br><br># Horovod: save checkpoints only on worker 0 to prevent other workers from<br># corrupting it.<br>if hvd.rank() == 0:<br>    checkpoint.save(checkpoint_dir)Sagemaker data parallel library enables to scale training with near linear efficiency, speeding up the training process. The model performs custom AllReduce process and optimizes device-to-device communication by fully utilizing AWS’s EC2 and network infrastructure.</pre><p>SageMaker data parallel library enables to scale training with near linear efficiency, speeding up the training process. The model performs custom AllReduce process and optimizes device-to-device communication by fully utilizing AWS’s EC2 and network infrastructure.</p><ol><li>Import sdp.init()</li><li>smdistributed.dataparallel process with local_rank.sdp.tensorflow.local_rank() gives us the local rank devices. The leader is rank 0 and workers are rank 1, 2,3 and so on.</li><li>Adjust learning rate based on the number of devices</li><li>Wrap tf.gradientTape with distributedGradientTape to perform AllReduce.</li><li>Broadcast the initial model variables from the leader node to all the worker nodes.</li><li>Make sure that only device 0 can save checkpoints.</li></ol><p>Model parallelism -</p><p>Use SageMaker model distributed parallel library to train large models.</p><ol><li>Import and initialize the library smp.initi()</li><li>The keras model needs to inherit from smp.DistributedModel instead of the keras model class</li><li>Set drop_reminder=True in the tf.Dataset.batch() method to ensure that the batch size is always divisible by the number of microbatches.</li><li>Random operations in the data pipeline all need to use the same seed. smp.dn_rank(), for example shuffle(ds, seed=smp.dp_rank()). This ensures consistency of data samples across devices that hold different model partitions.</li><li>Forward and backward logic needs to be in a step function with a smp.step decoration.</li><li>Perform post processing on the outputs across micro batches using StepOutput methods such as reduce_mean. The smp,step function must have a return value that depends on the output of smp.DistributedModel.</li></ol><pre># SMP: Initialize<br>smp.init()<br><br># SMP: Define smp.DistributedModel the same way as Keras sub-classing API<br>class MyModel(smp.DistributedModel):<br>    def __init__(self, maxlen, vocab_size, embed_dim, num_heads, feed_forward_dim, num_layers, learning_rate):<br>        super(MyModel, self).__init__(maxlen, vocab_size, embed_dim, num_heads, feed_forward_dim, num_layers, learning_rate)<br>        <br>        self.embedding_layer = gpt_model.TokenAndPositionEmbedding(maxlen, vocab_size, embed_dim)<br>        self.transformer_blocks = [<br>            gpt_model.TransformerBlock(embed_dim, num_heads, feed_forward_dim)<br>            for i in range(num_layers)<br>        ]<br>        self.dense = tf.keras.layers.Dense(vocab_size)<br>        <br>    def call(self, inputs, training=None):<br>        x_emb = self.embedding_layer(inputs)<br>        x = x_emb<br><br>        for transformer_block in self.transformer_blocks:<br>            x = transformer_block(x)<br>        outputs = self.dense(x)<br>        return [outputs, x_emb]<br><br><br># SMP: Define smp.step. Return any tensors needed outside<br>@smp.step<br>def get_grads(texts, labels):<br>    predictions = model(texts, training=True)<br>    loss = loss_fn(labels, predictions[0])<br>    grads = optimizer.get_gradients(loss, model.trainable_variables)<br>    return grads, loss, predictions[0]<br><br>@tf.function<br>def train_step(texts, labels, first_batch):<br>    gradients, loss, predictions = get_grads(texts, labels)<br>    # SMP: Accumulate the gradients across microbatches<br>    gradients = [g.accumulate() for g in gradients]<br>    optimizer.apply_gradients(zip(gradients, model.trainable_variables))<br>    <br>    # SMP: Average the loss across microbatches<br>    train_loss(loss.reduce_mean())<br>    # SMP: Merge predictions across microbatches<br>    train_accuracy(labels, predictions.merge())<br>    return loss.reduce_mean()<br><br>histories = []<br><br>for _ in range(epochs):<br>    train_loss.reset_states()<br>    train_accuracy.reset_states()<br><br>    for texts, labels in text_ds:<br>        for i in range(128):<br>            text = tf.expand_dims(texts[0][i], axis=0)<br>            label = tf.expand_dims(labels[0][i], axis=0)<br>            train_step(text, label)</pre><h3>Conclusion:</h3><p>SageMaker debugger is another great tool in the arsenal to monitor and track model training right from the notebook or using SageMaker library APIs. Debugger provides utilities to visualize profiling data, training loss, CPU and GPU utilization as well as I/O wait for multi-gpu training which can be very useful for detect any bottleneck. For training AWS provides cost effective and scalable infrastructure including EC2 instances with CPU and GPUs such as ml.p3.16xlarge, EFS scalable file storage, FSx for luster for high performance file storage and EFA for high inter-node network performance. This shows how AWS SageMaker can help you to achieve fine tuning of large language models like BERT with billions of parameters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e5e38526382" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative AI and Large Language Models]]></title>
            <link>https://medium.com/@gautam.karmakar/generative-ai-and-large-language-models-fc4220a36a69?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc4220a36a69</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Tue, 03 Oct 2023 16:12:45 GMT</pubDate>
            <atom:updated>2023-10-04T01:57:05.937Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*_WUw-14e_SxnlSM2e_m7cA@2x.jpeg" /><figcaption>Image Courtesy: towardsdatascience.com</figcaption></figure><p><strong>Transformer Architecture for building Language Model</strong></p><p>Transformer architecture dramatically improved performance of language models compared to previous state of the RNN based models and led to explosion in generative capability. The significant difference for which transformer architecture was immensely successful is because of its ability to learn the context and relevance of all the words in a sentence and not just based on its neighbors.It applies attention weights to the relationship of each word with every other word no matter where the word is located in the sentence. Lets see an example question as “The teacher taught the student with the book.”. Transformer algorithms have the ability to understand the context that it’s about the book, the teacher and someone is talking about teaching the book. The attention in this way over the whole sentence and between every word that the transformer learns during the training process makes it most powerful. Let’s look at this diagram called attention map to illustrate the concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r-1w6_gQ7L0T8bp2Cl6xRQ.png" /><figcaption>Encoder-Decoder Architecture from Original Transformer Paper Attention Is All You Need”</figcaption></figure><p>There is a strong attention or relation between book and teacher and book and student. This is called self-attention and the ability to learn these relationships gives the model the ability to encode the context of the sentence and therefore better understand the meaning.</p><p>Now that we know why transformer models are so powerful in language understanding, let’s look at a high level of the architecture of transformers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yH2iAp6KgVVZI1LAr6tFAA.png" /><figcaption>Encoder-Decoder Architecture from Original Transformer Paper Attention Is All You Need”</figcaption></figure><p>Transformer architecture has two components: encoder and decoder. They work in conjunction and have similarities between them. This diagram is taken from the original transformer paper. The input to the model is tokens which is a numerical representation of words in the sentence. This number is generated based on the position of the word in the dictionary out of all possible words. Token IDs can be generated based on complete words or part of words. It Is important to note that once you select a tokenizer for training you have to use the same tokenizer to generate the text for model output. Now these tokens are passed along to an embedding layer. Embedding generates a mullti-dimensional vector and each token is represented as a unique vector in the embedding space. Embedding has been there for a long time in language models and the most popular was word2vec using the same concept. The location and angular distance between these vectors represents the relationship between words, however it still doesn’t encode the context of the entire sentence and meaning of the words in the context if they are used. Beside the token for each word a positional tokens are also generated and passed into the encoder and decoder, positional tokens carry the order of the words in the sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mSYqlKQOER2bZ8ap" /><figcaption>Word embedding representation in two domensions</figcaption></figure><p>Once these tokens and corresponding vectors are ready they are passed along both encoder and decoder to the self attention layer. Here the model analyzes the relationships between words in the input sequence and attends different parts of the sentence to better capture contextual dependencies between words. The output of the self attention layer is weights of the model encoding the importance of the word and relationship relative to all other words in the input sequence. In a transformed model there are parallel self attention that learn simultaneously and independently to each other. The number of attention layers or attention heads determines how large the model would be constrained by many factors such as compute limitation, budget, time, amount of input data the details of which we will cover later. The intuition here is that different self attention is learning different aspects of the input sequence such as relationship between people aspect and another attention is relationship of action words. Given a sufficient amount of examples in the input data and training for enough time, the model will eventually pick up these nuances in the language. Now the language model for a specific task will use a fully connected feed forward neural network to produce output depending on the task type and then through a softmax layer. The output of this layer is a vector of logits proportional to the probability score of each word in the vocabulary. In general terms, the highest probability word wins and is predicted, however that may not always be the case and there are other methods of choosing predicted words in the sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESGFkz7Mc5BXpH3MaQRgfQ.png" /><figcaption>Encoder-Decoder Architecture from Original Transformer Paper Attention Is All You Need”</figcaption></figure><p>Getting back to the encoder and decoder component of transformer architecture, let’s look at it more closely. Encoder encodes the input sequences into a deep representation of the structure and meaning of the input. The decoder working from the input token uses the encoder’s contextual understanding to generate new tokens. It does this in loop to continually improve till it meets an end condition. There are many variations of this architecture that evolved over time. The encoder only model works as a sequence to sequence model. BERT is an example of an encoder only model used for classification tasks such as sentiment analysis. Encoder-decoder model performs well on sequence to sequence tasks such as translation and an example is BART model and T5. Finally, decoder only models are growing in popularity these days and there are many examples like GPT, BLOOM and LLaMA.</p><h3>Configuration of Generative Models</h3><p>As we have seen, the output from the transformer encoder-decoder model is a probability distribution across the entire vocabulary, hence prediction has to choose the best result out of all. Now all model variations of the transformer used to use greedy decoding and predict the word with highest probability. This method works very well in short generation problems but often most frequent words or repeated sequences of words get overweight and influence the prediction. The later model uses random weighted sampling which selects a word irrespective of their probability. This method is good for removing the influence of repeated words but the model could predict creative word and sequence and cause the outcome to wander off. There are also hybrid approaches where it first selects top probability scores and then randomly selects final predictions. One of the popular methods is top p top k and then random sampling method. You limit the model to choose from only K tokens and limit results that have combined probabilities equal to p. This adds randomness in selection while preventing selecting totally improbable words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gHX7F6Wf0vOD7owbQjNQ4A.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>In the above example in the greedy method word cake is selected having highest probability, however using random weight sampling banana could be selected. You can use another configuration called temperature which is a factor that influences the shape of the probability distribution in the softmax layer. Higher temperature has higher randomness in the final selection. In contrast to the top k and top p parameters, changing the temperature actually alters the predictions that the model will make. If you choose a low value of temperature, say less than one, the resulting probability distribution from the softmax layer is more strongly peaked with the probability being concentrated in a smaller number of words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBBWNITfWu-zzbuT8X8vaA.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>In the above example where the temperature value is less than one there is a less randomness and probability distribution is peaked against word cake and most likely this word will be selected. In the second example higher temperature values greater than 1 increase randomness and make probability distribution flatter smoother making words like banana to be possibly selected. If you keep the temperature value 1 then softmax function and distribution will be unaltered. For illustration lets see the image below to see how prompts and these configurations can be tuned together to influence optimal model results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*43YDy-NcLq_I1rEme0EVQA.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p><strong>How business can quickly start LLM projects — LLM Lifecycle</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LUquSu-KGu60RkmRIWKZIA.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>Building your own model from scratch and deploying in production that brings expected outcomes used to take over a year for business before. It involved tedious tasks of collecting data, cleaning and curating, developing models which often required specialized skills of AI and ML and intensive process of building training, evaluation and inference pipelines. With many pioneers of AI like Google, Microsoft and Facebook open sourced their hard earned models that others could leverage using techniques like transfer learning and fine tuning. As more and more large language models are now available in the open source many use cases can be served by having defined business outcome, use case, prompt engineering and fine tuning configuration parameter as discussed above bringing time to idea to result in weeks. Having said that, it doesn’t mean that training your own model from scratch is no longer needed. Often use cases that are very specific and unique to a business and data domain would still need building and training models from scratch. First you choose a framework such as pytorch and then an open source model that suits mostly your business use case. There are plenty of open source models and they are differently trained and have slightly different capabilities. Hence a deeper understanding of these models is very much required.</p><p>The first step is to define the scope of the use case very meticulously to ensure that everyone involved in the project has the same understanding of what is expected from the proiect and what problem it is meant to solve. Choice of framework and most importantly the model significantly dampened on this as different LLMs are trained on different types of data and training and optimization are done differently to have the capability to do better in some types of language problems over others. The size and model architecture varies from model to model. You should think about what problem model will solve, what type of application will use it, do you need a model to perform different tasks including long-form text generation or high degree of capability or a task that is very specific such as sentiment analysis. You also need to define your budget in terms of compute cost and project timeline. This will drive any fine tuning activities and how long you have to optimize the model to improve accuracy. Once you determine a model the next step is to evaluate its performance for the task it is identified and how much fine tuning you need. In this evaluation stage human expertise also can be leveraged to provide feedback to models for improving accuracy. Once the optimization and evaluation is complete you need to define an application and pipeline for model inference where the model will be deployed as an application or API to perform its prediction.There is also a different method of inference and requires optimization for performance using either CPU or GPU compute.</p><p>Now that we have a use case defined, the model selected let’s look at the initial training process. As these models are pre-trained on very large datasets it will be required to further fine tune using task specific datasets for a specific use case. There could be use cases where the model needs to be fine tuned for more than one type of language tasks. In that case it is possible to train models for multi-tasking.</p><p>The pretrained model is more likely trained on vast amounts of data from multiple sources belonging to different domains. Most common data sources are the internet, wikipedia and other large data repositories. One other consideration for fine tuning besides improving model performance for a specific use case or domain would be to remove any bias, moral or ethical concerns, avoid copyright infringement issue, compliance and meet data privacy requirements. The three types of models, encoder only, encoder-decoder and decoder only models are pre-trained for different objectives and often perform best in different tasks. Encoder only autoencoder models are trained for denoising objectives hence good for token generation tasks and understanding bi-directional context. They are good for classification tasks such as sentiment analysis or token based tasks such as named entity recognition. Some well known models are BERT and RoBERTa. Decoder only autoregressive models are pre-trained using a casual language modeling objective, hence the objective is to predict the next token based on the sequence of input tokens. Decoder only models often perform pretty well with zero shot or few shot learning and gaining popularity now as they can perform a range of tasks. The most popular examples of autoregressive models are GPT and BLOOM.</p><p>The final variation of the transformer model is the sequence-to-sequence model that uses both the encoder and decoder parts of the original transformer architecture. The exact details of the</p><p>Pre-training objectives vary from model to model. A popular sequence-to-sequence model T5, pre-trains the encoder using span corruption, which masks random sequences of input tokens. Those mass sequences are then replaced with a unique Sentinel token, shown here as x. Sentinel tokens are special tokens added to the vocabulary, but do not correspond to any actual word from the input text. The decoder is then tasked with reconstructing the mask token sequences auto-regressively. The output is the Sentinel token followed by the predicted tokens. You can use sequence-to-sequence models for translation, summarization, and question-answering. They are generally useful in cases where you have a body of texts as both input and output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j0TeuekZq4TZpK9svOKp4A.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>Now here I will digress from the model to compute a little in order to see some of the implication of training strategy on compute and on the budget for all practical purposes. In the real world we work within some constraints. It could be a certain budget limit, access to certain computer resources and time. A good training strategy is the one which can get the most out of all these constraints in place. Lets look at the constraints in real life in which mostly models are developed and deployed and we adjust our strategies based on these factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G2C-bsHtMT2EKJVsDb5jeA.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>As models grow in size and complexity, often a single specialized computer such as a GPU may not be sufficient for training. Now GPT, BlooM and T5 these models are pre-trained using very large datasets and utilizing expensive distributed GPU compute, however depending on the objective of the fine tuning it also can be quite compute intensive and may exceed limitations of a single GPU machine. In fact even if the model does fit into memory of a single GPU machine it still can take advantage of a cluster of GPU machines to accelerate the training process. The method of using distributed computers for training is to shard the large datasets into different machines and replicate models into these machines to train on subsets of data in parallel. A popular technique developed in pytorch is called distributed data parallel or DDP in short.</p><p>DDP replicates the model in all machines and sends a subset of batches of data into these machines to run training in parallel and then perform a synchronization operation for model weight updates. However as it copies models into all available machines and if the model (model parameters could be in billions) is too large for a single machine then look for solutions like Pytorch’s Fully sharded data parallel or FSDP and Microsoft’s zero redundancy optimizer or ZeRo. ZeRO stage 1 shards the model’s optimizer and reduces memory footprint by a scale of 4 and stage 2 shards gradient across chips and reduces memory footprint by a scale of 8 and stage 3 parallelize all component of model and provide linear scale back of memory footprint linearly with number of GPUs. FSDP does data parallelism as in DDP and also deploy one of the strategies mentioned for ZeRo.</p><p>As models grow in size and complexity, often a single specialized computer such as a GPU may not be sufficient for training. Now GPT, BlooM and T5 these models are pre-trained using very large datasets and utilizing expensive distributed GPU compute, however depending on the objective of the fine tuning it also can be quite compute intensive and may exceed limitations of a single GPU machine. In fact even if the model does fit into memory of a single GPU machine it still can take advantage of a cluster of GPU machines to accelerate the training process. The method of using distributed computers for training is to shard the large datasets into different machines and replicate models into these machines to train on subsets of data in parallel. A popular technique developed in pytorch is called distributed data parallel or DDP in short.</p><p>DDP replicates the model in all machines and sends a subset of batches of data into these machines to run training in parallel and then perform a synchronization operation for model weight updates. However as it copies models into all available machines and if the model (model parameters could be in billions) is too large for a single machine then look for solutions like Pytorch’s Fully sharded data parallel or FSDP and Microsoft’s zero redundancy optimizer or ZeRo. ZeRO stage 1 shards the model’s optimizer and reduces memory footprint by a scale of 4 and stage 2 shards gradient across chips and reduces memory footprint by a scale of 8 and stage 3 parallelize all component of model and provide linear scale back of memory footprint linearly with number of GPUs. FSDP does data parallelism as in DDP and also deploy one of the strategies mentioned for ZeRo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k1bjHAchgbh5yVUG2-MlQA.png" /><figcaption>Source: Brown et al. 2020 “Language Models are Few shot Learners”</figcaption></figure><h3>Scaling Laws and compute models of LLMs</h3><p>Now we can dive into this more and try to understand compute models for large language models although this applies to other domains of AI such as vision. As we have already seen, LLMs are very large and often have billions of model parameters that needs to fit into memory for training and without enough memory training processes will fail with an out of memory error. In order to understand how large models require a lot of memory let’s look at some math. Each parameter can take up to 32-bit float memory as a real number in a computer. a 32-bit float takes up 4 byte of space and so, a billion parameters will take 4 GB of GPU memory which is a lot. Other than model parameters, you need to store optimizer states, variables, gradients and activation functions. This all could add up to 20 times of the memory needed for parameters and could be 80 GB GPU memory necessary for training. Although memory capacity is continuously improving it is still hard to get this much of memory even in the data center. NVIDIA GPU A100 now has 80 GB of memory and is not often easy to get. Also, the latest LLMs have multi billions of parameters that require orders of magnitude of memory footprint. This led us to compromise on the precision of floating point and choose options like 16-bit half precision or even 8-bit int.The corresponding data types used in deep learning frameworks and libraries are FP32 for 32-bit full position, FP16, or Bfloat16 for 16-bit half precision, and int8 eight-bit integers. The range of numbers you can represent with FP32 goes from approximately 3*10^-38 to 3*10³⁸. By default, model weights, activations, and other model parameters are stored in FP32. Here comes a technique called quantization. Statistically projects the original 32-bit floating point numbers into a lower precision space, using scaling factors calculated based on the range of the original 32-bit floating point numbers. Let’s look at an example. Suppose you want to store a PI to six decimal places in different positions. Floating point numbers are stored as a series of bits, zeros and ones. The 32 bits to store numbers in full precision with FP32 consist of one bit for the sign where zero indicates a positive number, and one a negative number. Then eight bits for the exponent of the number, and 23 bits representing the fraction of the number. The fraction is also referred to as the mantissa, or significant. It represents the precision bits off the number. If you convert the 32-bit floating point value back to a decimal value, you notice the slight loss in precision. For reference, here’s the real value of Pi to 19 decimal places. Now, let’s see what</p><p>happens if you project this FP32 representation of Pi into the FP16, 16-bit lower precision space. The 16 bits consists of one bit for the sign, as you saw for FP32, but now FP16 only</p><p>assigns five bits to represent the exponent and 10 bits to represent the fraction. Therefore, the range of numbers you can represent with FP16 is vastly smaller from negative 65,504 to positive 65,504. The original FP32 value gets projected to 3.140625 in the 16-bit space. Notice that you lose some precision with this projection. There are only six places after the decimal point now. You’ll find that this loss in precision is acceptable in most cases because you’re trying to optimize for memory footprint. Storing a value in FP32 requires four bytes of memory. In contrast, storing a value on FP16 requires only two bytes of memory, so with quantization you have reduced the memory requirement by half. The AI research community has explored ways to optimize16-bit quantization. One datatype in BFLOAT16, has recently become a popular alternative to FP16. BFLOAT16, short for Brain Floating Point Format developed at Google Brain has become a popular choice in deep learning. Many LLMs, including FLAN-T5, have been pre-trained with BFLOAT16. BFLOAT16 or BF16 is a hybrid between half precision FP16 and full precision FP32. BF16 significantly helps with training stability and is supported by newer GPU’s such as NVIDIA’s A100. BFLOAT16 is often described as a truncated 32-bit float, as it captures the full dynamic range of the full 32-bit float, which uses only 16-bits. BFLOAT16 uses the full eight bits to represent the exponent, but truncates the fraction to just seven bits. This not only saves memory, but also increases model performance by speeding up calculations. The downside is that BF16 is not well suited for integer calculations, but these are relatively rare in deep learning. For completeness, let’s have a look at what happens if you quantize Pi from the 32-bit into the even lower precision eight bit space. If you use one bit for the sign INT8 values are represented by the remaining seven bits. This gives you a range to represent numbers from negative 128 to positive 127 and unsurprisingly Pi gets projected two or three in the 8-bit lower precision space. This brings new memory requirements down from originally four bytes to just one byte, but obviously results in a pretty dramatic loss of precision. BFLOAT16 has become a popular choice of precision in deep learning as it maintains the dynamic range of FP32, but reduces the memory footprint by half. Many LLMs, including FLAN-T5, have been pre-trained with BFOLAT16. Lookout for a mention of BFLOAT16 in next week’s lab. Now let’s return to the challenge of fitting models into GPU memory and take a look at the impact quantization can have. By applying quantization, you can reduce your memory consumption required to store the model parameters down to only two gigabyte using 16-bit half precision of 50% saving and you could further reduce the memory footprint by another 50% by representing the model parameters as eight bit integers, which requires only one gigabyte of GPU RAM. Note that in all these cases you still have a model with one billion parameters. As you can see, the circles representing the models are the same size. Quantization will give you the same degree of savings when it comes to training. As you heard earlier, you’ll quickly hit the limit of a single NVIDIA A100 GPU with 80 gigabytes of memory. When you try to train a one billion parameter model at 32-bit full precision, you’ll need to consider using either 16-bit or eight bit quantization if you want to train on a single GPU and remember, many models now have sizes in excess of 50 billion or even 100 billion parameters. Meaning you’d need up to 500 times more memory capacity to train them, tens of thousands of gigabytes. These enormous models dwarf the one billion parameter model we’ve been considering, shown here to scale on the left. As modal scales beyond a few billion parameters, it becomes impossible to train them on a single GPU. Instead, you’ll need to turn to distributed computing techniques while you train your model across multiple GPUs. This could require access to hundreds of GPUs, which is very expensive. Another reason why you won’t pre-train your own model from scratch most of the time. One of the metrics for model scale is petaFlop per second a day which is defined as the number of floating point of operation per second running for the entire day. Just to set the stage one petaFlop per second a day would take 8 NVIDIA V100 for a complete day to run and more recently powerful two NVIDIA A100 will be required to run a complete day. The picture shows different large language model and number of petaFlop operations required to train these models.</p><p>The data is clear about the case that these large models require huge amounts of compute and therefore are often expensive to train and evaluate. You can pretty much see a linear relationship between the amount of data and amount of parameters with the model performance. So as more and more data is available and more and more advanced computers are coming up these models are getting better and better. Based on the paper by Kaplan et al. we can see that model performance has nearly linear relationship with compute and training datasize.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CfKwvTE49uMUl5qLNTdnyw.png" /><figcaption>Source: Kaplan et al. 2020 Scaling Laws for Neural Language Models</figcaption></figure><p>In a paper published in 2022, a group of researchers led by Jordan Hoffmann, Sebastian Borgeaud and Arthur Mensch carried out a detailed study of the performance of language models of various sizes and quantities of training data. The paper finds out what would be the optimal dataset size and model parameters for a given compute budget which is more aligned with real life situations. The paper is often referred to as Chinchilla paper. The Chinchilla paper hints that many of the 100 billion parameter large language models like GPT-3 may actually be over parameterized, meaning they have more parameters than they need to achieve a good understanding of language and under training so that they would benefit from seeing more training data. The important takeaway from the paper is that the optimal dataset size is about 20 times larger than the model parameter size. For example, for a 70 Billion parameter model the token size is optimal at 1.4 Trillion or 20 times the number of parameters. LLaMA is a model that is trained using Chinchilla paper recommendation and one of the findings in the paper is that compute optimal Chinchilla models outperform non compute optimal models like GPT-3. One other interesting finding thet Bloomberg GPT model achieved good performance with 50 Billion parameters and domain specific model like Bloomberg GPT a pretarined from scratch is with financial data to get good performance on domain specific use cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0ZjIY93YoUqVfAWbTxqsQ.png" /><figcaption>Source: Wu et al. 2023 “BloombergGPT — A large language model for Finance”</figcaption></figure><p>Bloomberg GPT model is trained with general data (49%) and financial data (51%) to achieve the good results with optimal compute model guidance from Chinchilla paper. It has slightly higher parameters than what Chinchilla paper recommends and lower tokens than recommended. This shows the limitation of domain data may force us to use larger models with higher parameters.=</p><p><strong>Instruction Fine tuning to improve performance for variety of tasks</strong></p><p>As said previously zero shot, few shot type prompt engineering may not be sufficient to get desired performance from large pretrained language models and hence instruction fine tuning which uses supervised machine learning technique can be very useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ha81hz0xoS8DWue6VCwnw.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>In this technique you build up a set of data with labels depending on the tasks. For example for translation tasks it will be a sentence in one language and a translated sentence into the target language as a label. For sentimen review it will have review sentences and labels with sentiment value. Many of these datasets are already available for fine tuning large language models such as amazon product review dataset. These techniques can significantly improve performance based on task specific fine tuning. The same also can be used for multi task models fine tuning where supervised training can be used with datasets for various tasks that model is to be improved for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxmI1cdhavCaCPhsmXNi0A.png" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>Flan T5 is a good example of a general purpose model that performs well on more than 140 tasks. One of the dataset It is trained with SAMsum dataset that is publicly available has 16 thousands dialogue like conversation. Although these may be good for most of the use case, however for a specific business use case of conversational dialogue such as customer service chatbot a company may want to use their own business data like dialogSum that are more human like conversation datasets having its own business terminology to better fine tune large language pretrained models.</p><p>There is a potential downside for task specific fine tuning is that the model can perform very well on specific tasks and it degrades in performance in other tasks which is known as catastrophic forgetting in deep learning. To avoid catastrophic forgetting models can be fine tuned with multiple tasks with 100 thousands of examples for each task. The one other way is to execute parameter efficient fine tuning or PEFT in short.</p><h3>Parameter efficient Fine Tuning</h3><p>We have seen that fine tuning large models requires hundreds of gigabytes of memory space and is computationally expensive. Besides model parameters, it also needs to store optimizer states, gradients, forward activations and temporary variables throughout the training process. To tackle this issue, PEFT introduces a method where instead of updating all weights of the model just tune selective weights using supervised learning methods. Some PEFT methods freezes most of the parameters of the original model and only tune a subset of parameters such as specific layers or components. Other techniques don’t update the weights of the original model, instead add new parameters or layers and fine tune this part only. With just 15–20% parameter update of the original models PEFT can significantly reduce entire model tuning overhead and improve performance required for the use cases. Often PEFT can be performed in a single GPU machine and it also solves catastrophic forgetting problems as it is leaving most of the model parameters unchanged. For multi-task fine tuning separate PEFT can be performed and multiple model versions can be used for inference serving different use cases. There are also several methods of PEFT available depending on the objectives such as parameter efficiency, compute efficiency, memory efficiency, training speed, model quality and inference costs. There is also a reparameterization method where a reduced number of parameters are generated using low rank transformation and updated and one of those such methods is LoRA. Lastly, additive methods freeze all parameters of the original model instead introduce new trainable layers typically after the attention and feed forward layers of the original model architecture. Soft-prompt methods focus on manipulating the input to achieve better performance such as adding trainable parameters to prompt embeddings or keeping the input fixed and retraining the embeddings weight.</p><h3>Reinforcement Learning From Human Feedback</h3><p>In reinforcement learning, models continuously learn by feedback from the environment through interactions. Using positive or negative rewards to each prediction by the model, it improves the model’s action for the next best prediction incrementally. In RLHF human feedback helps LLMs to be better at being most useful to the objective, relevant and harmless One specific example could be personalized AI assistance where it learns from individual preferences via interacting with the individual and updating model’s behavior using continuous feedback. In the RLHF method an alignment criteria is defined such as helpfulness which will define the model’s objective and therefore how it is performing is measured by the alignment objective. In case it is asked House is hot and the answers are 1. Nothing you can do about hot houses and 2. You can turn on air conditioning, the second answer will get rewarded by humans and the model will learn from this feedback. The human labeller can use instructions for how to give this feedback. Scaling instruction fine-tuned language model by Chung et al gave many examples of these instructions and results achieved through these RLHF mechanisms. To convert human feedback into training data you can use pairwise completion and reward and sort them by their rank.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vYv99reRMEfWyKb_" /><figcaption>Source: Steiennon et al 2020 “Learning to summarize from human feedback”</figcaption></figure><p>Use a reward model as a binary classifier to classify the model’s completion such as toxic or non-toxic completion. Use the output from the reward model pass it to a reinforcement model to update the weight of the instruction fine tuned model to further improve alignment driven performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kZM_gVwyIc4IDlr0" /><figcaption>Image Courtesy: <a href="https://www.coursera.org/learn/generative-ai-with-llms/home/week/1">https://www.coursera.org/learn/generative-ai-with-llms/</a></figcaption></figure><p>There are several choices for selecting a RL algorithm and a popular choice is PPO or proximal policy optimization which GPT models use. PPO is pretty complicated but at a high level it has two phases. In phase one it runs an experiment where LLM generates several completion of the prompt which will be then used against a reward model in phase-2.The expected reward objective is used as objective for the PPO policy. The quantity is generated using a separate head of the LLM called the value function. Lets say that a LLM generates several completion based on a prompt. It is then passed to a reward model which then generates a reward for each completion. The completion total rewards are then compared with the future reward approximated by the value function of the LLM and then PPO will minimize the loss between these two rewards for a given state S. This makes a small update to the weights of the LLM in order to improve approximation by the value function. As the reward model was rained using human feedback hence this produces a human in the loop scenario for AI optimization. These techniques are constantly evolving and AI performance is continuously improving with more research and experiments. With that I think AI has achieved great results so far but we still have a long journey into building AI applications that augment human ability and improve lives and be fair to every human being at all times.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc4220a36a69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I passed Oracle Cloud Professional Architect Exam in Two Weeks]]></title>
            <link>https://medium.com/@gautam.karmakar/how-i-passed-oracle-cloud-professional-architect-exam-in-two-weeks-971f3824c68b?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/971f3824c68b</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[oracle-cloud]]></category>
            <category><![CDATA[oracle]]></category>
            <category><![CDATA[certifica]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Fri, 25 Mar 2022 22:26:04 GMT</pubDate>
            <atom:updated>2022-03-25T22:26:04.465Z</atom:updated>
            <content:encoded><![CDATA[<h3>Oracle Cloud Infrastructure 2021 Architect Associate Certification</h3><p>I have passed both Oracle Cloud infrastructure architect associate and professional in 3 weeks and wanted to share how I approached learning and practicing oracle cloud technologies with limited time. I came with a background in AWS, Azure and GCP cloud services prior working with Oracle cloud which really helped me to grasp the concepts pretty well pretty fast. But of course there are nuances in every cloud provider’s services specially in how they manage identity and access and networking. Also, each cloud provider has different kinds of PaaS and SaaS in contrast to IaaS that I needed to learn. Here I will share how I prepared for the exam and my answers to the official and free practice test available for everyone which I highly recommend to complete and understand.</p><h3>About Oracle Cloud:</h3><p>Whether you are a global enterprise or a small business or an ISV, Oracle cloud has infrastructure and services to run your information technologies at scale. From infrastructure services such as Compute, network and storage to cloud native services and support of open source technologies Oracle has reimagined and redesigned how cloud technologies and services can fuel next-gen digital transformation. It started with an idea to solve today’s and tomorrow’s business and not yesterday’s problem to help business to rediscover their customer, products and services in a way to create clear distinction in the market. To learn more about why customers are choosing oracle cloud please visit <a href="https://www.oracle.com/cloud/why-oci/">https://www.oracle.com/cloud/why-oci/</a>.</p><p>Here are the services the Oracle Cloud infrastructure offers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UOudhQJ6a9hdIaw8" /></figure><h3>About the Exam:</h3><p>Oracle cloud infrastructure architect associate exam is designed for those who are or are going to be Oracle cloud architects to design and implement oracle cloud infrastructure and services for customers. The exam will test your ability to understand the foundational concepts and understanding of applicability for each product and service.</p><p>Here are the exam topics that will be tested for understanding (from the official exam page) and corresponding documentation links for preparation.</p><p>Compute</p><ul><li>Choosing the right compute — <a href="https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm#virtualmachines">Compute Shapes</a></li><li>Compute Instance Configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/Compute/Concepts/instancemanagement.htm">Instance Configuration</a></li><li>Automatic scaling configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/Compute/Tasks/autoscalinginstancepools.htm">Autoscaling</a></li><li>Description of OCI Computing Image Options — <a href="https://docs.oracle.com/en-us/iaas/Content/Compute/Concepts/computeoverview.htm">Compute Images</a></li><li>OS management description and configuration — <a href="https://docs.oracle.com/en-us/iaas/os-management/home.htm">OS Management Service</a></li></ul><p>Network</p><ul><li>Implementation and management of virtual cloud networks, Public and Private Subnet Descriptions, Public and private IP addresses and virtual NIC descriptions, Virtual cloud network routing configuration, Security list, and network security group configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/overview.htm">Networking Complete Reference</a></li><li>Implementation and management of VCN connections</li><li>Understanding VCN connection options — <a href="https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/VCNpeering.htm#Access_to_Other_VCNs_Peering">VCN Peering</a></li><li>Site-to-site VPN configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/Network/Tasks/overviewIPsec.htm#top">S2S VPN</a></li><li>Fast connect description — <a href="https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/fastconnect.htm">FastConnect</a></li><li>Configuring local and remote peering</li><li>DNS and traffic management configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/DNS/Concepts/dnszonemanagement.htm#Overview_of_the_DNS_Service">DNS Management</a></li><li>Public and Private DNS Zone Descriptions</li><li>Traffic management configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/TrafficManagement/Concepts/overview.htm">Traffic Management</a></li><li>Load balancing description — <a href="https://docs.oracle.com/en-us/iaas/Content/Balance/Concepts/balanceoverview.htm">Load Balancing</a></li><li>layer-7 Load balancer description and configuration</li><li>layer-4 network load balancer description and configuration</li></ul><p>Storage</p><ul><li>Deploy and manage block storage — <a href="https://docs.oracle.com/en-us/iaas/Content/Block/Tasks/creatingavolume.htm">Create and Manage Block Volume</a></li><li>Understanding the performance layer of block volumes</li><li>Deploy block and boot volumes — <a href="https://docs.oracle.com/en-us/iaas/Content/Block/Concepts/bootvolumes.htm">Deploy and manage Boot Volume</a></li><li>Volume group, backup, clone configuration</li><li>Object storage configuration and management — <a href="https://docs.oracle.com/en-us/iaas/Content/Object/Concepts/objectstorageoverview.htm">Object Storage</a></li><li>Understanding the object storage layer</li><li>Object storage security description — <a href="https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usingpreauthenticatedrequests.htm#Using_PreAuthenticated_Requests">Object Storage PAR</a></li><li>Object storage versioning, lifecycle management, and maintenance rule implementation — <a href="https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/usinglifecyclepolicies.htm">LifeCycle Management</a></li><li>File storage implementation and management</li><li>File storage configuration</li><li>File storage security description — <a href="https://docs.oracle.com/en-us/iaas/Content/File/Concepts/securitylayers.htm">File Storage Security</a></li></ul><p>Database</p><ul><li>Understanding database options — <a href="https://docs.oracle.com/en-us/iaas/Content/Database/Concepts/databaseoverview.htm">DB Options in OCI</a></li><li>Autonomous Database Configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/Database/Concepts/adboverview.htm">Autonomous DB features</a></li><li>DB system configuration and management</li><li>MySQL and NoSQL Database Services Description</li></ul><p>Observation and Management — <a href="https://docs.oracle.com/en-us/iaas/Content/Monitoring/Concepts/monitoringoverview.htm">Monitoring Service</a></p><ul><li>Understanding the observation and management platform</li><li>Monitoring configuration and alarm settings</li><li>Logging configuration — <a href="https://docs.oracle.com/en-us/iaas/logging-analytics/index.html">Log Analytics</a></li></ul><p>Identity and Access Management — <a href="https://docs.oracle.com/en-us/iaas/Content/Identity/Concepts/overview.htm">IAM Overview</a></p><ul><li>Understanding AuthN and AuthZ</li><li>Compartment implementation</li><li>Implementation of conditional and extended policies</li><li>Configuring dynamic group and tag-based access control</li></ul><p>Security</p><ul><li>Understanding security services — <a href="https://docs.oracle.com/en-us/iaas/Content/Security/Concepts/security_features.htm">Security Services</a></li><li>Configuring Data Guard, Security Zones and Security Advisors — <a href="https://docs.oracle.com/en-us/iaas/security-zone/using/managing-security-zones.htm#managing-security-zones">Security Zones</a></li><li>Vulnerability Scan and Web Application Firewall Configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/WAF/Firewall/firewall_management.htm">OCI Firewall Management</a></li><li>OCI Vault Description and Configuration — <a href="https://docs.oracle.com/en-us/iaas/Content/KeyManagement/Concepts/keyoverview.htm">Key Vault Service</a></li></ul><p>The good thing is associate certification is free and you can find information on how to register for the exam here <a href="https://education.oracle.com/oracle-cloud-infrastructure-2021-architect-associate/pexam_1Z0-1072-21">https://education.oracle.com/oracle-cloud-infrastructure-2021-architect-associate/pexam_1Z0-1072-21</a></p><p><strong>How did I prepare in two weeks:</strong></p><p>Coming from AWS/Azure/GCP background I knew that I have to familiarize myself with Oracle cloud products and services in terms of concepts to understand what each of these are supposed to be used for. I have read product documentation extensively starting with<a href="https://docs.oracle.com/en-us/iaas/Content/GSG/Concepts/concepts.htm"> Key Concepts and Terminology (oracle.com)</a> to just have a better idea on key products and services. Understanding Oracle cloud region, availability domain and fault domain. How they can help architecting a global, fault tolerant multi-tier application and also provide resiliency from failure and disaster. One thing that stuck to me is the organization of resources and management by tenancy. Understanding tenancy, compartments for organizing resources and applying policy and budget restrictions was very important. You can find the details<a href="https://docs.oracle.com/en-us/iaas/Content/GSG/Concepts/settinguptenancy.htm"> here</a>.</p><p>I focused on Identity and access management service because I knew it would be one thing that is an important part of the exam and also often makes one cloud service differ from each other. I have focused on understanding users, groups, policies and how they become effective in performing day to day tasks from the end user perspective. The quick way of learning is by doing hands-on work and seeing how access and permissions really work. You can find detailed step by step instructions to practice<a href="https://docs.oracle.com/en/cloud/get-started/subscriptions-cloud/secure-platform-cloud-services.html"> here</a>. I spent a complete day practicing on IAM and seeing myself creating a small practice application on a E4 VM to see how these policies and roles work.</p><p>On the third day I dedicated it to networking as it is the most critical component of the exam and understanding how to design and implement a robust network architecture that is highly secured is very important. This also includes network security components such as VCN, IP address ranges, subnets, security list, network security group, routing, NAT Gateway, Service Gateway, Internet Gateway, bastion host, load balancer, Traffic manager, DNS service, FastConnect, VPN service, DRG. I put specific emphasis on communication and connectivity between subnets within a VCN, VCN peering between regions or within a region, connecting to on-premise and multi-tier application network design. I have found that most of the information required me to answer questions on networking<a href="https://docs.oracle.com/en-us/iaas/Content/Network/Concepts/overview.htm"> here</a>.</p><p>Fourth day I focused on storage and data services like databases and data warehouses. Storage wasn’t that difficult as pretty much the same as others but focus on boot disk, block storage for drives, file storage and its unique features, object storage, how application can securely access these storage types, storage tiers, lifecycle management, backup and archival methods. Different backup options within a AD or across region, cloning vs backup, how and when a VM can be backed up and what is the best option given a scenario is very important to understand.</p><p>Rest of the week I brushed up on several topics that are Oracle cloud specific. After knocking down components that I am already familiar with in the next 3 days I focused on components like <a href="https://docs.oracle.com/cd/B19306_01/server.102/b14239/concepts.htm#g1049956">Data Guard</a>, <a href="https://docs.oracle.com/en/cloud/paas/goldengate-service/index.html">Golden Gate</a>, <a href="https://docs.oracle.com/en-us/iaas/Content/APIGateway/Concepts/apigatewayoverview.htm">API Gateway</a>, <a href="https://docs.oracle.com/en-us/iaas/Content/DataTransfer/home.htm">data transfer</a>, <a href="https://docs.oracle.com/en-us/iaas/Content/Logging/home.htm">Logging</a> &amp; <a href="https://docs.oracle.com/en-us/iaas/logging-analytics/doc/logging-analytics1.html">Log Analytics</a>, <a href="https://docs.oracle.com/en-us/iaas/Content/ContEng/Concepts/contengoverview.htm">Oracle Kubernetes Service</a>.</p><p>Security was a big topic and I expected 5–10 questions specific to security, so I spent a good amount understanding network security layers including public and private subnet design, security zones, data encryption options, key management, securing compute, block volume, use principals, service Gateway, outbound internet access via NAT and inbound access via internet gateway. Securing file storage using NFS ACL, Posix control and IAM policy. WAF and policies.</p><p>The best place to learn the most important aspects of securing infrastructure, data and application is <a href="https://docs.oracle.com/en-us/iaas/Content/Security/Reference/configuration_security.htm">Security Best Practice</a>.</p><p>Last few days I spent on Oracle’s official learning path page <a href="https://mylearn.oracle.com/learning-path/become-an-oci-architect-professional/35644/97984">Learning Path</a> where I could quickly brush through videos at 1.25X speed and also complete all the skill checks paying attention to questions I was getting wrong. As the practice exam wasn’t timed, I used my own time tracking to see how long I was taking to answer each question but more importantly reading through all the questions carefully gave me insights to see where the critical information was to be focused on.</p><p><strong>Conclusion:</strong></p><p>It helped me to come from another cloud service background to quickly understand key concepts in Oracle cloud. But even if you are starting with little to no cloud knowledge, going through these documents and practicing using Oracle’s free tier should give you enough to sit for the exam confidently. Following official documentation and performing each operation like creating a tenancy, compartment, launch a VM, set up group, policy for access, running a sample application, setting up VCN and subnets, peering, using storage, creating load balancer, routing traffic and launching database, taking backup, setting up data guard will help you to clear the concepts that you read. Hopefully this post was helpful for you to prepare for your Oracle professional architect exam. Please let me know how you cleared your certification exam.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=971f3824c68b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AZ-220: Azure IOT Developer Specialty Certification]]></title>
            <link>https://medium.com/@gautam.karmakar/az-220-azure-iot-developer-specialty-certification-155465c94b05?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/155465c94b05</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[certification]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 20:41:37 GMT</pubDate>
            <atom:updated>2020-07-10T21:22:42.456Z</atom:updated>
            <content:encoded><![CDATA[<h3>Become An Azure IoT Developer Specialist</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*pz3WsSAwpOCWROTXdnyhFA.jpeg" /><figcaption>Intel: Internet of things</figcaption></figure><h3>Introduction:</h3><p>Microsoft recently announced Azure IoT developer specialty certification AZ-200. This exam is to certify specialized skills in developing applications that can run on the IoT device and IoT edge. As a skilled IoT developer you should be able to manage the life cycle of devices., configure and maintain using cloud service and other tools. You should be able to deploy a solution using Azure IoT product offering, manage device topology, connectivity, debugging, and security, as well as collect, process and manage IoT related data pipelines. You should be able to apply analytics using azure data analytics solution for events and messages for real-time data analysis and long-term history retention.</p><h3>Why IOT developer certification is important?</h3><p>There will be 80B devices are connected by 2025 and producing 180 Zeta-byte of data. By end of this year, there will be a total USD 267 Billion investment in some type of IoT solution deployment (Boston Consulting Group 2017) by manufacturers, and around 94% of businesses projected to be using IoT by the end of 2021 (Hypothesis 2019). In the commercial business internet of things (IoT) continues to grow in popularity to solve critical business problems and make better business decisions with speed and accuracy. The IoT application is used cutting across multiple industry types mainly, manufacturing, energy, retail, and healthcare. Based on research performed recently at least 85% of enterprise IoT decision makers suggested that they have IoT projects in either proof of concept, purchase, or use phase. The few among the most popular reasons for IoT adoption are operation optimization, industrial automation, employee productivity, safety and security, condition-based maintenance. Business is witnessing the benefit of adopting an IoT solution in terms of increasing efficiency, increase yield, and improve quality. Along with complexity and investment that are typically the main challenge for IoT adoption, not having enough skilled developers and engineers for IoT applications are also very critical for the adoption and acceleration of IoT solutions for enterprises. Having hands-on expertise of implementing and managing IoT application is definitely will be I high demand as more and more connected devices become ubiquitous and business finds more and more value in collecting, analyzing and using machine-generated data.</p><h3>AZ-200: Azure IoT Developer certification outline</h3><p>As per the exam guideline, these are the skills that will be measured in the exam.</p><p>· Implement the IoT solution infrastructure (15–20%)</p><p>· Provision and manage devices (20–25%)</p><p>· Implement Edge (15–20%)</p><p>· Process and manage data (15–20%)</p><p>· Monitor, troubleshoot, and optimize IoT solutions (15–20%)</p><p>· Implement security (15–20%)</p><p>*As per the exam website above outline will change after July 21st, 2020.</p><p>You can download the exam outline to read in details <a href="https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE4nBeC">here</a></p><p>In the exam, you will be asked a question to test your knowledge in four segments. How to implement IoT products and services, how to configure and manage different components such as device twin, edge module, how to collect data using event hub or service bus endpoint and analyze using stream analytics or time series insights and set up security using keys, certificate and protect device authentication and authorization. You need to know different SKUs available for Azure IoT and their capacity limitation. You need to be familiar with a different messaging protocol such as AMQP, MQTT, and HTTPS and their applicability based on scenarios. You need to able to configure and use endpoint whether built-in or custom and set consumer group. The exam will test knowledge around the usage of device twin to manage states, get metadata for devices, and how to process information and updates using device twin. You will be asked steps to enroll and register devices, set up connectivity with IoT Hub, operate, and manage devices to cloud and cloud to device communication. You need to know how to implement edge devices that sit between device and hub, install edge modules, and containerized images and manage them using REST API. You need to know about IoT central which is a PaaS service from Microsoft for IoT solution and how to configure IoT central, set up devices to communicate but I got only a few questions on IoT central.</p><p>The big part of the questions will be around how to analyze data collected from IoT devices and which azure services will be appropriate. You need to understand the basics of azure stream analytics and time series insights for filter and aggregate data and what are the different functions to use and when. How can you configure stream analytics or time series insights to consume data from the IoT hub endpoint and what are the orders of steps to be performed?</p><p>Monitoring and logging are very important to understand how to set up log monitoring, how to connect metrics to the security center, and what information is needed to troubleshoot connectivity issues or loss of information.</p><p>Security of IoT components is very important to understand, How to securely provision devices, set up a trust relationship, manage keys using HSM, certify devices using root X.509 CA using a chain of trust authentication. What are the different attention mechanisms available and when they are applicable? How to manage the re-provisioning of devices, manage the situation when device security could be compromised.</p><h3>Preparation &amp; Study materials</h3><p>For my preparation, I have used Azure IoT product <a href="https://docs.microsoft.com/en-us/azure/iot-hub/">documentation</a> very closely and started taking notes on key information around the topics of the exam. I have picked up key topics based on the exam outline and started reading the documentation on those topics. I have put more focus on an overview of each component, how-to guides thoroughly, concept pages, and quickstarts. I have completed all these quick starts and how-to guides using my Azure subscription and ensure that I can do that without any issue every time. I have taken notes when I see that these are the functions for some services and use this when you want this. For example, the fastest way to implement an update on devices could be IoT hub direct update method using REST API. You can use device twin to set the desired state for a target room temperature etc. The different SKUs B1/S1, B2/S2 &amp; B3/S3, and their limitation in terms of how much messages they can process per minute. Besides documentation which sometimes I was getting lost into, I completed open online free training by Microsoft that you can find in the <a href="https://docs.microsoft.com/en-us/learn/certifications/exams/az-220#certification-exams">AZ-220</a> exam page itself. You can see IoT related courses and hands-on training and complete them thoroughly to ensure that how each step is executed to complete specific operations in IoT services <a href="https://docs.microsoft.com/en-us/learn/browse/?products=azure-iot">IoT-Learn</a>. Make sure that you read each line of the sample code that you will be running to see the sequences of operation as well as what each command is achieving close to understand how are you are developing the solution. Microsoft created a guide for developer <a href="https://azure.microsoft.com/en-us/resources/iot-developers-guide/">IoT-Developer-Guide</a> that I quickly browsed to brush up concepts and details around installing, configuring, and developing IOT Hub, IoT central, Device management, and Device twin interoperation. I didn’t pay for a practice exam, but you can if you want to also to test your readiness.</p><h3>Lesson Learned</h3><p>In my personal experience, I have built applications using Azure IoT solution that helped customer to achieve their business objective. I also think that more and more industries are finding ways to use machine-generated data for real-time decision making for reducing cost, improving quality of service and safety will create a huge gap of a skilled developer in this field of technology. The exam was helpful for me to brush up and validate my knowledge. But I believe that certificate alone isn’t enough. One needs to have a depth and breadth of knowledge as well as hands-on development experience that can solve real-world problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=155465c94b05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed Machine learning in Azure Databricks]]></title>
            <link>https://medium.com/@gautam.karmakar/distributed-machine-learning-in-azure-databricks-63e3a55ed815?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/63e3a55ed815</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Wed, 19 Dec 2018 04:10:51 GMT</pubDate>
            <atom:updated>2018-12-21T19:34:59.711Z</atom:updated>
            <content:encoded><![CDATA[<p>Azure Databricks provide fully managed, highly scalable and available unified compute platform using Databricks distribution of open source Spark. Azure makes it very simple by automating many steps in setting up and getting started using Databricks spark.</p><p>Developers can get started in minutes to create a workspace, configure and start cluster and build data models using R, Python, Scala and SQL APIs. It also provides popular deep learning frameworks such as TensorFlow, Pytorch, ScikitLearn and Spark MLib.</p><p>It is also integrated with Azure Storage, Azure CosmosDB, EventHub, Azure Data Factory and Azure Active Directory. Azure Databricks makes it simple, productive and collaborative environment for a data scientist, data engineers and data analyst of the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*Qt4s04RQYSIDxyFJ0mXD0g.png" /></figure><p>Image courtesy:<a href="https://azure.microsoft.com/en-us/services/databricks/">https://azure.microsoft.com/en-us/services/databricks/</a></p><p>Azure Databricks integrated with other Azure Data &amp; Analytics services provides a comprehensive solution for collecting data, preparing features, building, training and testing machine learning model. Event hub and ADF can load stream and batch data into azure data storage for scalable and low-cost data storage.</p><p>Azure Databricks can use spark APIs to prepare data and build models. Azure CosmosDB can serve model at scale through web and mobile interface. You can leverage powerful distributed computing for scalable data processing of Spark to build Data Warehouse and Analytics platform, Model Machine Learning and data science Platform and Real-time analytics platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*X3FhTm2k3r5ntz1F48w5oQ.png" /></figure><p>Image courtesy: <a href="https://azure.microsoft.com/en-us/services/databricks/">https://azure.microsoft.com/en-us/services/databricks/</a></p><p><strong>Modern Data Warehouse &amp; Advanced Analytics:</strong> Collect all data types from all sources, process and transform data at scale using Azure Data Factory and Azure Databricks, analyze using Azure SQL DW, Analysis service and Power BI visualization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*fppPf_1zPoG69-GAydkUQg.png" /></figure><p>Image Courtesy: <a href="https://azure.microsoft.com/en-us/services/databricks/">https://azure.microsoft.com/en-us/services/databricks/</a></p><p><strong>Real-time Analytics:</strong> Get insights from streaming data with ease. Capture data continuously from any streaming source, or logs from website clickstreams, and process it in near-real time using Databricks and serve using CosmosDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*DrMjnvMISy_BGXXeVS8N9w.png" /></figure><p>Image Courtesy: <a href="https://azure.microsoft.com/en-us/services/databricks/">https://azure.microsoft.com/en-us/services/databricks/</a></p><p><strong>Azure Workspace:</strong> To use azure Databricks first thing is to create an azure Databricks workspace. Now I am not going to go step by step how to get started with azure Databricks here because there is a great tutorial <a href="https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal">https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal</a> that you can refer to learn how to get here. At this tutorial, I will assume you have an azure subscription (there is always a free trial which gets you $200.00 to get started), azure Databricks workspace and a cluster created. You need to select 5.0 ML Beta Databricks runtime while creating a cluster for ML application. It will have pretty much all the API pre-installed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*CyiVAbmSWtCWeaqupP2Rfw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*TtGDWEIdiOPXwFfkNVorpA.png" /></figure><p><strong>Machine Learning in Azure Databricks:</strong></p><p>Now, we will develop a simple classification model that will classify customer data into categories weather customer will churn or not. For this tutorial, we’ll be using the Orange Telecoms Churn Dataset. It consists of cleaned customer activity data (features), along with a churn label specifying whether the customer cancelled their subscription or not. The data can be fetched from BigML’s S3 bucket, <a href="https://bml-data.s3.amazonaws.com/churn-bigml-80.csv">churn-80</a> and <a href="https://bml-data.s3.amazonaws.com/churn-bigml-20.csv">churn-20</a>. The two sets are from the same batch but have been split by an 80/20 ratio. We’ll use the larger set for training and cross-validation purposes, and the smaller set for final testing and model performance evaluation.</p><p>We have to load the data first to Databricks file store using azure Databricks workspace as seen here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*E5nO0SURgQ60qc7QzunXXw.png" /></figure><p>Once data is loaded open a notebook using python option and start coding as shown here or go straight in the GitHub here <a href="https://github.com/GKarmakar/DatabricksTutorial">https://github.com/GKarmakar/DatabricksTutorial</a>.</p><p>Lets look at the file using Databricks %fs magic command</p><p>%fs ls /FileStore/tables</p><p>Check the data in the file using head command</p><p>%fs head /FileStore/tables/churn.csv</p><p>[Truncated to first 65536 bytes] State,Account_length,Area_code,International_plan,Voice_mail_plan,Number_vmail_messages,Total_day_minutes,Total_day_calls,Total_day_charge,Total_eve_minutes,Total_eve_calls,Total_eve_charge,Total_night_minutes,Total_night_calls,Total_night_charge,Total_intl_minutes,Total_intl_calls,Total_intl_charge,Customer_service_calls,Churn KS,128,415,No,Yes,25,265.1,110,45.07,197.4,99,16.78,244.7,91,11.01,10,3,2.7,1,FALSE OH,107,415,No,Yes,26,161.6,123,27.47,195.5,103,16.62,254.4,103,11.45,13.7,3,3.7,1,FALSE NJ,137,415,No,No,0,243.4,114,41.38,121.2,110,10.3,162.6,104,7.32,12.2,5,3.29,0,FALSE OH,84,408,Yes,No,0,299.4,71,50.9,61.9,88,5.26,196.9,89,8.86,6.6,7,1.78,2,FALSE OK,75,415,Yes,No,0,166.7,113,28.34,148.3,122,12.61,186.9,121,8.41,10.1,3,2.73,3,FALSE AL,118,510,Yes,No,0,223.4,98,37.98,220.6,101,18.75,203.9,118,9.18,6.3,6,1.7,0,FALSE MA,121,510,No,Yes,24,218.2,88,37.09,348.5,108,29.62,212.6,118,9.57,7.5,7,2.03,3,FALSE MO,147,415,Yes,No,0,157,79,26.69,103.1,94,8.76,211.8,96,9.53,7.1,6,1.92,0,FALSE WV,141,415,Yes,Yes,37,258.6,84,43.96,222,111,18.87,326.4,97,14.69,11.2,5,3.02,0,FALSE RI,74,415,No,No,0,187.7,127,31.91,163.4,148,13.89,196,94,8.82,9.1,5,2.46,0,FALSE IA,168,408,No,No,0,128.8,96,21.9,104.9,71,8.92,141.1,128,6.35,11.2,2,3.02,1,FALSE MT,95,510,No,No,0,156.6,88,26.62,247.6,75,21.05,192.3,115,8.65,12.3,5,3.32,3,FALSE IA,62,415,No,No,0,120.7,70,20.52,307.2,76,26.11,203,99,9.14,13.1,6,3.54,4,FALSE ID,85,408,No,Yes,27,196.4,139,33.39,280.9,90,23.88,89.3,75,4.02,13.8,4,3.73,1,FALSE VT,93,510,No,No,0,190.7,114,32.42,218.2,111,18.55,129.6,121,5.83,8.1,3,2.19,3,FALSE VA,76,510,No,Yes,33,189.7,66,32.25,212.8,65,18.09,165.7,108,7.46,10,5,2.7,1,FALSE TX,73,415,No,No,0,224.4,90,38.15,159.5,88,13.56,192.8,74,8.68,13,2,3.51,1,FALSE FL,147,415,No,No,0,155.1,117,26.37,239.7,93,20.37,208.8,133,9.4,10.6,4,2.86,0,FALSE</p><p>A more convenient way to view and process data in databricks using python is to create a dataframe and also create a temporary view.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef42c5c6431fe85365233c817b394aac/href">https://medium.com/media/ef42c5c6431fe85365233c817b394aac/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCjZ2UcUGonszvAS2H5bAQ.png" /></figure><p>It is also a good practice to review data types of the fields in the file and we will see why this was needed later.</p><p>churnDF.printSchema()</p><p>output:</p><p>root | — State: string (nullable = true) | — Account_length: integer (nullable = true) | — Area_code: integer (nullable = true) | — International_plan: string (nullable = true) | — Voice_mail_plan: string (nullable = true) | — Number_vmail_messages: integer (nullable = true) | — Total_day_minutes: double (nullable = true) | — Total_day_calls: integer (nullable = true) | — Total_day_charge: double (nullable = true) | — Total_eve_minutes: double (nullable = true) | — Total_eve_calls: integer (nullable = true) | — Total_eve_charge: double (nullable = true) | — Total_night_minutes: double (nullable = true) | — Total_night_calls: integer (nullable = true) | — Total_night_charge: double (nullable = true) | — Total_intl_minutes: double (nullable = true) | — Total_intl_calls: integer (nullable = true) | — Total_intl_charge: double (nullable = true) | — Customer_service_calls: integer (nullable = true) | — Churn: boolean (nullable = true)</p><p>we see that area code is integer which can be converted to string as bellow:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30afca883249a418e01746513e3bc331/href">https://medium.com/media/30afca883249a418e01746513e3bc331/href</a></iframe><p>now lets check the schema again.</p><p>churnDF.printSchema()</p><p>root | — state: string (nullable = true) | — Account_length: integer (nullable = true) | — Area_code: string (nullable = true) | — International_plan: string (nullable = true) | — Voice_mail_plan: string (nullable = true) | — Number_vmail_messages: integer (nullable = true) | — Total_day_minutes: double (nullable = true) | — Total_day_calls: integer (nullable = true) | — Total_day_charge: double (nullable = true) | — Total_eve_minutes: double (nullable = true) | — Total_eve_calls: integer (nullable = true) | — Total_eve_charge: double (nullable = true) | — Total_night_minutes: double (nullable = true) | — Total_night_calls: integer (nullable = true) | — Total_night_charge: double (nullable = true) | — Total_intl_minutes: double (nullable = true) | — Total_intl_calls: integer (nullable = true) | — Total_intl_charge: double (nullable = true) | — Customer_service_calls: integer (nullable = true) | — Churn: boolean (nullable = true)</p><p>Note that we created a tempview before, lets plot data as a part pf exploratory data analysis and gain insights into the data before we can start building model.</p><p>%sql <br>SELECT Churn, (Total_day_charge+Total_eve_charge+Total_night_charge+Total_intl_charge)<br>FROM ChurnView</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LBaNUK3eUmioc9LbiRiqYg.png" /></figure><p>What we see here is very interesteing customer who will not churn has almost nice normal distribution but customer who will churn has bi-modal distribution and also there is outlier. Its intuitive that customer that will churn are paying higher bills.</p><p>display(churnDF)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3uw5_NCOuNMfnxB4kjT7g.png" /></figure><p><strong>Feature engineering: </strong>In statistical theory features of data fields that are highly correlated aren’t going to be very useful for the modelling input features mapping to output label. They are not going to add any new information hence one of the pair could be dropped from the data, which will not impact model negatively but reduce complexity which is desirable.</p><p>To identify a correlation, we will use sns pairplot function. One more thing in order to understand the correlation we will have to convert spark dataframe into a pandas dataframe. When we convert to pandas dataframe we can’t take advantage of spark cluster instead all the operation on pandas dataframe will be executed in driver node. In that case, it makes sense to sample the data and then perform EDA as shown bellow.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20f14551dd185d4481d5fc61287a496e/href">https://medium.com/media/20f14551dd185d4481d5fc61287a496e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dR0fVTQNDVzBpxooeJqddA.png" /></figure><p>This shows that total minutes column and total charge columns are positively correlated and hence we can get rid one of these columns from each pair.</p><p>Also, we will split data to create training and test set for model training and evaluation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0bd266f05df4714e26e03a719b2acb9/href">https://medium.com/media/d0bd266f05df4714e26e03a719b2acb9/href</a></iframe><p>Next, we will convert Yes/No columns into True/False column for later encoding.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/922263b069bd2586da05261e2920a90a/href">https://medium.com/media/922263b069bd2586da05261e2920a90a/href</a></iframe><p>One last thing before getting into a model is to pre-process data is to apply one-hot encoding for categorical features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2f2f7f45e9e11d54c36bd1773c1ef2c/href">https://medium.com/media/d2f2f7f45e9e11d54c36bd1773c1ef2c/href</a></iframe><p>after that, we will vectorize all the features for feeding into spark ML model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1469c5d4f8bd173b75a1f8e4c4bf079/href">https://medium.com/media/c1469c5d4f8bd173b75a1f8e4c4bf079/href</a></iframe><p>Now we are coming to the fun part where we will create a classifier and test performance. But before we get into modelling, it is often best practice to start with a simpler model first. All other models that we will create later will at least have to beat the baseline model to even qualify for consideration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2eab7f7025f4e775585e58388ccbdd4/href">https://medium.com/media/b2eab7f7025f4e775585e58388ccbdd4/href</a></iframe><p>After executing the code above we get:</p><p>Baseline Training F-score: 0.7865595703874185</p><p>Baseline Test F-score: 0.789413945028712</p><p>Now, lets try a decision tree classifier which is often a better model to try because of its ability to model non-linear relationship in the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0fe9e1ffa0ce0b5adc91c9ac47503d7/href">https://medium.com/media/c0fe9e1ffa0ce0b5adc91c9ac47503d7/href</a></iframe><p>We get,</p><p>Decision Tree Training F-score: 0.9506130577975226</p><p>Decision Tree Test F-score: 0.9218495344352897</p><p>This is better than our baseline model and we can see that we may have other models that we can try that are more complex and may prone to overfit.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ad3959e0351014387f8f9358a0ac323/href">https://medium.com/media/3ad3959e0351014387f8f9358a0ac323/href</a></iframe><p>Logistic Regression Training F-score: 0.8345316824370106 Logistic Regression Test F-score: 0.8377146377762487</p><p>Logistic regression doesn’t do better than decision tree which is understandable as logistic regression does better when data is linearly dependent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8136147d8572b92041b2915ece2283e/href">https://medium.com/media/d8136147d8572b92041b2915ece2283e/href</a></iframe><p>Gradient Boosted Tree Training F-score: 0.9802317325509573</p><p>Gradient Boosted Tree Test F-score: 0.8377146377762487</p><p>Here is one thing to be noticed that GBT did well on training data, but not on test most likely overfitting. Here it comes the magic of hyperparameter tuning which we are not going to get into this tutorial. But just to note here that there are techniques to fight to overfit using regularization techniques.</p><p>One of my favourite models is a random forest. Random forest uses many decision tree models and often perform better.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79b2bb1f166f0852f266c1937bf84ce3/href">https://medium.com/media/79b2bb1f166f0852f266c1937bf84ce3/href</a></iframe><p>We can see that the best performing model is still GBT and we can save the model for future use.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b580a90e0af71a63559221cf76a9916d/href">https://medium.com/media/b580a90e0af71a63559221cf76a9916d/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63e3a55ed815" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Machine Learning Solutions — Building end to end scalable ML Model and deploy anywhere —…]]></title>
            <link>https://medium.com/@gautam.karmakar/azure-machine-learning-solutions-building-end-to-end-scalable-ml-model-and-deploy-anywhere-950f92a9d190?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/950f92a9d190</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cloud-machine-learning]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Sat, 24 Nov 2018 18:50:45 GMT</pubDate>
            <atom:updated>2018-11-26T01:25:22.229Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Azure Machine Learning Solutions — Build &amp; Deploy Scalable Models — Cloud, Edge or Hybrid.</strong></h3><p><strong>What is Azure Machine Learning?</strong></p><p>It’s a cloud service offered in Azure to build, train, deploy and monitor machine learning models that integrate with border azure services.</p><p><strong>What is machine learning</strong>?</p><p>Machine learning is a technique of data science where computers can learn from data without being explicitly programmed to predict future behaviour, trends, pattern and outcomes. Some examples are detecting spam emails, classifying images into multiple groups, forecast weather of tomorrow or next week. For example, if a machine sees a similarity of characteristics between a current transaction with previously recorded as fraud transaction, it can generate an alert and block the transaction flagged it as a potential for fraud. It also probabilistic rather deterministic. For example, saying that the transaction is either appropriate of fraud with complete certainty it will say that there is 85.5% chance that transaction is fraud.</p><p><strong>Azure Machine Learning — A Closer look:</strong></p><p>Azure Machine Learning provides a suite of tools and technology that enables a developer to build, train, evaluate, deploy and manage a machine learning model art scale. It supports most popular developer’s tools that are easy to use, flexible and integrated. For example, as an IDE for developers, it offers Jupyter Notebook (OSS) and Visual Studio tools for AI. This makes it easy to explore and visualize data, clean and transform, engineer features, build and test models and deploy in a scalable infrastructure. It lets you experiment to your local machine and then deploy when you are ready for a large-scale computer infrastructure of your choice.</p><p>It also comes with full-scale Azure Machine Learning SDK for python, one of the most popular languages in machine learning today. It comes with the most popular python packages which you will need to build machine learning models. There are many open source packages available such as</p><p>· Scikit-learn</p><p>· Tensorflow</p><p>· PyTorch</p><p>· CNTK</p><p>· MXNet</p><p>You will get a native support of Azure Batch AI to compute infrastructure along with advanced hyperparameter tuning services. You can deploy them using container image such as docker. It means you can either use azure container services (ACS) or Azure Kubernetes Services (AKS).</p><p><strong>How it is different from Azure ML studio?</strong></p><p>Azure ML studio is a no code approach to build, train and deploy the model. You can get quickly started with Azure ML studio if you are starting fresh and need to build and deploy model quickly and that doesn’t need a lot of skills of machine learning. This is a drag and drop approach and you can get most of the popular models such as classification and regression-type problems. You can also quickly deploy the model using web service which will use REST API for you. But if you want the most flexible environment where you want to control development and deployment options you need to use Azure machine learning for richer model development experience.</p><p>Azure machine learning model also comes with the feature called automated machine learning where a model is built for you that you can train with your data, test and deploy like any other model. These are sets of model collections that are very rich, powerful and highly optimized. Automated machine learning also comes with a suite of features for data processing that are very common in machine learning applications.</p><p>You can use these tools through interactive IDE such as Jupyter notebook, Visual Studio Tool for AI or use data prep SDK in python as part of border Azure Machine Learning SDK package. Currently, data prep SDK supports <strong>impute missing value, fuzzy group auto join, automated file type detection, varying schema processing</strong> and others. These tools can be used as a cross-platform manner in Windows, Mac, Linux and Spark as a scale-out manner for high performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*CpHhvF8X6QVKhXHP8HZ1YQ.png" /></figure><p>Azure automated machine learning and tools are constantly getting updated with richer sets of features so please keep an eye for new additions.</p><p><strong>Experiment:</strong></p><p>Now, we will use Jupyter notebook to build, train, deploy a machine learning model using azure machine learning. In this tutorial, we will use python inside the Jupyter notebook for everything. But environment setting can also be done using a portal. All we need is a subscription and choice of region. I recommend you using region close to where you are running application.</p><p><strong>Install SDK</strong></p><p>Before you install SDK, it is better if you create an isolated python or Conda environment if you have anaconda installed. I also want to mention that everything I ran on Mac, not on windows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/706b3afb2d09f42fa1c61e219fcf7543/href">https://medium.com/media/706b3afb2d09f42fa1c61e219fcf7543/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/133e2913ede59cba67872669abea3b97/href">https://medium.com/media/133e2913ede59cba67872669abea3b97/href</a></iframe><p>Now, using Jupyter notebook we will complete this experiment of image classification problem. You may be already familiar with the MNIST image classification problem. MNIST digit recognition problem is an entry point to computer vision originally created by Yann Lecun in the 80s. But we will use the slightly difficult problem of Fashion MNIST dataset. You can read about the dataset here <a href="https://research.zalando.com/welcome/mission/research-projects/fashion-mnist/">https://research.zalando.com/welcome/mission/research-projects/fashion-mnist/</a></p><p>Everything written here onwards will also be in the GitHub so, you can directly jump to GitHub <a href="https://github.com/GKarmakar/Azure-Machine-Learning">https://github.com/GKarmakar/Azure-Machine-Learning</a> or continue read here and then check code in the GitHub.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2279f5ab7c28229dbf43cb20aa407a1f/href">https://medium.com/media/2279f5ab7c28229dbf43cb20aa407a1f/href</a></iframe><p>Make sure you get a valid response to above command. Now before we can run an experiment using azure ml, we need to create a workspace which is like a workstation where you create an experiment, register models and run experiment.</p><p>The command to create workspace is</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a604760a474b8631dacb1231714e386e/href">https://medium.com/media/a604760a474b8631dacb1231714e386e/href</a></iframe><p>Note that you need to substitute your own Azure subscription ID and a name of the resource group is your choice too. You can use an existing resource group just set <strong>create_resource_group = False</strong>. Regarding the region, you can choose the region close to your location.</p><p>To use this workspace you need to save a configuration file, that you can upload later for any experiment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0291c055cb9d1abc788df2c5b76b109/href">https://medium.com/media/a0291c055cb9d1abc788df2c5b76b109/href</a></iframe><h4>Create a Folder to Host All Sample Projects</h4><p>Finally, create a folder where all the sample projects will be hosted.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79ecc83ffa29e547471b1762565090fb/href">https://medium.com/media/79ecc83ffa29e547471b1762565090fb/href</a></iframe><p>Import the Fashion MNIST dataset This guide uses the Fashion MNIST dataset which contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels), as seen here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*GNLB2jtcfb_xTqgQd9ntJA.png" /></figure><p>Fashion MNIST is intended as a drop-in replacement for the classic MNIST dataset — often used as the “Hello, World” of machine learning programs for computer vision. The MNIST dataset contains images of handwritten digits (0, 1, 2, etc) in an identical format to the articles of clothing we’ll use here.</p><p>This guide uses Fashion MNIST for variety, and because it’s a slightly more challenging problem than regular MNIST. Both datasets are relatively small and are used to verify that an algorithm works as expected. They’re good starting points to test and debug code.</p><p>We will use 60,000 images to train the network and 10,000 images to evaluate how accurately the network learned to classify images. You can access the Fashion MNIST directly from TensorFlow, just import and load the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1aef131a4f8c95dabaede57db2c83d7/href">https://medium.com/media/e1aef131a4f8c95dabaede57db2c83d7/href</a></iframe><p>Let’s display some image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*gWUoemcOWkjXF5Ew2z2IFg.png" /></figure><p>Some more images with their labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*dvf7ISOGFaMdb4pAd_UeWw.png" /></figure><p>Now getting into the main part of the model, lets go.</p><p>Here we will configure an Azure Automated ML API</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/254e68c9bf97a1986bcc4c83334b9ae8/href">https://medium.com/media/254e68c9bf97a1986bcc4c83334b9ae8/href</a></iframe><p>See that training data is passed using X and y variable. Also, I used the blacklist parameter to ask Automated ML API to exclude knn and LinearSVM. This is optional and specific to this problem where I think knn will be a lot slower and LinearSVM won’t be able to understand the complexity of the data but you can choose whatever relevant for your dataset.</p><p>After the configuration is done, we will simply train the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eca18a7ce173cd675021a2807c1a9798/href">https://medium.com/media/eca18a7ce173cd675021a2807c1a9798/href</a></iframe><p>We should see the following output.</p><pre>Parent Run ID: AutoML_30a6dc57-5129-4343-84ab-03787fdbe011<br>***********************************************************************************************<br>ITERATION: The iteration being evaluated.<br>PIPELINE: A summary description of the pipeline being evaluated.<br>DURATION: Time taken for the current iteration.<br>METRIC: The result of computing score on the fitted pipeline.<br>BEST: The best observed score thus far.<br>***********************************************************************************************<br><br> ITERATION     PIPELINE                               DURATION                METRIC      BEST<br>         0      MaxAbsScaler LightGBMClassifier       0:01:41.595400           0.985     0.985<br>Stopping criteria reached at iteration 1. Ending experiment.</pre><p>One other important thing to notice that I have specified the desired accuracy as well as measuring metrics as 97% and AUC_weighted. I think these are a cool and useful feature of automated ML.</p><p>Now, we can retrieve the best model from the experiment and predict using the best model. This will also register the best model into your workspace under the experiment that we created above to reuse later or deploy in production.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7118b408daf17eb23f4d8013ae9060e7/href">https://medium.com/media/7118b408daf17eb23f4d8013ae9060e7/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc132eea088ebf253e477e7543b366d1/href">https://medium.com/media/cc132eea088ebf253e477e7543b366d1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*52sF0ZG_8YZK2ARAJctDPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*r4IjzXofpI-4ZhfB6NWv3Q.png" /></figure><p>Use Jupyter extension to display graphs and tables of all results. You will also get a URL for azure portal to explore same output.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d43cb9753df9b391923e9fd74b630a31/href">https://medium.com/media/d43cb9753df9b391923e9fd74b630a31/href</a></iframe><p><strong>Retrieve the model for run and deploy remotely:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b9f2eb0e3d5f93307327cbf69fd6dfdc/href">https://medium.com/media/b9f2eb0e3d5f93307327cbf69fd6dfdc/href</a></iframe><h3>Deploy as web service</h3><p>Once you’ve tested the model and are satisfied with the results, deploy the model as a web service hosted in ACI.</p><p>To build the correct environment for ACI, provide the following:</p><ul><li>A scoring script to show how to use the model</li><li>An environment file to show what packages need to be installed</li><li>A configuration file to build the ACI</li><li>The model you trained before</li></ul><h3>Create scoring script</h3><p>Create the scoring script, called score.py, used by the web service call to show how to use the model.</p><p>You must include two required functions into the scoring script:</p><ul><li>The init() function, which typically loads the model into a global object. This function is run only once when the Docker container is started.</li><li>The run(input_data) function uses the model to predict a value based on the input data. Inputs and outputs to the run typically use JSON for serialization and de-serialization, but other formats are supported.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a54563fa045ec8d0c91ef8ede3bdb00/href">https://medium.com/media/5a54563fa045ec8d0c91ef8ede3bdb00/href</a></iframe><h3>Create environment file</h3><p>Next, create an environment file, called myenv.yml, that specifies all of the script’s package dependencies. This file is used to ensure that all of those dependencies are installed in the Docker image. This model needs scikit-learn and azureml-sdk.</p><h3>Create configuration file</h3><p>Create a deployment configuration file and specify the number of CPUs and gigabyte of RAM needed for your ACI container. While it depends on your model, the default of 1 core and 1 gigabyte of RAM is usually sufficient for many models. If you feel you need more later, you would have to recreate the image and redeploy the service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86a9867f71d1a3f15c8380d6c3bb3f30/href">https://medium.com/media/86a9867f71d1a3f15c8380d6c3bb3f30/href</a></iframe><h3>Test deployed service</h3><p>Earlier you scored all the test data with the local version of the model. Now, you can test the deployed model with a random sample of 30 images from the test data.</p><p>The following code goes through these steps:</p><ol><li>Send the data as a JSON array to the web service hosted in ACI.</li><li>Use the SDK’s run API to invoke the service. You can also make raw calls using any HTTP tool such as curl.</li><li>Print the returned predictions and plot them along with the input images. Red font and inverse image (white on black) is used to highlight the misclassified samples.</li><li>Since the model accuracy is high, you might have to run the following code a few times before you can see a misclassified sample.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/911a0f7fb80b4b57b1dbff18cdcd74b5/href">https://medium.com/media/911a0f7fb80b4b57b1dbff18cdcd74b5/href</a></iframe><p>You can also send raw HTTP request to test the web service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03fdb5d9f7455e93692aee274078c2d9/href">https://medium.com/media/03fdb5d9f7455e93692aee274078c2d9/href</a></iframe><p><strong>Conclusion:</strong></p><p>In this tutorial, we have done.</p><ul><li>Set up your experimentation and environment</li><li>Create and train a automated ML model for classification</li><li>Retrieve the model from your workspace</li><li>Test the model locally</li><li>Deploy the model to ACI</li><li>Test the deployed model</li></ul><p>There are many more that you can learn from documentation (preview) here <a href="https://docs.microsoft.com/en-us/azure/machine-learning/service/">https://docs.microsoft.com/en-us/azure/machine-learning/service/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=950f92a9d190" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tensorflow Estimator Premade models for scikit learn Fans]]></title>
            <link>https://medium.com/@gautam.karmakar/tensorflow-estimator-premade-models-for-scikit-learn-fans-59abb5149b3f?source=rss-3f842eaeb840------2</link>
            <guid isPermaLink="false">https://medium.com/p/59abb5149b3f</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gautam Karmakar]]></dc:creator>
            <pubDate>Sat, 29 Sep 2018 20:52:20 GMT</pubDate>
            <atom:updated>2018-09-29T20:52:20.406Z</atom:updated>
            <content:encoded><![CDATA[<p>Tensorflow Estimator Premade models for scikit learn Fans</p><p>The intention of this tutorial to show how to use tensorflow estimator pre-baked model for folks that are</p><p>used to use scikit learn models. This tutorial will show step by step how to use tensorflow estimator premade model similarly we use scikit learn.</p><p>In case you are not familiar with scikit learn here is few lines of code how scikit learn can be used to train and predict using a classifier.</p><pre>from sklearn import datasets</pre><pre>iris = datasets.load_iris()</pre><pre>from sklearn import svm</pre><pre>clf = svm.SVC(gamma = ‘scale’) #You can tune these any other parameters</pre><pre>clf.fit(iris.data,iris.labels</pre><pre>import pickle</pre><pre>s = pickle.dumps(clf)</pre><pre>clf2 = pickle.loads(s)</pre><pre>predicted = clf2.predict(iris.data[0:1])</pre><pre>print(“Predicted {} and expected {}”.format(predicted, iris.labels[0]))</pre><p>for more on scikitlearn: <a href="http://scikit-learn.org/stable/tutorial/basic/tutorial.html">http://scikit-learn.org/stable/tutorial/basic/tutorial.html</a></p><p>Prior to using this code please install required software — tensorflow and pandas. If you are using anaconda you may already have pandas installed.</p><pre>!pip install tensorflow</pre><pre>!pip install pandas</pre><pre>import tensorflow as tf</pre><pre>print(tf.__version__)</pre><p>Before getting into tensorflow estimator let’s understand tensorflow estimator API and underlying components that estimator abstracts from. Estimator API was added in tensorflow in release 1.1.</p><p>It is built on top of tensorflow core components as seen in the figure below. The most important about</p><p>an estimator is to even though it is a high-level representation which dramatically reduces the number of lines in the code it also provides all the strengths of tensorflow such as distributed training, deploys and serving features.</p><p>Keras, on the other hand, is also another API that runs on top of core tensorflow and tightly integrated. Keras makes it easy to use deep learning models using layers, activation and optimizers. Keras also provides callbacks and checkpoints, save and load models. Keras is available as tf.keras which means you are using keras high-level APIs on top of tensofow.</p><p>Now, as estimator and keras both provide high level API that reduces amount of coding needed if tensorflow was directly used and even make developers life easier there is a choice to make to use either one of them. Estimator can be used to run distributed training on GPU, TPU or CPU. It can be used in on-premise or in cloud infrastructure in mutiple servers. For example, you can use estimator to distribute and scale your training in Google Cloud ML engine which will scael automatically and economically. Also, estimator integrates with tensorboard which provides browser based visualization of model performance metrics. Another advantage of estimator is that it also comes with premade models likes of scikit learn libraries where there is already a prebuilt model that you can just use and only focus on tuning hyperparameters. We all know how cool scikit learn is, so here we go same using tensorflow. Using keras you get all the building blocks but you still need to build the model, like add conv layers, pooling layers, batch normalization, dropout, relu etc. One cool thing though that model written using keras can be converted into estimator model using simple one line code by tf.keras.estimator.model_to_estimator.</p><p>Model:</p><p>This simple classifier will classify flowers based on their sepals and petals features using IRIS dataset.</p><p>The dataset is labelled so we would be able to use a supervised model to classify. We will use features like</p><p>· sepal length</p><p>· sepal width</p><p>· petal length</p><p>· petal width</p><p>There are three categories of flowers: Iris sertosa[0], iris versicolor[1], iris virginica[2] The features are represented as float32 numeric and categories are int32 numeric.</p><pre>import iris_data</pre><pre>(X_train, y_train),  (X_test, y_test) = iris_data.load_data()</pre><pre>#Lets create features dictionary</pre><pre>features = []</pre><pre>for key in X_train.keys():<br>    features.append(tf.feature_column.numeric_column(key=key))</pre><pre>#Now we will create an input function to feed into estimator model.</pre><pre>def train_input_fn(features, labels, batch_size):</pre><pre>dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))</pre><pre>#shuffle and repeat the examples</pre><pre>dataset = dataset.shuffle(1000).repeat().batch(batch_size)</pre><pre>return dataset</pre><pre>#Simlarly we will define evaluation and prediction input functions.</pre><pre>def eval_input_fn(features, labels, batch_size):</pre><pre>features=dict(features)</pre><pre>inputs = (features, labels)</pre><pre>dataset = tf.data.Dataset.from_tensor_slices(inputs)</pre><pre>dataset = dataset.batch(batch_size)</pre><pre>return dataset</pre><p>Now we are ready to instantiate an estimator premade model.</p><p># Build a DNN with 2 hidden layers and 10 nodes in each hidden layer.</p><pre>classifier = tf.estimator.DNNClassifier(</pre><pre>feature_columns=features,</pre><pre># Two hidden layers of 10 nodes each.</pre><pre>hidden_units=[10, 10],</pre><pre># The model must choose between 3 classes.</pre><pre>n_classes=3)</pre><p>We will train this model using iris dataset and features that we declared.</p><pre>#Train the model</pre><pre>classifier.train(</pre><pre>input_fn=lambda:train_input_fn(X_train, y_train, 32),</pre><pre>steps=100)</pre><pre>#Let’s evaluate this model to check performance on eval data.</pre><pre># Evaluate the model.</pre><pre>eval_result = classifier.evaluate(</pre><pre>input_fn=lambda:eval_input_fn(X_test, y_test, 32))</pre><pre>print(&#39;\nTest set accuracy: {accuracy:0.3f}\n&#39;.format(**eval_result))</pre><p>Here is my output: accuracy 0.967</p><pre>INFO:tensorflow:Calling model_fn.</pre><pre>INFO:tensorflow:Done calling model_fn.</pre><pre>INFO:tensorflow:Starting evaluation at 2018-09-29-20:22:24</pre><pre>INFO:tensorflow:Graph was finalized.</pre><pre>INFO:tensorflow:Restoring parameters from /var/folders/9j/wnpgv9rd1852fcj3qqttm7_40000gn/T/tmp_kkjel5n/model.ckpt-100</pre><pre>INFO:tensorflow:Running local_init_op.</pre><pre>INFO:tensorflow:Done running local_init_op.</pre><pre>INFO:tensorflow:Finished evaluation at 2018-09-29-20:22:24</pre><pre>INFO:tensorflow:Saving dict for global step 100: accuracy = 0.96666664, average_loss = 0.14495544, global_step = 100, loss = 4.3486633</pre><pre>INFO:tensorflow:Saving &#39;checkpoint_path&#39; summary for global step 100: /var/folders/9j/wnpgv9rd1852fcj3qqttm7_40000gn/T/tmp_kkjel5n/model.ckpt-100</pre><pre>Test set accuracy: 0.967</pre><p>We can manufacture some data as our training example and predict on it to see how our model is performing on previously unseen data — A must for model testing.</p><pre># Generate predictions from the model</pre><pre>expected = [&#39;Setosa&#39;, &#39;Versicolor&#39;, &#39;Virginica&#39;]</pre><pre>predict_x = {</pre><pre>&#39;SepalLength&#39;: [5.1, 5.9, 6.9],</pre><pre>&#39;SepalWidth&#39;: [3.3, 3.0, 3.1],</pre><pre>&#39;PetalLength&#39;: [1.7, 4.2, 5.4],</pre><pre>&#39;PetalWidth&#39;: [0.5, 1.5, 2.1],</pre><pre>}</pre><pre>predictions = classifier.predict(</pre><pre>input_fn=lambda:iris_data.eval_input_fn(predict_x,</pre><pre>labels=None,</pre><pre>batch_size=32))</pre><p>Our prediction module returns a generator to see the prediction we need call few times.</p><pre>template = (&#39;\nPrediction is &quot;{}&quot; ({:.1f}%), expected &quot;{}&quot;&#39;)</pre><pre>SPECIES = [&#39;Setosa&#39;, &#39;Versicolor&#39;, &#39;Virginica&#39;]</pre><pre>for pred_dict, expec in zip(predictions, expected):</pre><pre>class_id = pred_dict[&#39;class_ids&#39;][0]</pre><pre>probability = pred_dict[&#39;probabilities&#39;][class_id]</pre><pre>print(template.format(SPECIES[class_id],</pre><pre>100 * probability, expec))</pre><p>Here is my output:</p><pre>INFO:tensorflow:Calling model_fn.</pre><pre>INFO:tensorflow:Done calling model_fn.</pre><pre>INFO:tensorflow:Graph was finalized.</pre><pre>INFO:tensorflow:Restoring parameters from /var/folders/9j/wnpgv9rd1852fcj3qqttm7_40000gn/T/tmp_kkjel5n/model.ckpt-100</pre><pre>INFO:tensorflow:Running local_init_op.</pre><pre>INFO:tensorflow:Done running local_init_op.</pre><pre>Prediction is &quot;Setosa&quot; (99.0%), expected &quot;Setosa&quot;</pre><pre>Prediction is &quot;Versicolor&quot; (87.2%), expected &quot;Versicolor&quot;</pre><pre>Prediction is &quot;Virginica&quot; (81.5%), expected &quot;Virginica&quot;</pre><pre>This post was created based on content present here <a href="https://www.tensorflow.org/guide/premade_estimators">https://www.tensorflow.org/guide/premade_estimators</a></pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59abb5149b3f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>