<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yousef Alkhanafseh on Medium]]></title>
        <description><![CDATA[Stories by Yousef Alkhanafseh on Medium]]></description>
        <link>https://medium.com/@alkhanafseh?source=rss-b06f7ab6709b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kFnfjS-wdwGaKlwwxH1TtA.jpeg</url>
            <title>Stories by Yousef Alkhanafseh on Medium</title>
            <link>https://medium.com/@alkhanafseh?source=rss-b06f7ab6709b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:47:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alkhanafseh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stargate AI: The World’s Largest Artificial Intelligence Project]]></title>
            <link>https://medium.com/turknettech/stargate-ai-the-worlds-largest-artificial-intelligence-project-bf604f960d00?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf604f960d00</guid>
            <category><![CDATA[stargate]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 10:22:53 GMT</pubDate>
            <atom:updated>2025-06-30T10:22:53.760Z</atom:updated>
            <content:encoded><![CDATA[<p>Discover how OpenAI, SoftBank, Oracle, and the UAE’s MGX plan to build Stargate AI — a $500 billion network of super‑fast data centers packed with millions of Nvidia chips.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hd4kskliz5_Owo25ajOltQ.png" /><figcaption>Image generated by Dall-E3</figcaption></figure><h3>INTRODUCTION</h3><p>Artificial Intelligence (AI) tools like ChatGPT, image creators, and voice assistants need a huge amount of computer power to work well. As more people use these tools every day, companies must build bigger and faster data centers to keep up. Four big names — OpenAI, SoftBank, Oracle, and MGX from the United Arab Emirates — have joined forces to create Stargate AI project. Think of it as the world’s largest computer factory. They plan to spend $500 billion <strong>[1]</strong> over the next four years to put up 20 mega campuses filled with powerful Nvidia chips. The very first site broke ground in Abilene, Texas in January 2025. Stargate’s goal is simple which can be summerized into giving researchers, businesses, and future AI systems all the speed they need — without running out of space or slowing down. It could open the door to the next wave of AI breakthroughs as well.</p><h3>STARGATE PROJECT GENERAL INFORMATION</h3><ul><li><strong>Lunch date and first location<br></strong>21 Jan 2025 at USA Texax, Abilene</li><li><strong>Partners<br></strong>OpenAI, SoftBank Group, Oracle and Abu-Dhabi MGX.</li><li><strong>Board seats<br></strong>Sam Altman (OpenAI), Masayoshi Son (SoftBank), Safra Catz (Oracle), and Jassim Bin Hamad (MGX).</li><li><strong>Partners Roles<br></strong>SoftBank majority equity; OpenAI holds “golden share” &amp; operational control; Oracle provides cloud software/hardware stack; MGX contributes sovereign wealth financing <strong>[2]</strong>.</li><li><strong>Cost<br></strong>$500 Bilion over four years (until 2029)</li><li><strong>Target footprint<br></strong>20 campuses (5–10 GW aggregate), each &gt; 1 GW, housing ≈ 15 million GPUs if fully built-out</li><li><strong>Technology<br></strong>Nvidia Blackwell &amp; Grace Hopper accelerators (liquid-cooled pods) linked through Spectrum‑X.</li></ul><h3>WHY IT MATTERS</h3><p>If it ships on time, Stargate will give researchers a giant playground for AGI work: models with trillions of parameters, real‑time robotics inference, huge scale simulations, and far more capable small language models that can run on your laptop.</p><h3>CURRENT CHALANGES</h3><p>Stargate still has big difficulties to clear. Each campus will drink up huge amounts of electricity and cooling water, so it must lock in steady, green power and reliable water rights first. Each site must also get local permits and pass environmental checks, which can take months. Skilled workers, chip designers, high‑voltage electricians, and AI engineers are already scarce, making hiring tough. Add the need to buy special parts on time, guard the network against hackers, and raise billions in funding without costs exploding <strong>[3]</strong>.</p><h3>THE WIDER RACE</h3><p>Microsoft ($80 B) <strong>[4]</strong>, Amazon ($20 Billion) <strong>[5]</strong>, Meta ($65 B) and several hyperscalers are all carving their own GPU mega farms spending tens of billions of dollars <strong>[6]</strong>. Whether any of these visions materialise on schedule will depend less on chip supply and more on who can secure the power, water, and high‑voltage transmission first.</p><h3>CONCLUSION</h3><p>Stargate AI is more than one mega‑data‑center project; it is the front door to the next industrial age. Just as railroads opened continents, airplanes shrank oceans, and silicon chips powered the digital boom, giant AI compute grids will shape every field that follows — medicine, energy, finance, entertainment, even the way we design chips themselves.</p><p>The global race is already under way. Microsoft, Amazon, Meta, Chinese hyperscalers, and now Stargate are spending hundreds of billions for hardware, power lines, and clean‑energy deals. In short, the gate is open, but it will not stay open forever. Whether you are a data center builder, a university researcher, or an investor, the chance to claim a seat in the AI era is here right now. Those who move first will decide where the future takes root — and how it unfolds. Those who wait may find the race finished before they ever leave the starting line.</p><h3>WORTH WATCHING</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FGhIJs4zbH0o%3Fstart%3D467%26feature%3Doembed%26start%3D467&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DGhIJs4zbH0o&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FGhIJs4zbH0o%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0be109e13f926f0c06140d25544cb78/href">https://medium.com/media/c0be109e13f926f0c06140d25544cb78/href</a></iframe><h3>REFERENCES</h3><p><strong>[1] </strong>OpenAI (2025).<strong> </strong>Announcing The Stargate Project. Accessed on [27.06.2025]. Retrieved from: <a href="https://openai.com/index/announcing-the-stargate-project/">https://openai.com/index/announcing-the-stargate-project/</a></p><p><strong>[2]</strong> Streets, M. (2025). Stargate AI explained: What’s in the $500 billion project. Accessed on [27.06.2025]. Retrieved from: <a href="https://www.techtarget.com/whatis/feature/Stargate-AI-explained-Whats-in-the-project">https://www.techtarget.com/whatis/feature/Stargate-AI-explained-Whats-in-the-project</a></p><p><strong>[3]</strong> Tomás, J. P. (2025). What is the Stargate AI project?. Accessed on [27.06.2025]. Retrieved from: <a href="https://www.rcrwireless.com/20250508/fundamentals/what-stargate-ai-project">https://www.rcrwireless.com/20250508/fundamentals/what-stargate-ai-project</a></p><p><strong>[4]</strong> Reuters (2025). Microsoft plans to invest $80 billion on AI-enabled data centers in fiscal 2025. Accessed on [27.06.2025]. Retrieved from: <a href="https://www.reuters.com/technology/artificial-intelligence/microsoft-plans-spend-80-bln-ai-enabled-data-centers-fiscal-2025-cnbc-reports-2025-01-03/?utm_source=chatgpt.com">https://www.reuters.com/technology/artificial-intelligence/microsoft-plans-spend-80-bln-ai-enabled-data-centers-fiscal-2025-cnbc-reports-2025-01-03/?utm_source=chatgpt.com</a></p><p><strong>[5]</strong> PYMNTS (2025). Amazon Announces $20 Billion Cloud/AI Project in Pennsylvania. Accessed on [27.06.2025]. Retrieved from: <a href="https://www.pymnts.com/amazon/2025/amazon-announces-20-billion-cloud-ai-project-in-pennsylvania/">https://www.pymnts.com/amazon/2025/amazon-announces-20-billion-cloud-ai-project-in-pennsylvania/</a></p><p><strong>[6]</strong> Perrigo B. (2025). Inside Amazon’s Race to Build the AI Industry’s Biggest Datacenters. Accessed on [27.06.2025]. Retrieved from: <a href="https://time.com/7273288/amazon-anthropic-openai-microsoft-stargate-datacenters/?utm_source=chatgpt.com">https://time.com/7273288/amazon-anthropic-openai-microsoft-stargate-datacenters/?utm_source=chatgpt.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf604f960d00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/stargate-ai-the-worlds-largest-artificial-intelligence-project-bf604f960d00">Stargate AI: The World’s Largest Artificial Intelligence Project</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Small Language Model for Sentiment Classification in ISP Chats]]></title>
            <link>https://medium.com/turknettech/small-language-model-for-sentiment-classification-in-isp-chats-ab058fb8435e?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab058fb8435e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[isp]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 10:22:23 GMT</pubDate>
            <atom:updated>2025-06-30T10:22:23.398Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial demonstrates how to use the Llama 3 model to classify ISP customer chat conversations by assigning an optimism score ranging from 0 to 100, where 100 represents extremely optimistic sentiment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*naQTZOWfrQ0cc2gsBBeJgg.png" /><figcaption>Image generated by Dall-E3</figcaption></figure><h3><strong>INTRODUCTION</strong></h3><p>Internet Service Providers (ISPs) manage thousands of live chat interactions daily, covering a wide range of topics such as customer complaints, billing inquiries, and technical support issues. Automatically identifying and flagging positive, optimistic conversations within these interactions can provide valuable insights. Such detection helps customer care teams recognize success stories, highlight positive customer feedback for agent motivation, and support downstream applications like churn prediction modeling.</p><p>While Large Language Models (LLMs) have proven highly effective for this classification task, they typically require extensive computational resources, including multiple gigabytes of VRAM and server-grade GPUs. In this study, we demonstrate an efficient alternative by deploying Small Langugae Model (SLM) of Llama 3-Instruct (8B variant) using Ollama, a lightweight, one-command deployment framework. This setup allows the model to run locally on a friendly enviroment.</p><h3>ECOSYSTEM</h3><p>This section summarizes the hardware specifications, operating system, and software versions used in the study, ensuring reproducibility and supporting future research efforts.</p><h4><strong>Operating System (OS)</strong></h4><p>Type<strong>:</strong> Windows Subsystem for Linux (WSL) Version 2, running Ubuntu 22.04.2 LTS</p><h4>Hardware</h4><ul><li><strong>CPU<br></strong>MaxClockSpeed: 2300<br>Name: 12th Gen Intel(R) Core(TM) i7–12650H<br>NumberOfCores: 10<br>NumberOfLogicalProcessors: 16</li><li><strong>RAM</strong><br>TotalPhysicalMemory: 16,087 MB, DDR5</li><li><strong>Disk</strong><br>Model: NVMe SAMSUNG MZVL41T0HBLB-00BTW<br>Size: 1 TB</li><li><strong>GPU <br></strong>Model: Nvidia RTX 4070 GPU (8 GB VRAM, 85W)</li></ul><h4>Software</h4><p>Programming Langugae: Python 3.11.4</p><p>SLM Deployment Tool: Omni-Layer Learning Language Acquisition Model (OLLAMA)</p><p>SLM Model: Large Language Model Meta AI 3 (LLaMA 3:8b), refer to [1] for more details about the model</p><h3><strong>ENVIROMENT SETUP</strong></h3><ol><li>Install Python, refer to <strong>[2]</strong>.</li><li>Prepare your GPU enviroment, refer to <strong>[3]</strong> GPU Driver(s) and Cuda Toolkit subsections under III. SOFTWARE INSTALLATIONS section.</li><li>Install OLLAMA</li></ol><p>Install Ollama for Ubuntu:</p><pre>curl -fsSL https://ollama.com/install.sh | sh</pre><p>Run the following command for only one time so that llama3 model is installed and started:</p><pre>ollama run llama3</pre><p>Then, close it:</p><pre>ctrl + d</pre><p>To see the installed models use ollama list and for currently ruuning models use ollama ps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*W5s7TNTezb0zOjjHRw7i0w.png" /></figure><p>Each time, please check if OLLAMA API is running in the background using sudo systemctl status ollama.service, for starting it use sudo systemctl start ollama.service, and for stopping it use sudo systemctl stop ollama.service.</p><p>The local HTTP API endpoint where your OLLAMA server is listening and waiting for requests is <a href="http://localhost:11434/">http://localhost:11434/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*iSc41EPI_0RHJSQVdOl3cQ.png" /></figure><h3><strong>RUNNING THE PIPELINE</strong></h3><p>In this study, we utilize the publicly available <em>Talkmap Telecom Conversation Corpus</em> (talkmap/telecom-conversation-corpus), which contains 3,726,699 rows. The dataset includes four key features: conversation_id (a unique identifier for each conversation), speaker (indicating the role of the participant, e.g., agent or customer), date_time (timestamp of each utterance), and text (the actual conversation content).</p><p>To use this dataset, first install the datasets library by running the following command:</p><pre>python3 -m pip install datasets</pre><p>Then, use the following Python code to access it as a Pandas DataFrame:</p><pre># Import required libraries<br>from datasets import load_dataset<br>import pandas as pd<br><br># Set pandas option to display all columns and rows when printing DataFrames<br>pd.set_option(&#39;display.max_columns&#39;, None)<br>pd.set_option(&#39;display.max_rows&#39;, None)<br><br># Load the &#39;talkmap/telecom-conversation-corpus&#39; dataset from Hugging Face Datasets Hub<br># Only load the &#39;train&#39; split<br># Convert the loaded dataset to a pandas DataFrame<br>dataset = load_dataset(&quot;talkmap/telecom-conversation-corpus&quot;, split=&quot;train&quot;).to_pandas()<br>dataset = dataset.dropna(subset=[&#39;text&#39;]) #drop null text rows.</pre><p>Now, prepare the dataset so that it can be fed into the Small Language Model (SLM) as a list of complete customer chats, where each chat will be assessed with a score between 0 and 100, indicating its level of optimism or pessimism.</p><pre>def closing_snippet(group, k=5):<br>    &quot;&quot;&quot;Return the last k agent lines (chronological order).&quot;&quot;&quot;<br>    agent_tail = (<br>        group[group[&#39;speaker&#39;] == &#39;client&#39;]<br>        .sort_values(&#39;date_time&#39;)<br>        .tail(k)[&#39;text&#39;]<br>    )<br>    return &#39;\n&#39;.join(agent_tail)<br><br>slices = (<br>    dataset.groupby(&#39;conversation_id&#39;)<br>           .apply(lambda g: pd.Series({<br>               &#39;total_text&#39;: &#39;\n&#39;.join(g[&#39;text&#39;]),<br>               &#39;client_text&#39;: &#39;\n&#39;.join(g[g[&#39;speaker&#39;] == &#39;client&#39;][&#39;text&#39;]),<br>               &#39;closing_client_text&#39;: closing_snippet(g)<br>           }))<br>           .reset_index()<br>)</pre><p>The following SLM request code processes each conversation and saves the final results to a CSV file named conversation_scores.csv which has conversation_id, general_text_score, general_client_score, and closing_client_score— the unique chat ID followed by optimism scores for the entire dialogue, for client messages only, and for the client’s final five messages, respectively.</p><pre>OLLAMA_URL = &quot;http://localhost:11434/api/generate&quot;<br>MODEL      = &quot;llama3&quot;<br>PROMPT_TMPL = (<br>    &quot;Please output a single integer between 0 and 100 (nothing else) that &quot;<br>    &quot;represents how optimistic the following text is.\n\n{}&quot;<br>)<br><br>def score(text: str) -&gt; int | None:<br>    payload = {&quot;model&quot;: MODEL, &quot;prompt&quot;: PROMPT_TMPL.format(text), &quot;stream&quot;: False}<br>    try:<br>        r = requests.post(OLLAMA_URL, json=payload, timeout=120)<br>        r.raise_for_status()<br>        num = re.search(r&#39;\d+&#39;, r.json()[&quot;response&quot;])<br>        return int(num.group()) if num else None<br>    except Exception as e:<br>        print(&quot;Ollama call failed:&quot;, e)<br>        return None<br><br>output_path = os.path.join(os.getcwd(), &quot;data&quot;, &quot;conversation_scores.csv&quot;)<br>os.makedirs(os.path.dirname(output_path), exist_ok=True)<br><br>with open(output_path, &quot;w&quot;, encoding=&quot;utf‑8&quot;) as f:<br>    f.write(&quot;conversation_id,general_text,general_client,closing_client\n&quot;)<br><br>    for _, row in slices.iterrows():<br>        cid   = row[&#39;conversation_id&#39;]<br>        s_cli = score(row[&#39;client_text&#39;])<br>        s_total  = score(row[&#39;total_text&#39;])<br>        s_clo = score(row[&#39;closing_client_text&#39;])<br><br>        print(f&quot;{cid} → total:{s_total}, client:{s_cli}, close:{s_clo}&quot;)<br>        with open(output_path, &quot;a&quot;, encoding=&quot;utf‑8&quot;) as f:<br>            f.write(f&quot;{cid},{s_total},{s_cli},{s_clo}\n&quot;)<br><br>print(f&quot;___________\nAll scores saved to {output_path}&quot;)</pre><p>During execution, the script will print each conversation_id along with its corresponding scores as shown below:</p><p>00000ecf65b04c5790f348e9455e790b → total:64, client:42, close:85<br>00004405b72d4e81908143e4427b41f3 → total:75, client:27, close:17<br>00005aa1d5704cf69dd295de02663eb5 → total:87, client:75, close:90<br>000091d575c74c168f0d923afac74a93 → total:65, client:67, close:12</p><p>To prevent it from printing the results during execution, simply comment out the line print(f”{cid} → total:{s_total}, client:{s_cli}, close:{s_clo}”) inside the script.</p><h3>RESULTS VERIFICATION</h3><p>For conversation <strong>000c54251dd949b88b657e314aa882a1</strong>, the model gave a low optimism profile — <strong>overall 13, client 14, closing 1</strong> — and the transcript backs it up: the customer sighs, calls the overage fee “ridiculous,” rejects the agent’s offer, demands a supervisor, and ends the call abruptly, all of which confirms the model’s accuracy in detecting an unhappy interaction.</p><pre>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:24:06+00:00,&quot;Hello, thank you for calling Union Mobile. My name is Marylouise, how can I assist you today?&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:24:13.846154+00:00,&quot;Hi, I&#39;m calling about my bill. I was charged for going over my data limit and I want to know why.&quot;<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:24:23.538462+00:00,&quot;Of course, Izora. Can you please verify your identity by providing me with your account PIN or the last four digits of the credit card associated with your account?&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:24:36.923077+00:00,&quot;Sure, it&#39;s *******.&quot;<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:24:38.307692+00:00,&quot;Thank you, Izora. Now apologize, but I&#39;m having trouble finding your account with that information. Can you please try again or&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:24:48+00:00,*sighs* Fine. It&#39;s *******.<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:24:49.846154+00:00,&quot;Thank apologize for the inconvenience, Izora. I, regarding your concern about the overage charge, I can see that you did indeed go over your data limit for the month. Our system automatically charges $10 for every additionalabyte of beyond the allocated amount.&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:25:09.230769+00:00,That&#39;s ridiculous! I didn&#39;t even use that much data!<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:25:13.384615+00:00,&quot;I understand your frustration, Izora. However, our system has automated, and we&#39;s based on the usage reported by your device. Have you considered upgrading your plan to one with more data? We have a plan that offers 5GB extra data for only $10 more per month.&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:25:34.615384+00:00,&quot;No, I don&#39;t want to upgrade my plan. I just want you to remove this stupidus charge.&quot;<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:25:42.461538+00:00,&quot;I apologize, but I&#39;m unable to remove the charge. Our policy clearly states that all data usage is billed accordingly, and we cannot make exceptions. However, I can offer you a one-time courtesy credit of $10 to help offset the cost of the overage charge. Would you like me to apply that to your account?&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:26:07.846153+00:00,&quot;No, that&#39;s not good enough. I demand to speak to a supervisor.&quot;<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:26:13.384615+00:00,&quot;C course, Izora. I understand that you&#39;re upset. Let me transfer you to my supervisor, Ms. Johnson. She&#39;ll be happy to assist you further.&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:26:24.461538+00:00,Finallysighs* Fine. Thank you.<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:26:26.307692+00:00,&quot;Thank you for your patience, Izora. Ms. Johnson will be with you shortly.&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:26:32.307692+00:00,&quot;Yeah, no kidding. This is ridiculous.&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:26:35.076923+00:00,&quot;*sighs* Fine. Yes, do it.&quot;<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:26:37.384615+00:00,No. Just make sure this doesn&#39;t happen again.<br>000c54251dd949b88b657e314aa882a1,client,2023-09-23T15:26:41.076923+00:00,*hangs up*<br>000c54251dd949b88b657e314aa882a1,agent,2023-09-23T15:26:42+00:00,&quot;*sighs* Well, that was a tough one. Next hope the customer leaves us feedback so we can improve our services. Oh well, next caller, please!&quot;</pre><p>The complete code can be downloaded from Github using the following command:</p><pre>git clone https://github.com/yousef-alkhanafseh/llama3-ISP-chat-rating.git</pre><h3>CONCLUSION</h3><p>In short, By pairing the 8‑B Llama 3‑Instruct model with Ollama, we proved that ISP chat sentiment can be scored locally on a single consumer GPU. The resulting 0–100 optimism scores surface standout conversations for agent coaching and churn models without sending data off‑prem. A bit of fine‑tuning or dashboard integration can turn this lightweight pipeline into a full‑blown, low‑cost sentiment engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*3oGUI8IT2uUczaTSgx7XwA.png" /><figcaption>Image generated by Dall-E3</figcaption></figure><h3><strong>REFERENCES</strong></h3><p><strong>[1]</strong> Ollama (2024). llama3:8b. Accessed on [24.06.2025]. Retrieved from: <a href="https://ollama.com/library/llama3:8b">https://ollama.com/library/llama3:8b</a></p><p><strong>[2]</strong> Marijan, B. (2025). How to Install Python 3 on Ubuntu. Accessed on [27.06.2025]. Retrieved from: <a href="https://phoenixnap.com/kb/how-to-install-python-3-ubuntu">https://phoenixnap.com/kb/how-to-install-python-3-ubuntu</a></p><p><strong>[3]</strong> Alkhanafseh, Y. (2025). Installation of Apache Spark with GPU Acceleration. Accessed on [26.06.2025]. Retrieved from: <a href="https://medium.com/turknettech/installation-of-apache-spark-with-gpu-acceleration-d884350bae46">https://medium.com/turknettech/installation-of-apache-spark-with-gpu-acceleration-d884350bae46</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab058fb8435e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/small-language-model-for-sentiment-classification-in-isp-chats-ab058fb8435e">Small Language Model for Sentiment Classification in ISP Chats</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding TensorFlow Transfer Learning]]></title>
            <link>https://medium.com/turknettech/understanding-tensorflow-transfer-learning-0875405a9f2c?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0875405a9f2c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Wed, 26 Feb 2025 11:07:04 GMT</pubDate>
            <atom:updated>2025-02-26T11:07:04.591Z</atom:updated>
            <content:encoded><![CDATA[<p>This article provides a step-by-step guide on performing transfer learning with pre-trained Artificial Intelligence (AI) models using TensorFlow. It explores how to use built-in datasets to fine-tune models efficiently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*rRpZzSEIii0JoKJ83KmEdg.png" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3><strong>INTRODUCTION</strong></h3><p>With the rapid advancement of AI, numerous pre-trained models are being developed and published. Many of these models feature impressive architectures and have been trained on massive datasets, making them highly valuable for a variety of tasks. However, they may not always deliver accurate predictions when applied to new datasets. One effective way to improve their performance is through <strong>transfer learning</strong>, where only specific layers of the pre-trained model are retrained while the rest remain frozen. This approach significantly reduces computational costs while enhancing model adaptability. In this article, we will explore how to perform transfer learning using <strong>TensorFlow and Python</strong>, focusing on the <strong>EfficientNetV2 model</strong> [1] with human and horse built-in dataset. The performance of the model before and after applying transfer learning will be compared and analyzed emphasizing the importance of transfer learning enhancement. By the end of this guide, readers will gain the skills to apply transfer learning to any pre-trained model available in TensorFlow, optimizing AI models efficiently for their own datasets.</p><h3><strong>USED HARDWARE &amp; SOFTWARE</strong></h3><p>During this study, a Mac Apple Chip device was used as the hardware. In addition, the following software, along with their respective versions, was used:</p><h4>Miniconda</h4><p>Miniconda is a lightweight distribution of Conda, a package and environment management system for Python. It is used for creating and managing isolated Python environments, ensuring dependency compatibility, and efficiently handling package installations. Its installation steps are as follows:</p><pre>curl -o Miniconda3-latest-MacOSX-arm64.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh</pre><pre>bash Miniconda3-latest-MacOSX-arm64.sh</pre><pre>source ~/.zshrc</pre><p>Ensure it is downloaded by;</p><pre>conda --version</pre><p>Create your own environment</p><pre>conda create -n tf_tl_tutorial python=3.12</pre><pre>conda activate tf_tl_tutorial</pre><p>Install required libraries</p><pre>pip install tensorflow <br>pip install tensorflow-datasets<br>pip install opencv-python<br>pip install scikit-learn<br>pip install matplotlib<br>pip install pandas<br>pip install plotly</pre><p>For some reasons, default C++ implementation of TensorFlow dataset, importing it may lead to some errors. To overcome this problem the puer Python can be used by running the following bash command;</p><pre>export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python</pre><p>Install Jupyter Notebook;</p><pre>conda install -c conda-forge jupyter</pre><pre>jupyter --version</pre><p>Install special Jupyter notebook Kernel for our enviroment;</p><pre>pip install ipykernel</pre><pre>python -m ipykernel install --user --name=tf_tl_learning --display-name &quot;tf_tl_learning&quot;</pre><p>Finally, open Jupyter Notebook and then create a new notebook under the <strong>tf_tl_learning</strong> kernel as it is shown in Figure 1, below.</p><pre>jupyter notebook</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*mYu9dPaSFYR1AIICOXpNkw.png" /><figcaption><strong>Figure 1.</strong> Jupyter Notebook Kernal Selection</figcaption></figure><h3><strong>TENSORFLOW BUILT-IN DATASETS</strong></h3><p>In general, TensorFlow has special module called tensorflow_datasets that contains several types of datasets related to different subjects [2] [3]. To retrieve the names of the built-in datasets available in TensorFlow Datasets, the list_builders() function can be used. This function returns a list of all available dataset names. You can select any data based on its name.</p><pre>import tensorflow_datasets as tfds<br><br>print(tfds.list_builders())</pre><p>After that, to load a dataset based on its name, the load() function is used. This function provides multiple arguments, but the most essential ones can be summerized as:</p><ul><li><strong>name</strong>: The name of the dataset to be loaded.</li><li><strong>split</strong>: Specifies which portion of the dataset to load (e.g., &quot;train&quot;, &quot;test&quot;, or &quot;validation&quot;). It can be used to combine the splited datasets as well [4].</li><li><strong>download</strong>: Determines whether to download the dataset locally. If the dataset has not been downloaded previously, this must be set to True.</li><li><strong>shuffle_files</strong>: Indicates whether to shuffle the dataset files during loading.</li><li><strong>with_info</strong>: If set to True, the function returns the dataset metadata .</li><li><strong>as_supervised</strong>: When True, the function returns the dataset as a tuple (input, label)</li></ul><p>During this study, the <strong>horses_or_humans</strong> dataset is supposed to be used. This dataset consists of images of humans and horses, making it suitable for image classification tasks. Figure 2 shows one example of each class, human (0) on left and horse on right (1), found in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*vp6ucZnC81Gxn9XgrgfV6w.png" /><figcaption><strong>Figure 2. </strong>Sample images from the TensorFlow “horses_or_humans” built-in dataset.</figcaption></figure><p>The data is loaded as follows:</p><pre>data, data_info = tfds.load(name=&#39;horses_or_humans&#39;,<br>                             split=&#39;train+test&#39;,<br>                             download=True,<br>                             shuffle_files=False,<br>                             with_info=True,<br>                             as_supervised=False)<br>data_info</pre><p>A detailed explanation of the dataset is provided in Figure 3 below, which is generated by setting with_info=True while loading the data. Briefly, the dataset consists of 1,283 colorful images of horses (labeled as 0) and humans (labeled as 1), each with a resolution of 300×300 pixels. The total size of the dataset on disk is 153.53 MB. It is essential to recognize that not all datasets within tensorflow_datasets share the same structure. Therefore, reviewing the dataset&#39;s information is crucial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxfjh-KQvQcetkN8zm8puA.png" /><figcaption><strong>Figure 3.</strong> Details of the dataset, along with brief explanations.</figcaption></figure><p>The dataset is stored in a dictionary format, where &#39;train&#39; and &#39;test&#39; serve as keys, and their corresponding datasets are the values. These values are instances of tensorflow.python.data.ops.prefetch_op._PrefetchDataset, which is an iterable dataset format.</p><p>Each sample within this dataset is represented as a dictionary containing:</p><ul><li>&#39;image&#39;: A Tensor representing the image pixels.</li><li>&#39;label&#39;: A Tensor representing the image label (indicating whether the image belongs to a horse or a human).</li></ul><p>Since we specified split=&#39;train+test&#39; when calling the load() method, the train and test datasets were combined into a single dataset. As a result, the entire dataset is treated as a single tensorflow.python.data.ops.prefetch_op._PrefetchDataset instance, containing all the data in one partition.</p><p>For reference, the following code can be used to convert between a Tensor, a NumPy array, and a list:</p><pre># tensor to numpy array<br>tensor.numpy()<br># tensor to list<br>tensor.numpy().tolist()<br><br># numpy array to tensor | list to tensor<br>import tensorflow<br>tensorflow.convert_to_tensor(numpy_array)</pre><p>In order to understand the data, one sample can be taken using .take(1) method and displaying it using matplotlib.pyplot.imshow().</p><pre>import matplotlib.pyplot as plt<br>for i in data.take(1):<br>    plt.imshow(i[&#39;image&#39;].numpy(), cmap=&quot;gray&quot;)</pre><p>Using the following script, the data is split into training, validation, and test with the ratios of %70 (898 samples), %15 (192 samples), and 15% (193 samples), respectively. Moreover, the X (Independent) dataset of each one is resized from (300, 300, 3) to (224, 224, 3) to be appropriate with EfficientNetV2 input shape.</p><pre>X = [i[&#39;image&#39;].numpy() for i in data]<br>y = [i[&#39;label&#39;].numpy() for i in data]<br><br>data_len = len(X)<br>train_ratio = 0.7<br>test_validation_ratio = 0.3<br><br>X_train = X[:int(data_len*train_ratio)]<br>y_train = np.array(y[:int(data_len*train_ratio)])<br><br>X_validation = X[int(data_len*train_ratio): int(data_len*train_ratio) + int(data_len*test_validation_ratio/2)]<br>y_validation = np.array(y[int(data_len*train_ratio): int(data_len*train_ratio) + int(data_len*test_validation_ratio/2)])<br><br>X_test = X[int(data_len*train_ratio) + int(data_len*test_validation_ratio/2):]<br>y_test = np.array(y[int(data_len*train_ratio) + int(data_len*test_validation_ratio/2):])<br><br>X_train_resized = np.array([cv2.resize(img, (384, 384)) for img in X_train])<br>X_validation_resized = np.array([cv2.resize(img, (384, 384)) for img in X_validation])<br>X_test_resized = np.array([cv2.resize(img, (384, 384)) for img in X_test])</pre><p>The distribution of each class in these datasets is shown in Figure 4, highlighting the balance between the training, validation, and test datasets for each class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*35RdgOupTSQWYizHGsv1jA.png" /><figcaption><strong>Figure 4.</strong> Distribution of data after splitting.</figcaption></figure><h3>TRANSFER LEARNING</h3><blockquote>A pre-trained model is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. You either use the pretrained model as is or use transfer learning to customize this model to a given task. [5]</blockquote><p>Therefore, pre-trained models, with some modifications, can often be sufficient for solving a problem without the need to build a new model from scratch. Transfer learning is not limited to a single technique; some popular approaches include retraining the entire model, freezing certain layers while retraining others, adding new custom layers, and using a hybrid approach that combines these methods.</p><p>Even though transfer learning is useful, it presents some challenges that may not always be resolved for a given task. Due to the inherent complexity of pre-trained models, the newly provided data may lead to overfitting, resulting in memorization rather than generalization [6].</p><p>In this study, the EfficientNetV2 model is fine-tuned using transfer learning to classify humans and horses by retraining only the last layer, the top is not included. The model itself has Total parameters of 20,332,641</p><p>TensorFlow Hub, in [7], states that all its pre-trained models are available on <a href="https://www.kaggle.com/models?tfhub-redirect=true"><strong>TFHub.dev</strong></a>, which redirects users to the Kaggle website. Additionally, pre-trained models can be accessed through tensorflow.keras.applications [8]. This tutorial adopts the second approach.</p><p>As our problem is classification problem, we should know the corresponding name of each class. In order to do that the following block of code can be used to load EfficientNetV2 class names:</p><pre>import json<br>import tensorflow as tf<br><br>imagenet_labels_url = &quot;https://storage.googleapis.com/download.tensorflow.org/data/imagenet_class_index.json&quot;<br>imagenet_labels_path = tf.keras.utils.get_file(&quot;imagenet_class_index.json&quot;, imagenet_labels_url)<br><br>with open(imagenet_labels_path) as f:<br>    class_idx = json.load(f)<br><br>imagenet_labels = {int(k): v[1] for k, v in class_idx.items()}</pre><p>After that, EfficientNetV2 can be loaded using our predefined efficientnet_v2_model function. This function includes the trainable and layers_num variables, which determine whether the model should be used as is or adapted via transfer learning, and specify the number of trainable layers, respectively.</p><ul><li>If trainable is True, it removes the default top, freezes all but the last layers_num layers, and adds a custom binary classification head (global average pooling + sigmoid).</li><li>If trainable is False, it keeps the entire model (with the default top) frozen without adding any new layers.</li><li>In both cases, the model is compiled with the Adam optimizer, a binary crossentropy loss, and accuracy as the metric.</li></ul><pre>def efficientnet_v2_model(trainable, layers_num=None):<br>    # Load the EfficientNetV2 model from tf.keras.applications<br><br>    # If fine-tuning is enabled, unfreeze only the last `layers_num` layers<br>    if trainable:<br>        base_model = tf.keras.applications.EfficientNetV2S(<br>            include_top=False,  # Remove default classifier<br>            input_shape=(384, 384, 3),<br>            weights=&#39;imagenet&#39;  # Use pre-trained weights<br>        )<br><br>        for layer in base_model.layers[:-layers_num]:  <br>            layer.trainable = False  # Keep early layers frozen<br>        for layer in base_model.layers[-layers_num:]:  <br>            layer.trainable = True   # Unfreeze the last `layers_num` layers<br><br>        # Define the model architecture<br>    <br>        model = tf_keras.Sequential([<br>            tf_keras.layers.InputLayer(input_shape=(384, 384, 3)),  # Explicit input layer<br>            base_model,  # EfficientNetV2 feature extractor<br>            tf_keras.layers.GlobalAveragePooling2D(),  # Convert feature map (7x7x1280) to vector (1280,)<br>            tf_keras.layers.Dense(1, activation=&#39;sigmoid&#39;)  # Output layer for binary classification<br>        ])<br>    <br>    else:<br><br>        base_model = tf.keras.applications.EfficientNetV2S(<br>            include_top=True,  # Remove default classifier<br>            input_shape=(384, 384, 3),<br>            weights=&#39;imagenet&#39;  # Use pre-trained weights<br>        )<br>        <br>        base_model.trainable = False  # Keep entire model frozen<br>        model = tf_keras.Sequential([<br>            base_model<br>        ])<br>    <br>    # Compile the model<br>    model.compile(optimizer=tf_keras.optimizers.Adam(0.00005), loss=&#39;binary_crossentropy&#39;, metrics=[&#39;accuracy&#39;])<br><br><br>    # Print model summary<br>    model.summary()<br><br>    return model</pre><p>In addition, train_model function is defined to be used to train the model over 15 epochs, using 32 samples as batch size, and validate the training process over validation dataset.</p><pre>def train_model(model, X_train, y_train, X_validation, y_validation):<br>    history = model.fit(X_train, y_train, epochs=15, batch_size = 2**5, validation_data = (X_validation, y_validation))<br>    return history</pre><p>Then, model_evaluate function is defined to display the model&#39;s performance metrics, including the classification report (accuracy, precision, recall) and the confusion matrix.</p><pre>def model_evaluate(result):<br>    y_pred = []<br>    for i in result.tolist():<br>        if i[0] &gt; 0.5:<br>            y_pred.append(1)<br>        else:<br>            y_pred.append(0)<br>            <br>    y_pred = np.array(y_pred)<br>    <br>    print(classification_report(y_test, y_pred))<br>    ConfusionMatrixDisplay(confusion_matrix(y_test, y_pred)).plot(cmap=&#39;Reds_r&#39;)</pre><p>First, let’s evaluate the model without training by running the following code snippet.</p><pre>model_without_tl = efficientnet_v2_model(False)<br>result = []<br>for i in range(X_test_resized.shape[0]):<br>    result.append(np.argmax(model_without_tl.predict(np.expand_dims(X_test_resized[i], axis=0))))<br><br><br>print(&quot;Actual | Predicted&quot;)<br>print(&quot;--------------------------&quot;)<br>for predicted_y, actual_y in zip(result, y_test):<br>    if actual_y == 0:<br>        actual_name = &#39;Horse&#39;<br>    else:<br>        actual_name = &#39;Human&#39;<br>    print(actual_name, &quot; |&quot; , imagenet_labels[predicted_y])</pre><p>In Figure 5, the model’s trainable parameters are zero, indicating it is loaded in a fully frozen state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8y9tJabFUAaO8E4zzhTWiQ.png" /><figcaption><strong>Figure 5.</strong> Untrained model summary</figcaption></figure><p>When examining the results, the untrained model, as expected, performs poorly and fails to accurately predict the labels. The top five predictions confirm that it never identifies any class correctly, largely because these two classes are not part of the original 1,000 class set.</p><pre>Actual | Predicted<br>--------------------------<br>Human  | jean<br>Human  | jean<br>Horse  | German_short-haired_pointer<br>Human  | cardigan<br>Human  | seashore</pre><p>Nevertheless, when we retrain only the last two layers of the model — comprising 3,841 out of 20,332,641 parameters, which accounts for just 0.01889% of the total — we achieve remarkable improvement.</p><pre>model_with_tl = efficientnet_v2_model(True, 2)<br>history = train_model(model_with_tl, X_train_resized, y_train, X_validation_resized, y_validation)<br>result = model_with_tl.predict(X_test_resized)<br>modelEvaluate(result)</pre><p>Figure 6 presents the training and validation accuracies across epochs, indicating that the model did not suffer from overfitting or underfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*EiZhm9k89R8HzkbI9Ypg1g.png" /><figcaption><strong>Figure 6.</strong> Training and validation accuracy over epochs.</figcaption></figure><p>After 15 epochs, the model correctly classifies all 94 human samples and 97 out of 98 horse samples, misclassifying just one horse as a human, refer to Figure 7. This demonstrates that, without the need to design a new model or train millions of parameters, a significant performance boost can be achieved with minimal effort by fine-tuning just the last two layers of EfficientNetV2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*sXnFxGbsfUB3HRzxW9qtWA.png" /><figcaption><strong>Figure 7.</strong> Confusion matrix of the results.</figcaption></figure><h3>CONCLUSION</h3><p>In summary, this study provides a detailed explanation of transfer learning in TensorFlow through a comprehensive case study. The built-in TensorFlow dataset used in the study is thoroughly examined and the obtained results are completely shared. The focus is on highlighting the importance of transfer learning, which allows users to minimize architecture design efforts, reduce hardware requirements, lower computational costs, and shorten execution time — while still achieving outstanding results. Ultimately, this tutorial serves as a practical guide, demonstrating how to effectively implement transfer learning using TensorFlow and Python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*MkVVd_GDlZD_w_1uhH121Q.png" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3>REFERENCES</h3><p>[1] Tan, M., &amp; Le, Q. (2021, July). Efficientnetv2: Smaller models and faster training. In <em>International conference on machine learning</em> (pp. 10096–10106). PMLR.</p><p>[2] TensorFlow (n.d). TensorFlow Datasets. Accessed on [23.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/datasets/overview">https://www.tensorflow.org/datasets/overview</a></p><p>[3] TensorFlow (n.d). TensorFlow Datasets: a collection of ready-to-use datasets. Accessed on [22.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/datasets">https://www.tensorflow.org/datasets</a></p><p>[4] TensorFlow (n.d). Splits and slicing. Accessed on [22.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/datasets/splits">https://www.tensorflow.org/datasets/splits</a></p><p>[5] TensorFlow (n.d). Transfer learning and fine-tuning. Accessed on [24.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/tutorials/images/transfer_learning">https://www.tensorflow.org/tutorials/images/transfer_learning</a></p><p>[6] TensorFlow (n.d). Transfer learning with TensorFlow Hub. Accessed on [26.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/tutorials/images/transfer_learning_with_hub">https://www.tensorflow.org/tutorials/images/transfer_learning_with_hub</a></p><p>[7] TensorFlow (n.d). TensorFlow Hub is a repository of trained machine learning models. Accessed on [20.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/hub">https://www.tensorflow.org/hub</a></p><p>[8] TensorFlow (n.d). Module TensorFlow Keras Applications. Accessed on [24.02.2025]. Retrieved from: <a href="https://www.tensorflow.org/api_docs/python/tf/keras/applications">https://www.tensorflow.org/api_docs/python/tf/keras/applications</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0875405a9f2c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/understanding-tensorflow-transfer-learning-0875405a9f2c">Understanding TensorFlow Transfer Learning</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing Python Data Tools: Operations and Resource Usage]]></title>
            <link>https://medium.com/turknettech/comparing-python-data-tools-operations-and-resource-usage-86b0faf84507?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/86b0faf84507</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Thu, 10 Oct 2024 07:36:45 GMT</pubDate>
            <atom:updated>2024-10-10T07:36:45.248Z</atom:updated>
            <content:encoded><![CDATA[<p>A performance comparison of Polars, Pandas, and PySpark libraries in Python, with a focus on execution time and resources utilization. The analysis highlights how each library handles resource utilization, providing insights into their efficiency for different operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*2pMRQFIP4xUukn8Ie0UiUQ.png" /></figure><h3><strong>I. INTRODUCTION</strong></h3><p>When working with large datasets, the efficiency of data operations in terms of execution time and resources usage is crucial. Operations that are too slow or resource-intensive can block data processing and lead to errors. It is essential for data manipulation tools to perform operations as fast as possible and manage resources effectively. This article provides a comparative analysis between three widely-used Polars, Pandas and Pyspark libraries, focusing on their performance across various data tasks. The comparison is conducted on 1.2 GB dataset taken from TurkNet ISP. The article evaluates their performance by covering the most commonly used operations in data processing such as loading, column selection, row selection, filtering, grouping &amp; aggregating, joining, pivoting, applying external function, string operation, datetime operation, and writing. It aims to help data professionals make decisions about which tool to use in various contexts as well.</p><h3><strong>II. DATA TOOLS OVERVIEW</strong></h3><p>The primary objective of data tools is to enable efficient data manipulation. While their core functions are similar, they differ in terms of performance, flexibility, and optimization. As this tutorial focuses on comparing three prominent Python data tools — Polars, Pandas, and PySpark — only these tools are reviewed.</p><ul><li><strong>Polars</strong></li></ul><p>Polars is a cutting-edge, open-source data manipulation and analysis tool built using the Rust programming language. It stands out for its exceptional speed and performance, primarily due to its use of SIMD (Single Instruction, Multiple Data) vectorization and columnar processing [1]. Polars offers an array of powerful features, including numerous built-in functions, an intuitive syntax, and support for DataFrame and Series structures. Additionally, it incorporates lazy evaluation to optimize computations. Polars currently supports approximately 80% of the functions available in Pandas [2]. It is designed to excel in parallel processing and maximize memory efficiency, making it an ideal choice for handling large datasets.</p><ul><li><strong>Pandas</strong></li></ul><p>Pandas is one of the most popular Python data manipulation tools, characterized by providing fast, flexible, and expressive operations related to both “relational” and “labeled” data. Specifically, Pandas has two data structures: Series (1-dimensional) and DataFrame (2-dimensional) [3]. In short, the first type can be considered a container for scalars, and the latter one as a container for Series. One of the greatest advantages of Pandas is the abundance of available supportive documentation. It supports several different third-party data tools such as NumPy and Pyspark.</p><ul><li><strong>Apache Spark</strong></li></ul><p>Apache spark, Pyspark in Python, is a large-scale data processing engine distinguished by its ability to process data in-memory across multiple interconnected machines, forming a cluster. This distributed architecture allows for highly efficient data processing at scale. The engine supports various programming languages, including Java, R, and Python, providing flexibility for developers across different ecosystems [4]. Additionally, it offers a range of built-in, highly effective functions that simplify complex data manipulation tasks, enhancing productivity and performance. The engine introduces three key data structures: Resilient Distributed Dataset (RDD), DataFrame, and DataSet, each designed to cater to different processing needs. For a detailed explanation of these structures, please refer to [5]. However, it is important to note that the DataSet API is not supported in Python, limiting its use to other supported languages [4]. Apache Spark also supports real-time data processing, making it suitable for both batch and stream processing tasks. It includes several powerful built-in libraries, such as MLlib, GraphX, and Structured Streaming.</p><h3><strong>III. SYSTEM SPECIFICATIONS and SOFTWARE VERSIONS</strong></h3><p>The system utilized for this work is configured with Windows Subsystem for Linux 2 (WSL 2), equipped with 16 GB of RAM and a 15-core CPU. Software specifications are as follow:</p><ul><li>Python 3.9.17, released on June 6, 2023.</li><li>Polars 1.5.0, released on August 14, 2024</li><li>Pandas 2.2.2, released on April 10, 2024</li><li>and Apache Spark (local mode) 3.5.2 released on August<strong> </strong>12, 2024.</li></ul><p>These software versions were chosen to represent the most recent releases, ensuring access to the latest features and optimizations.</p><h3><strong>IV. DATASET OVERVIEW</strong></h3><p>This section provides an overall overview of the dataset, detailing its key attributes, data types, and overall structure. The dataset, sourced from TurkNet ISP, encompasses customer internet session data. It has 24 files, totaling 1.2 GB, and consists of 9,365,106 rows and 13 columns. It is crucial to note that the data has been fully anonymized using a randomization technique, ensuring that even the author has no access to the actual information. This anonymization process ensures the complete protection of customer privacy while still allowing for insightful and meaningful analysis. A sample of 5 rows is shown in Table 1, below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZK4VBqcb1HLzksJnappWhg.png" /></figure><p>The dataset includes 8 columns with integer data types and 5 columns with string data types, as outlined in the dictionary below:</p><pre>{<br>  &quot;username&quot;: &quot;str&quot;,<br>  &quot;private_ip&quot;: &quot;str&quot;,<br>  &quot;private_port_begin&quot;: &quot;int&quot;,<br>  &quot;private_port_end&quot;: &quot;int&quot;,<br>  &quot;real_ip&quot;: &quot;str&quot;,<br>  &quot;port_begin&quot;: &quot;int&quot;,<br>  &quot;port_end&quot;: &quot;int&quot;,<br>  &quot;start_time&quot;: &quot;int&quot;,<br>  &quot;log_time&quot;: &quot;int&quot;,<br>  &quot;download&quot;: &quot;int&quot;,<br>  &quot;upload&quot;: &quot;int&quot;,<br>  &quot;terminatecause&quot;: &quot;str&quot;,<br>  &quot;statustype&quot;: &quot;str&quot;,<br>}</pre><h3>V. DATA TOOLS COMPARISION</h3><p>The comparison is based on two key metrics: the execution time of various data operations and the efficiency of resource utilization. These metrics provide a comprehensive evaluation of each tool’s performance, measuring both speed and how effectively they leverage the available system resources.</p><p><strong>V.I. Data Operations</strong></p><p>The comparison was conducted across twelve different data operation tasks, including data loading, column selection, row selection, filtering, grouping &amp; aggregating, joining, pivoting, sorting, applying external functions, string operations, datetime operations, and writing. According to the experimental results, Pandas demonstrated the poorest overall performance, excelling only in the row selection operation. Although the Polars official website claims that certain operations take only 8 seconds while PySpark takes 55 seconds for the same task, this comparison holds true only when PySpark is not optimized. However, based on our findings, after optimizing PySpark configurations, it is evident that Apache Spark outperforms Polars in six different operations, while Polars outperforms Apache Spark in five operations (see Figure 1 for detailed results). Overall, Apache Spark completed all the data operations on the specified dataset in 13,049 milliseconds, whereas Polars and Pandas took 16,241 milliseconds and 71,146 milliseconds, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qSLIUbj0XdHCc4Y_gr2cwg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*yqzzLVsnw62ImV5n2OhSQQ.png" /><figcaption><strong>Execution Time Comparison for Data Operation Tasks: Polars vs. Pandas vs. PySpark</strong></figcaption></figure><h4>V.II. Resources Usage</h4><p>The efficient use of hardware resources is crucial when choosing the right data tool. Typically, four key resources are prioritized in data processing: Central Processing Unit (CPU), Random Access Memory (RAM), , Graphics Processing Unit (GPU), and disk storage. This tutorial focuses specifically on the first two, detailing their utilization levels following the data manipulation tasks outlined in the previous section.</p><ul><li><strong>CPU</strong></li></ul><p>PySpark is the most efficient data tool in utilizing all the available CPU cores, with an average usage of around 90%. Polars ranks second, utilizing all available cores but at an average rate of 35%. In contrast, Pandas is the least efficient in terms of CPU utilization, as it only uses a single core. However, Pandas’ performance can be significantly improved by incorporating additional libraries like Multiprocessing. The CPU core utilization, as measured using the htop command, for PySpark, Polars, and Pandas is shown in Figures 2.a, 2.b, and 2.d, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0WAnXfKLh9-I09MICp6aPQ.png" /><figcaption><strong>Figure 2. a.</strong> Pyspark CPU Usage.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6aTN50_RfBSIjK1edKbLdQ.png" /><figcaption><strong>Figure 2. b. </strong>Polars CPU Usage.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-1JETbWHZS8x8uwGM95Z4A.png" /><figcaption><strong>Figure 2. c.</strong> Pandas CPU Usage.</figcaption></figure><ul><li><strong>RAM</strong></li></ul><p>In contrast to CPU utilization, Pandas demonstrated the best performance in terms of RAM usage, consuming 6.93 GB out of 14.6 GB. However, this is close to Polars and PySpark, which used 7.15 GB and 8.55 GB, respectively. Pandas performs better in this aspect due to its lower CPU core usage and lack of parallel processing. As a result, all three data tools show relatively similar performance in RAM utilization. The RAM usage for PySpark, Polars, and Pandas is shown in Figures 3.a, 3.b, and 3.d, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_MQKzP4b7Ws-Jm854Hwgg.png" /><figcaption><strong>Figure 3. a.</strong> Pandsa RAM Usage.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LFmTP6SpsJ8uU0YtrGUPeg.png" /><figcaption><strong>Figure 3. b.</strong> Polars RAM Usage.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4kJhlm50uJGPzQJ9XVswlQ.png" /><figcaption><strong>Figure 3. c.</strong> Pyspark RAM Usage.</figcaption></figure><h3><strong>VI. CONCLUSION</strong></h3><p>In conclusion, this study compared three prominent Python data manipulation tools — Polars, Pandas, and PySpark — based on two key metrics: execution time for various data operation tasks and the efficiency of resource utilization, including CPU cores and RAM. The first metric assessed performance across twelve critical subtasks. Apache Spark outperformed its counterparts, completing all data operations in 13,049 milliseconds, while Polars and Pandas took 16,241 milliseconds and 71,146 milliseconds, respectively. The Polars official website claims that it outperforms Pyspark, however, based on our findings, after optimizing PySpark configurations, it is evident that Apache Spark outperforms Polars. In terms of CPU utilization, PySpark led the way, followed by Polars and then Pandas. For RAM usage, Pandas was the most efficient, consuming the least memory, whereas PySpark and Polars had comparable performances, with a slight advantage for Polars. In future, GPU based comparision can be conducted between these data tools.</p><p>Used data and Python notebooks can be cloned from GitHub using the following command:</p><pre>git clone https://github.com/yousef-alkhanafseh/python_data_tools.git</pre><h3>VII. REFERENCES</h3><p><strong>[1]</strong> Polars (2024). DataFrames for the new era. Accessed on [07.10.2024]. Retrieved from: <a href="https://pola.rs/">https://pola.rs/</a></p><p><strong>[2]</strong> Ali, M. (2023). An Introduction to Polars: Python’s Tool for Large-Scale Data Analysis. Accessed on [08.10.2024]. Retrieved from: <a href="https://www.datacamp.com/blog/an-introduction-to-polars-python-s-tool-for-large-scale-data-analysis">https://www.datacamp.com/blog/an-introduction-to-polars-python-s-tool-for-large-scale-data-analysis</a></p><p><strong>[3]</strong> Pandas (n.d.). Package overview. Accessed on [09.10.2024]. Retrieved from: <a href="https://pandas.pydata.org/docs/getting_started/overview.html">https://pandas.pydata.org/docs/getting_started/overview.html</a></p><p><strong>[4]</strong> Apache Spark (n.d.). Overview. Accessed on [09.10.2024]. Retrieved from: <a href="https://spark.apache.org/docs/latest/">https://spark.apache.org/docs/latest/</a></p><p><strong>[5]</strong> rimmalapudi (2024). Spark RDD vs DataFrame vs Dataset. Accessed on [09.10.2024]. Retrieved from: <a href="https://sparkbyexamples.com/spark/spark-rdd-vs-dataframe-vs-dataset/">https://sparkbyexamples.com/spark/spark-rdd-vs-dataframe-vs-dataset/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86b0faf84507" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/comparing-python-data-tools-operations-and-resource-usage-86b0faf84507">Comparing Python Data Tools: Operations and Resource Usage</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Essential Apache Spark Configurations, Including GPU Acceleration]]></title>
            <link>https://medium.com/turknettech/essential-apache-spark-configurations-including-gpu-acceleration-fc73df4eb0d5?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc73df4eb0d5</guid>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 11:44:26 GMT</pubDate>
            <atom:updated>2024-09-05T11:44:26.852Z</atom:updated>
            <content:encoded><![CDATA[<p>This guide covers essential application properties, execution behavior, SQL optimizations, networking settings, and RAPIDS GPU configurations critical for enhancing Apache Spark performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*xDySSficBbB9o4PFzzz-IA.png" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3>I. INTRODUCTION</h3><p>As data processing demands grow, the tools used to manage these large datasets are becoming increasingly complex. Even with the capability to build substantial infrastructure, inefficiencies can arise from the misconfiguration of these tools. This article aims to address this issue by focusing on the critical configurations of Apache Spark, a powerful data processing engine. Key aspects such as application properties, runtime environment, SQL operations, GPU acceleration, and networking are examined. By gaining a deeper understanding of these configurations, this study aims to enhance the efficiency of Apache Spark, enabling it to maximize resource utilization and significantly reduce execution time. Practitioners are expected to improve Apache Spark resource utilization, cost efficiency, reliability, performance tuning, decision-making, and future-proofing capabilities.</p><h3>II. APACHE SPARK WITH GPU: INSTALLATION</h3><p>To install Apache Spark integrated with GPU, please refer to the “<a href="https://medium.com/@alkhanafseh/installation-apache-spark-with-gpu-acceleration-d884350bae46"><strong>Installation Apache Spark with GPU Acceleration</strong></a>” article for detailed instructions.</p><h3>III. CONFIGURATIONS</h3><p>Apache Spark configurations can generally be set using three different methods. The first method involves specifying configurations in the spark-defaults.conf and spark-env.sh configuration files. However, this approach is inefficient in standalone or cluster modes because the master and slave nodes must be restarted whenever the configurations are changed. The second method involves setting configurations at runtime within the script itself using the SparkConf function. Nevertheless, this method has limitations as some configurations cannot be applied once the job has already started <strong>[1]</strong>. Therefore, this approach may not be ideal. The final and most effective method is to define configurations when invoking spark-submit, as it ensures all configurations take effect when the job is initiated. However, certain configurations must be set using the first method in specific situations, such as when operating in Standalone mode, where the number of workers needs to be defined before they are started.. The most critical Apache Spark configurations can be categorized into sevven main areas: application properties, runtime environment, execution behavior,<strong> </strong>memory management,<strong> </strong>SQL settings, networking, and GPU RAPIDS <strong>[2]</strong> configurations. Please note that memory amounts are specified in kilobytes, megabytes, gigabytes, or terabytes using the abbreviations “k”, “m”, “g”, or “t”, respectively.</p><p><strong>— Application Properties</strong></p><ul><li><strong>spark.app.name [1]</strong></li></ul><p>Each application runs on Apache Spark has a name can be set using this configuration.</p><ul><li><strong>spark.master [1]</strong></li></ul><p>It defines the Apache Spark application master type. Refer to <strong>[3]</strong> for more information about mater types.</p><ul><li><strong>spark.submit.deployMode [1]</strong></li></ul><p>It is responsible for setting the deployment mode of the Apache Spark driver program, either client or cluster. In the client mode, the Spark driver runs on the machine where the Spark job is submitted (the client machine). On the other hand, in the cluster mode, the Spark driver runs on one of the worker nodes in the cluster, not on the client machine. For more information about the available modes, please refer to <strong>[4]</strong>.</p><ul><li><strong>spark.executor.instances [1]</strong></li></ul><p>The spark.executor.instances configuration is responsible about setting the number of executors to be launched when a Spark application starts.</p><ul><li><strong>spark.dynamicAllocation.enabled [1]</strong></li></ul><p>In general, Apache Spark has the ability to dynamically allocate resources based on the load on the cluster by increasing or decreasing resources such as executors and memory. This feature is enabled or disabled using this configuration. However, if this configuration is enabled, other settings such as spark.dynamicAllocation.minExecutors (minimum number of executors), spark.dynamicAllocation.maxExecutors (maximum number of executors), and spark.dynamicAllocation.initialExecutors (initial number of executors) should be defined.</p><ul><li><strong>spark.driver.maxResultSize [1]</strong></li></ul><p>It specifies the maximum amount of data that can be serialized at the same time. This configuration is important when there is a large shuffle in tasks because if the limit is too small, it could lead to out-of-memory errors.</p><ul><li><strong>spark.driver.memory [1]</strong></li></ul><p>It sets the amount of memory for driver process.</p><ul><li><strong>spark.executor.memory [1]</strong></li></ul><p>It sets the amount of memory for each executor process.</p><ul><li><strong>spark.driver.cores [1]</strong></li></ul><p>It specifies the number of cores assigned to each driver process.</p><ul><li><strong>spark.executor.cores [1]</strong></li></ul><p>It specifies the number of cores assigned to each executor process.</p><ul><li><strong>spark.task.cpus [1]</strong></li></ul><p>It specifies the number of CPU cores to be used for each task.</p><p><strong>— Runtime Environment</strong></p><ul><li><strong>spark.jars [1]</strong></li></ul><p>If there are additional jar files, such as MSSQL, Kafka, RAPIDS, etc., that are not directly installed in the Spark jar folder, they can be specified through this configuration. This allows the Spark job to use them even if they are located in a different folder. If there are multiple jar files, they must be provided as a comma-separated string of paths.</p><ul><li><strong>spark.executor.extraJavaOptions [1]</strong></li></ul><p>It is supposed to be used for setting system properties or tuning the Java Virtual Machine (JVM) for the executor process. It can specify the type of grabage collector and collection pause time as well.</p><ul><li><strong>spark.driver.extraJavaOptions [1]</strong></li></ul><p>It is similer to spark.executor.extraJavaOptions, however, it is used for driver porcess.</p><p><strong>— Execution Behavior</strong></p><ul><li><strong>spark.broadcast.blockSize [1]</strong></li></ul><p>It determines the size of each broadcasted data block. If this configuration value is set too high, the number of blocks will be small, reducing parallelism. Conversely, if the value is set too low, the script’s performance may suffer due to increased memory usage and overhead for metadata management.</p><ul><li><strong>spark.default.parallelism [1]</strong></li></ul><p>It is similar to spark.sql.shuffle.partitions, but this configuration is specifically applicable to Resilient Distributed Dataset (RDD) operations.</p><ul><li><strong>spark.executor.heartbeatInterval [1]</strong></li></ul><p>In general, executors are connected to the driver(s). The driver can detect the connectivity of each executor by receiving messages from them that include their liveness status, health, and confirm that they are functioning correctly. These messages are called heartbeats. The interval for these messages to be sent from executors to the driver(s) is set using the spark.executor.heartbeatInterval configuration. This configuration should be set to a value less than spark.network.timeout to ensure the driver has enough time to handle executor errors. For example, if an executor fails, its tasks can be reassigned to another executor.</p><p><strong>— Memory Management [1]</strong></p><p>Briefly, Apache Spark’s memory is divided into three components: Spark memory, user memory, and reserved memory. The proportion of the total executor memory allocated to Spark’s operational tasks is controlled by the spark.memory.fraction setting. This Spark memory is further divided into two categories: execution memory and storage memory. The allocation of storage memory within the spark.memory.fraction is governed by the spark.memory.storageFraction parameter, with the remaining portion dedicated to execution memory. <strong><em>Storage memory</em></strong> is responsible for tasks such as caching/persisting data, serialization, and broadcasting, while <strong><em>execution memory</em></strong> is utilized for computations like shuffling, sorting, aggregating, and other intermediate operations during the execution of transformations and actions. Please refer to <strong>[5]</strong> for more information about Apache Spark memory management.</p><p><strong>— Networking</strong></p><ul><li><strong>spark.rpc.message.maxSize [1]</strong></li></ul><p>It sets the maximum size of Remote Procedure Call (RPC) between driver(s) and executors. An RPC message is the data packet sent over the network that carries the request for a procedure to be executed remotely.</p><ul><li><strong>spark.network.timeout [1]</strong></li></ul><p>It is used to set the timeout duration, in seconds, for Spark network interactions. It’s important to keep this value relatively high, as network interactions can sometimes be slow. If the value is too low, a timeout error may occur.</p><h4>— SQL Configurations</h4><ul><li><strong>spark.sql.files.maxPartitionBytes [6]</strong></li></ul><p>It is responsible for specifying the maximum number of bytes a single partition can contain, and this configuration is applicable only when reading data from files.</p><ul><li><strong>spark.sql.files.maxPartitionNum [6]</strong></li></ul><p>It is responsible for specifying the maximum number of partitions can be split from data files. This configuration also is applicable only when reading data from files.</p><ul><li><strong>spark.sql.shuffle.partitions [6]</strong></li></ul><p>This configuration is used to set the number of data partitions during shuffling operations, which occur as a result of transformations like joins and aggregations. This is only applicable to DataFrame type.</p><ul><li><strong>spark.sql.adaptive.enabled [6]</strong></li></ul><p>It is responsible about activating the Adaptive Query Execution (AQE) which is a technique used to select the most efficient query execution plan, which in turn reduces the execution time required for queries.</p><ul><li><strong>spark.sql.adaptive.skewJoin.enabled [6]</strong></li></ul><p>It improves the performance of join operations between skewed datasets by splitting them into evenly sized chunks. To activate this feature, both the current configuration and spark.sql.adaptive.enabled must be set to true.</p><ul><li><strong>spark.sql.autoBroadcastJoinThreshold [6]</strong></li></ul><p>It is used to configure the amount of bytes that a table can be broadcasted to workers during transformations such as join or window operations. It can be disabled by setting its value to -1.</p><ul><li><strong>spark.sql.cache.serializer [1][7]</strong></li></ul><p>It defines the type of serializer used for data caching, with compression applied to enhance storage efficiency.</p><ul><li><strong>spark.sql.session.timeZone [1]</strong></li></ul><p>It sets the session timezone and can be specified as an area/city (e.g., Turkey/Istanbul), a zone offset (e.g., +01:00), or an alias (e.g., UTC). It is crucial to keep it synchronized with the server’s system timezone, otherwise, Spark job may not be able to start.</p><p><strong>— GPU RAPIDS configurations</strong></p><ul><li><strong>spark.worker.resource.gpu.discoveryScript [8]</strong></li></ul><p>It specifies the path to the bash script that an Apache Spark worker uses to identify the available GPUs on the machine. RAPIDS provides a pre-prepared script for this purpose, which can be found at <strong>[9]</strong>.</p><ul><li><strong>spark.rapids.sql.enabled [10]</strong></li></ul><p>It is a crucial configuration that determines whether SQL operations are enabled or disabled on the GPU.</p><ul><li><strong>spark.plugins [11]</strong></li></ul><p>It is used to enable an Apache Spark application to utilize plugins or other existing software with specific capabilities, all without affecting the core functionality of Apache Spark.</p><ul><li><strong>spark.rapids.memory.pinnedPool.size [10]</strong></li></ul><p>This configuration controls the size of the memory pool used for pinned memory. Pinned memory, also known as page-locked memory, is a region of memory that is locked in place and cannot be swapped out by the operating system. This allows for faster data transfer between the CPU and GPU, as the GPU can directly access the memory without worrying about the Operating System (OS) moving it around.</p><ul><li><strong>spark.executor.resource.gpu.amount [11]</strong></li></ul><p>It controls the number of GPU(s) per executor.</p><ul><li><strong>spark.rapids.sql.concurrentGpuTasks [10]</strong></li></ul><p>It determines how many Spark tasks can execute on the GP simultaneously.</p><ul><li><strong>spark.task.resource.gpu.amount [11]</strong></li></ul><p>It controls the number of GPUs allocated per task, determining the level of parallelism for tasks on the available GPUs.</p><ul><li><strong>spark.rapids.sql.explain [10]</strong></li></ul><p>It indicates which operations or portions of the code can be executed on the GPU and which cannot.</p><ul><li><strong>spark.rapids.sql.incompatibleDateFormats.enabled [12]</strong></li></ul><p>It enables Apache Spark on GPU to handle undefined date formats. Generally, the GPU can only process predefined formats, as listed in <strong>[13]</strong>. However, if the desired date format is not included, this configuration allows the GPU to attempt to interpret and process the incompatible date formats on its own.</p><ul><li><strong>spark.rapids.force.caller.classloader [14]</strong></li></ul><p>By enabling this setting, the application ensures that the correct classes are loaded during runtime. This sometimes can be necessary for avoiding conflicts or incompatibilities with other components.</p><h3>IV. CASE STUDY</h3><blockquote><em>As a case study, consider the installation of an Apache Spark client type in Standalone mode on a single machine equipped with 2 GPUs, 125GB of RAM, and 31 CPU cores. Write the appropriate Apache Spark configurations for this setup. Additionally, ensure that only one worker is active, and it is configured to fully utilize both GPUs simultaneously. Please note that the application to be run involves some storage operations, such as caching and broadcasting, but primarily focuses on execution-heavy tasks like joins and aggregations.</em></blockquote><p>The answer to this question can be found at [REFHERE].</p><h3>V. CONCLUSION</h3><p>In summary, this study delved into the most critical Apache Spark configurations, covering a comprehensive range of aspects, including application properties, runtime environment, execution behavior, memory management, SQL settings, networking, and GPU acceleration using RAPIDS. The discussed configurations are essential for ensuring that Spark jobs run efficiently without encountering common resource-related errors, such as those related to RAM, CPU, and GPU. By thoroughly understanding and correctly applying these configurations, users can optimize the performance of their Spark jobs and completely be able to use the available resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*jAqJteSuQ3h-pWyh_gfY5A.png" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3>VI. REFERENCES</h3><p><strong>[1]</strong> Apache Spark (n.d). Spark Configuration. Accessed on [02.09.2024]. Retrieved from: <a href="https://spark.apache.org/docs/latest/configuration.html">https://spark.apache.org/docs/latest/configuration.html</a></p><p><strong>[2] </strong>Nvidia Docs Hub (n.d.). RAPIDS Accelerator for Apache Spark. Accessed on [04.09.2024]. Retrieved from: <a href="https://docs.nvidia.com/spark-rapids/index.html">https://docs.nvidia.com/spark-rapids/index.html</a></p><p><strong>[3] </strong>Apache Spark (n.d). Launching Applications with spark-submit. Accessed on [05.09.2024]. Retrieved from: <a href="https://spark.apache.org/docs/latest/submitting-applications.html#master-urls">https://spark.apache.org/docs/latest/submitting-applications.html#master-urls</a></p><p><strong>[4] </strong>Nelamali, N. (2024). Spark Deploy Modes — Client vs Cluster Explained. Accessed on [03.09.2024]. Retrieved from: <a href="https://sparkbyexamples.com/spark/spark-deploy-modes-client-vs-cluster/">https://sparkbyexamples.com/spark/spark-deploy-modes-client-vs-cluster/</a></p><p><strong>[5]</strong> Alkhanafseh, Y. DNS big data processing for detecting customers behaviour of isp using an optimized apache spark cluster.</p><p><strong>[6] </strong>Apache Spark (n.d.). Performance Tuning. Accessed on [04.09.2024]. Retrieved from: <a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html">https://spark.apache.org/docs/latest/sql-performance-tuning.html</a></p><p><strong>[7] </strong>Nvidia RAPIDS (n.d.). RAPIDS Cache Serializerv. Accessed on [04.09.2024]. Retrieved from: <a href="https://nvidia.github.io/spark-rapids/docs/additional-functionality/cache-serializer.html">https://nvidia.github.io/spark-rapids/docs/additional-functionality/cache-serializer.html</a></p><p><strong>[8] </strong>Nvidia Docs Hub (n.d.). On-prem Cluster or Local Mode. Accessed on [03.09.2024]. Retrieved from: <a href="https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/on-premise.html">https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/on-premise.html</a></p><p><strong>[9] </strong>Graves, T. &amp; Hyun, D. (2019). Spark. Accessed on [03.09.2024]. Retrieved from: <a href="https://github.com/apache/spark/blob/master/examples/src/main/scripts/getGpusResources.sh">https://github.com/apache/spark/blob/master/examples/src/main/scripts/getGpusResources.sh</a></p><p><strong>[10]</strong> spark-rapids (n.d.). RAPIDS Accelerator for Apache Spark Configuration. Accessed on [03.09.2024]. Retrieved from: <a href="https://nvidia.github.io/spark-rapids/docs/configs.html">https://nvidia.github.io/spark-rapids/docs/configs.html</a></p><p><strong>[11] </strong>Nvidia (n.d.).<strong> </strong>Getting Started with GPU-Accelerated Apache Spark 3. Accessed on [05.09.2024]. Retrieved from: <a href="https://www.nvidia.com/en-us/ai-data-science/spark-ebook/getting-started-spark-3/">https://www.nvidia.com/en-us/ai-data-science/spark-ebook/getting-started-spark-3/</a></p><p><strong>[12]</strong> spark-rapids (n.d.). RAPIDS Accelerator for Apache Spark Advanced Configuration. Accessed on [03.09.2024]. Retrieved from: <a href="https://nvidia.github.io/spark-rapids/docs/additional-functionality/advanced_configs.html">https://nvidia.github.io/spark-rapids/docs/additional-functionality/advanced_configs.html</a></p><p><strong>[13]</strong> spark-rapids (n.d.). Parsing strings as dates or timestamps. Accessed on [06.09.2024]. Retrieved from: <a href="https://nvidia.github.io/spark-rapids/docs/compatibility.html#parsing-strings-as-dates-or-timestamps">https://nvidia.github.io/spark-rapids/docs/compatibility.html#parsing-strings-as-dates-or-timestamps</a></p><p><strong>[14]</strong> Hewlett Packard Enterprise (n.d.). Nvidia Spark-RAPIDS Accelerator for Spark. Accessed on [05.09.2024]. Retrieved from: <a href="https://docs.ezmeral.hpe.com/runtime-enterprise/56/reference/kubernetes-applications/spark/spark-gpu-optimization.html">https://docs.ezmeral.hpe.com/runtime-enterprise/56/reference/kubernetes-applications/spark/spark-gpu-optimization.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc73df4eb0d5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/essential-apache-spark-configurations-including-gpu-acceleration-fc73df4eb0d5">Essential Apache Spark Configurations, Including GPU Acceleration</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installation of Apache Spark with GPU Acceleration]]></title>
            <link>https://medium.com/turknettech/installation-of-apache-spark-with-gpu-acceleration-d884350bae46?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d884350bae46</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Thu, 05 Sep 2024 11:39:05 GMT</pubDate>
            <atom:updated>2024-09-05T11:39:05.772Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial provides a step-by-step guide to installing Apache Spark 3.3.0 with RAPIDS GPU acceleration in Standalone mode, ensuring a smooth and efficient setup process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kfi7GG4DD2s11CEk" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3>I. INTRODUCTION</h3><p>In recent times, with the widespread accessibility of the web, numerous guides and tutorials have emerged detailing how to install Apache Spark across various operating systems (OSs). However, many of these guides often miss small but crucial details, which can complicate the installation process or even cause it to fail entirely. This documentation aims to provide a comprehensive, step-by-step guide to installing Apache Spark in a standalone cluster on Nvidia GPUs. By collecting all relevant information, this guide simplfy the installation process, enabling it to be completed within just 5 to 10 minutes. Designed to accommodate varying levels of technical expertise, the instructions are presented in a simple manner, making it easy for users with no prior experience in Spark or GPU setups to follow.</p><h3>II. HARDWARE &amp; SOFTWARE VERSIONS</h3><h4><strong>— Hardware</strong></h4><p>The specifications of the machine used are as follows:</p><ul><li>1 machine equipped with 2 Nvidia RTX 4090 GPUs</li><li>125 GB of RAM</li><li>31 CPU cores</li></ul><h4>— Software</h4><p>The fundemntal softwares utilized during this article and their versions are as follow:</p><ul><li>Ubuntu 22.05</li><li>Cuda 11.5</li><li>Apache Spark 3.3.0</li><li>Python 3.10.12</li></ul><p>It is evident that the latest version of Apache Spark is 3.5.2, while version 3.3.0 is being used. This choice is intentionally done due to known conflicts between the current Spark version and RAPIDS GPU acceleration, where certain tasks experience significant delays and become stuck at specific points. For instance, when using the latest version tasks are negatively affcted by A SCALAR WAS LEAKED (ID: ) error.</p><h3>III. SOFTWARE INSTALLATIONS</h3><h4>— Java 8</h4><p>Please refer to <strong>[1]</strong> under <em>Installation, Section I: Java</em> to install Java 8.</p><h4>— GPU Driver(s)</h4><p>In many cases, GPU drivers may already be installed. However, if they are not, they can be installed using the following command:</p><pre>$ ubuntu-drivers install</pre><p>To ensure the GPU driver(s) are installed, use the following command, the output must be as the one in Fiugre 1.</p><pre>$ nvidia-smi</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R-C0Y7nq47SOF5QAqk394A.png" /><figcaption><strong>Figure 1.</strong> Nvidia drivers</figcaption></figure><h4><strong>— Cuda Toolkit</strong></h4><p>NVIDIA provides an application responsible for offering a development environment to create high-performance, GPU-accelerated applications. While installing the CUDA toolkit is not necessary for Apache Spark<strong> [2]</strong>, it is beneficial for debugging purposes. If the operating system differs from the currently used one, CUDA can be installed from <strong>[3]</strong>. For Ubuntu 22.04, it can be installed using the following commands:</p><pre>$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin<br>$ sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600<br>$ wget https://developer.download.nvidia.com/compute/cuda/11.5.0/local_installers/cuda-repo-ubuntu2004-11-5-local_11.5.0-495.29.05-1_amd64.deb<br>$ sudo dpkg -i cuda-repo-ubuntu2004-11-5-local_11.5.0-495.29.05-1_amd64.deb<br>$ sudo apt-key add /var/cuda-repo-ubuntu2004-11-5-local/7fa2af80.pub<br>$ sudo apt-get update<br>$ sudo apt-get -y install cuda</pre><p>Verify the installation of NVIDIA’s CUDA Compiler (nvcc) by running the following command: (The output of this command is shown in Figure 2)</p><pre>$ nvcc --version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*tvlI7vWbell-O3AIp_ELSg.png" /><figcaption><strong>Figure 2.</strong> NVCC version</figcaption></figure><h4><strong>— Apache Spark</strong></h4><p>To install Apache Spark, please follow these commands:</p><pre>$ wget https://archive.apache.org/dist/spark/spark-3.3.0/spark-3.3.0-bin-hadoop2.tgz<br>$ tar xvf spark-3.3.0-bin-hadoop2.tgz<br>$ mkdir /opt/spark<br>$ mv spark-3.3.0-bin-hadoop2 /opt/spark<br>$ chmod -R 777 /opt/spark</pre><p>Since Apache Spark uses RAPIDS GPU acceleration, a specific JAR file must be downloaded to enable it. It can be downloaded using the following commands:</p><pre>$ cd /opt/spark/jars<br>$ wget https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark_2.12/22.06.0/rapids-4-spark_2.12-22.06.0.jar</pre><p>After that, to start the master and worker nodes from any path in the terminal, the Apache Spark bin directory needs to be added to the .bashrc file.</p><pre>$ cd<br>$ nano .bashrc</pre><p>Then please copy and paste the following to the end of the file;</p><p>export SPARK_HOME=/opt/spark</p><p>export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin</p><p>Enable the changes in bashrcfile;</p><pre>$ source .bashrc</pre><p>To forward Apache Spark logs to specific directories, use the following commands to create the destination directories:</p><pre>$ cd /opt/spark<br>$ mkdir spark-worker<br>$ mkdir spark-events<br>$ mkdir spark-warehouse<br>$ mkdir spark-temp</pre><p>They will be defined in conf files in the next steps.</p><p>After that, modify the configuration files to set the master IP, specify the new log directories, define the number of workers, and configure the number of GPUs per worker. You can achieve this by editing the following files:</p><pre>$ cd /opt/spark/conf<br>$ cp spark-defaults.conf.template spark-defaults.conf<br>$ cp spark-env.sh.template spark-env.sh</pre><p>Modify spark-defaults.conf file:</p><pre>$ nano spark-defaults.conf</pre><p>Add the following lines to the end of the file:</p><p>spark.master spark://&lt;master_ip&gt;:7077 # example: spark://11.2.4.123:7077<br>spark.local.dir file:///opt/spark/spark-temp<br>spark.eventLog.dir file:///opt/spark/spark-events<br>spark.sql.warehouse.dir file:///opt/spark/spark-warehouse</p><p>Next, download the getGpusResources.sh script before modifying the configuration files, as it is intended to be referenced within them. This script is responsible for detecting installed GPU drivers, enabling Apache Spark to utilize the available GPU resources.</p><pre>$ cd /opt/spark<br>$ mkdir mygpu<br>$ cd mygpu/<br>$ wget https://github.com/apache/spark/raw/master/examples/src/main/scripts/getGpusResources.sh</pre><p>Now, modify spark-env.sh file</p><pre>$ nano spark-env.sh</pre><p>Add the following lines to the end of the file:</p><p>SPARK_MASTER_IP=&lt;master_ip&gt;<br>SPARK_WORKER_INSTANCES=1<br>SPARK_WORKER_CORES=28<br>SPARK_WORKER_MEMORY=110G<br>SPARK_WORKER_OPTS=”-Dspark.worker.resource.gpu.amount=2 -Dspark.worker.resource.gpu.discoveryScript=/opt/spark/mygpu/getGpusResources.sh”<br>SPARK_WORKER_DIR=/opt/spark/spark-worker</p><p>Please note that from the above stated the first five configurations must be adjusted according to the available IP addresses and system resources. Please, refer to [4] for more information about Apache Spark configurations.</p><p>Before intiating Apche Spark master and workers, please ensure that the following ports are opened:</p><pre>$ sudo ufw allow 7077/tcp<br>$ sudo ufw allow 4040/tcp<br>$ sudo ufw allow 8080/tcp</pre><p>Eventually, start master and workers using the following commands:</p><pre>$ start-master.sh</pre><pre>$ start-workers.sh spark://&lt;master_ip&gt;:7077</pre><p>You can access the Apache Spark Master User Interface (UI) by opening a web browser and entering http://&lt;master_ip&gt;:8080, where &lt;master_ip&gt; is the IP address of your Spark master node. The UI is the same as the one in Figure 3, below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1YQZVgLHBxam3veV24qhfw.png" /><figcaption><strong>Figure 3.</strong> Apache Spark UI</figcaption></figure><p>In order to increase your information about Apache Spark configurations, the most important Apache Spark GPU configurations, along with other configurations, are thoroughly explained in the article “<a href="https://medium.com/turknettech/essential-apache-spark-configurations-including-gpu-acceleration-fc73df4eb0d5"><strong>Essential Apache Spark Configurations, Including GPU Acceleration</strong></a>”.</p><h3>IV. CONCLUSION</h3><p>In conclusion, this guide provides an easy and clear way to install Apache Spark on Nvidia GPUs in a standalone cluster. It gathers all the important steps in one place, helping users avoid common problems and complete the installation in just 5 to 10 minutes. The instructions are simple enough for anyone to follow, even if they have no experience with Spark or GPU setups.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W3fgQcUvwRNjt5Jn" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3>V. REFERENCES</h3><p><strong>[1]</strong> Alkhanafseh, Y. (2024). Real-Time Stream Processing Using Flink and Kafka with Scala and Zeppelin (Part 1): Installations. Accessed on [05.09.2024]. Retrieved from: <a href="https://medium.com/turknettech/real-time-stream-processing-using-flink-and-kafka-with-scala-and-zeppelin-part-1-installations-f2bd906f47fb">https://medium.com/turknettech/real-time-stream-processing-using-flink-and-kafka-with-scala-and-zeppelin-part-1-installations-f2bd906f47fb</a></p><p><strong>[2]</strong> NVIDIA Docs hub (n.d.). On-prem Cluster or Local Mode. Accessed on [05.09.2024]. Retrieved from: <a href="https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/on-premise.html">https://docs.nvidia.com/spark-rapids/user-guide/latest/getting-started/on-premise.html</a></p><p><strong>[3]</strong> NVIDIA Developer (n.d.). CUDA Toolkit 11.5 Downloads. Accessed on [05.09.2024]. Retrieved from: [llş] <a href="https://developer.nvidia.com/cuda-11-5-0-download-archive?target_os=Linux">https://developer.nvidia.com/cuda-11-5-0-download-archive?target_os=Linux</a></p><p><strong>[4]</strong> Alkhanafseh, Y, (2024). Essential Apache Spark Configurations, Including GPU Acceleration. Accessed on [05.09.2024]. Retrieved from: <a href="https://medium.com/turknettech/essential-apache-spark-configurations-including-gpu-acceleration-fc73df4eb0d5">https://medium.com/turknettech/essential-apache-spark-configurations-including-gpu-acceleration-fc73df4eb0d5</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d884350bae46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/installation-of-apache-spark-with-gpu-acceleration-d884350bae46">Installation of Apache Spark with GPU Acceleration</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Read and Write from MSSQL Using Pyspark in Python]]></title>
            <link>https://medium.com/turknettech/how-to-read-and-write-from-mssql-using-pyspark-in-python-1207c07af864?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1207c07af864</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Mon, 01 Jul 2024 11:25:17 GMT</pubDate>
            <atom:updated>2024-07-01T11:25:17.187Z</atom:updated>
            <content:encoded><![CDATA[<p>This tutorial covers MSSQL read and write operations using the partitioning technique in standalone mode with PySpark and Python. It provides detailed explanations of essential dependencies and step-by-step instructions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*unjUViubZf4I_XMegAz1Sw.png" /><figcaption>Image generated by Dall-E 3</figcaption></figure><h3><strong>I. INTRODUCTION</strong></h3><p>The volume of data generated and collected daily is growing at an unprecedented rate. One of the most traditional and reliable methods for storing this data is through Structured Query Language (SQL) databases such as Microsoft SQL Server (MSSQL). However, in SQL servers managing and processing large datasets necessitates complex methodologies, such as parallel processing, to avoid significant delays and inefficiencies. One advanced approach to read and write large volumes of data from SQL databases involves using Apache Spark. This tutorial provides a comprehensive guide on effectively reading and writing data from SQL using PySpark and Python. It includes all necessary dependencies, detailed tuning techniques to optimize performance, and a practical case study to illustrate the process. This tutorial emphasizes the importance of partitioning in these operations as well.</p><blockquote>Please note that this tutorial utilizes pyspark version 3.5.1 and Python version 3.9.5</blockquote><h3><strong>II. INSTALLING DEPENDENCIES</strong></h3><p>As stated before, this tutorial uses standalone PySpark mode, which utilizes the Python PySpark module directly. If PySpark is not installed, please install it using the following command:</p><pre>python3 -m pip install pyspark</pre><p>The following three dependencies must be installed to make Apache Spark able to connect to MSSQL.</p><pre>mkdir pyspark_mssql<br>cd pyspark_mssql<br>mkdir jars<br>cd jars</pre><pre>wget https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/7.0.0.jre8/mssql-jdbc-7.0.0.jre8.jar</pre><pre>wget https://repo1.maven.org/maven2/com/microsoft/azure/spark-mssql-connector_2.12/1.2.0/spark-mssql-connector_2.12-1.2.0.jar</pre><pre>wget https://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.49/mysql-connector-java-5.1.49.jar</pre><p>It is crucial to note that these dependency versions must align with the Apache Spark version in use; otherwise, errors such as the following may occur:</p><p>java.lang.UnsupportedClassVersionError:com/microsoft/sqlserver/jdbc/SQLServerDriver has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0</p><h3>III. APACHE SPARK IMPORTANT ATTRIBUTES</h3><p>This tutorial focuses specifically on demonstrating the PySpark read from and write to MSSQL operations. Consequently, only read() and write() variables will be discussed. These variables can be classified under three categories which are connection, read, and write.</p><ul><li><strong>Connection options</strong></li></ul><p>1. url is the JDBC connection URL, which includes details such as the database type, IP address, database name, username, password, and other connection parameters.</p><p>2. driver contains the class name of the JDBC driver.</p><p>3. dbtablecan either read an entire table or act as a query. However, according to Apache Spark documentation, specifying both query and partitionColumn options simultaneously is not permitted [1]. Therefore, to enable parallel reading of data from MSSQL, it is essential to utilize the dbtable option.</p><p>4. numPartitionsThe maximum number of partitions that can be used for parallelism in table reading and writing. It determines the maximum number of concurrent JDBC connections. If the number of partitions of the DataFrame to be written to the database exceeds this limit, it must be reduced to match the value of numPartitions.</p><ul><li><strong>Read options</strong></li></ul><p>1. fetchsizedetermines how many rows to fetch per transaction. It is not about parallel or partition <strong>[1]</strong>, it is about enhancing the performance of the query itself such as reducing logs amount and increasing the utilization of the network.</p><p>2. partitionColumn specifies the column to be used for partitioning. The selected column type must be of one of the following types: numeric, date, or timestamp. Any other data type will result in an error.</p><p>3. lowerBound specifies the begining of the stride of partitions.</p><p>4. upperBound specifies the end of the stride of paratitions.</p><p>Notice that lowerBound and upperBound are just used to decide the start and end of partition stride. They are not related to any other processes such as filtering.</p><p>The partitionColumn, lowerBound, upperBound, and numPartitions options are explained in Figure 1 below. In short, the data is divided into chunks based on the numPartitions and partitionColumn options, which define the number of chunks and the column used for partitioning, respectively. These data chunks are designed to be processed in parallel by the available resources, such as CPU, RAM, and GPU. Subsequently, the start and end points of the partitioning process for that column are determined by the lowerBound and upperBound options, which specify the initial and final values for that column. For example, if the data consists of <em>n</em> columns and <em>r</em> rows, then partitionColumn must be one of the <em>n</em> columns. The lowerBound should be set to the value of the first row, and the upperBound should be set to the value of the last row. The numPartitions must be specified by the user based on the nature of the data and the available resources. This parameter is typically determined through a trial-and-error approach to optimize performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJYvvS57V99T-Ngczx6NFw.png" /><figcaption><strong>Figure 1.</strong> Partitioning Options for Reading Data from MSSQL with PySpark in Python.</figcaption></figure><ul><li><strong>Write options</strong></li></ul><p>1. isolationLevel specifies the transaction (operations such as insertions, updates, or deletions) isolation level to use for the database connection [2]. It is responsible about defining the of how operations in one transaction are isolated from those in other transactions. It can be <em>READ_UNCOMMITTED</em><strong> </strong>which allows<strong> </strong>dirty reads, <em>READ_COMMITTED</em><strong> </strong>which prevents dirty reads, <strong>REPEATABLE_READ </strong>which prevents dirty reads and non-repeatable reads, <strong>SERIALIZABLE </strong>which prevents dirty reads, non-repeatable reads, and phantom (repeated on the same records) reads, or <strong>NONE</strong> which do not use specify isolationLevel<em>.</em></p><p>2. batchsizedetermines how many rows to insert per transaction. It is similar to fetchsize, but is used for the writing process.</p><h3><strong>IV. CASE STUDY</strong></h3><p>The following case study demonstrates the usage of the options explained in the previous section in Python. It includes reading a table from MSSQL, performing a simple transformation on it, and then writing it back to MSSQL.</p><p>The complete code can be downloaded from the following GitHub repository: <a href="https://github.com/yousef-alkhanafseh/pyspark_mssql">pyspark_mssql</a>. It also can be directly cloned using the following command:</p><pre>git clone https://github.com/yousef-alkhanafseh/pyspark_mssql.git</pre><p>The case study also includes a performance comparison between scenarios with and without the partitioning process. The details of the table to be used are presented in Table 1 below. The table has a size of 2135.398 MB, consisting of 61 columns (15 of type INT and 46 of type VARCHAR) and 3,571,543 rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*O3WlfNqHe8lOIRzElgNbMg.png" /><figcaption><strong>Table 1.</strong> Data Features and Corresponding Values.</figcaption></figure><p>The prepared script is structured into the following eight distinct parts, each serving a specific function to ensure comprehensive and efficient execution:</p><ul><li><strong>Importing the Required Models</strong></li></ul><pre>import os<br>import time<br>from pyspark import SparkConf<br>from pyspark.sql import SparkSession</pre><ul><li><strong>Defining the Paths for JAR Files</strong></li></ul><pre>jars_dir = os.path.join(os.getcwd().replace(&quot;src&quot;, &quot;&quot;), &quot;jars&quot;)<br>jdbc_driver_path = [os.path.join(jars_dir, i) for i in os.listdir(jars_dir)]<br>jdbc_driver_path = &quot;,&quot;.join(jdbc_driver_path)</pre><ul><li><strong>Initiating Spark Session</strong></li></ul><pre>start_time = time.time()<br>conf = SparkConf().setMaster(&quot;local[*]&quot;).set(&quot;spark.sql.debug.maxToStringFields&quot;, 1000) \<br>                                    .set(&quot;spark.executor.heartbeatInterval&quot;, 200000) \<br>                                    .set(&quot;spark.network.timeout&quot;, 300000) \<br>                                    .set(&quot;spark.sql.execution.arrow.pyspark.enabled&quot;, &quot;true&quot;) \<br>                                    .set(&quot;spark.jars&quot;, jdbc_driver_path) \<br>                                    .set(&quot;spark.ui.port&quot;, 4040) \<br>                                    .set(&quot;spark.driver.cores&quot;, &quot;5&quot;) \<br>                                    .set(&quot;spark.executor.cores&quot;, &quot;5&quot;) \<br>                                    .set(&quot;spark.driver.memory&quot;, &quot;1G&quot;)  \<br>                                    .set(&quot;spark.executor.memory&quot;, &quot;1G&quot;)  \<br>                                    .set(&quot;spark.executor.instances&quot;, &quot;2&quot;) \<br>                                    .setAppName(&quot;PYSPARK_MSSQL_TUTORIAL&quot;)<br><br>spark = SparkSession.builder.config(conf=conf).getOrCreate()<br>sc = spark.sparkContext<br>sc.setLogLevel(&quot;WARN&quot;)</pre><p>It is crucial to note that the following configurations must be set based on the available CPU and RAM resources:</p><p>spark.driver.cores<br>spark.executor.cores<br>spark.driver.memory<br>spark.executor.memory<br>spark.executor.instances</p><p>Since the spark.ui.port is set to 4040, the Spark UI can be accessed at <a href="http://localhost:4040/">http://localhost:4040/</a>.</p><ul><li><strong>Defining Various Variables</strong></li></ul><p>Please configure the values for db_ip, db_name, db_username, and db_password as per your specific requirements. Subsequently, optimize the values of numPartitions, fetchsize, batchsize based on the available resources inculding number of slaves, network configurations, CPU, GPU, RAM, and database features.</p><pre>db_ip = &quot;&lt;db_ip&gt;&quot;<br>db_name = &quot;&lt;db_name&gt;&quot;<br>db_username = &quot;&lt;db_username&gt;&quot;<br>db_password = &quot;&lt;db_password&gt;&quot;<br>table_name = &quot;&lt;table_name&gt;&quot;<br><br>numPartitions = 10<br>fetchsize = 3000<br>batchsize = 3000<br><br>driver = &quot;com.microsoft.sqlserver.jdbc.SQLServerDriver&quot;<br>jdbc_url = &quot;jdbc:sqlserver://{};databaseName={};user={};password={};&quot;.format(db_ip, db_name, db_username, db_password)</pre><ul><li><strong>Determining the Values for </strong><strong>upperBound and </strong><strong>lowerBound</strong></li></ul><p>This section of code is included because the lower and upper bounds are initially unknown. Moreover, the partition column is also not found in the table itself. Consequently, this code is responsible for initiating partition column as number of rows and then determining ithese values. However, if the partition column, lower and upper bounds are already known, they can be directly assigned to lower_bound and upper_bound.</p><pre>bound_query = f&quot;(SELECT ROW_NUMBER() OVER (ORDER BY username) AS row_num FROM {table_name}) as my_table&quot;<br>bound_df = spark.read \<br>    .format(&quot;jdbc&quot;) \<br>    .option(&quot;url&quot;, jdbc_url) \<br>    .option(&quot;dbtable&quot;, bound_query) \<br>    .option(&quot;driver&quot;, driver) \<br>    .load()<br><br>lower_bound = bound_df.first()[&quot;row_num&quot;]<br>upper_bound = bound_df.orderBy(&quot;row_num&quot;, ascending=False).first()[&quot;row_num&quot;]</pre><ul><li><strong>Reading Data Using Partitions</strong></li></ul><pre>main_query = f&quot;(SELECT *, ROW_NUMBER() OVER (ORDER BY username) AS row_num FROM {table_name}) as my_table&quot;<br>df = spark.read \<br>    .format(&quot;jdbc&quot;) \<br>    .option(&quot;url&quot;,jdbc_url) \<br>    .option(&quot;driver&quot;, driver) \<br>    .option(&quot;dbtable&quot;, main_query) \<br>    .option(&quot;partitionColumn&quot;, &quot;row_num&quot;) \<br>    .option(&quot;numPartitions&quot;, numPartitions) \<br>    .option(&quot;lowerBound&quot;, str(lower_bound)) \<br>    .option(&quot;upperBound&quot;, str(upper_bound)) \<br>    .option(&quot;fetchsize&quot;, fetchsize) \<br>    .load()</pre><ul><li><strong>Applying Simple Transformations with </strong><strong>groupBy() and </strong><strong>orderBy()</strong></li></ul><p>The data is grouped by the username column, and the count of each user is calculated. The data is then ordered in descending order based on the count column, resulting in a dataset with only two columns.</p><pre>final_df = df.groupBy(&quot;username&quot;).count()<br>final_df = final_df.orderBy(&quot;count&quot;, ascending=False)</pre><ul><li><strong>Writing Data Back to MSSQL</strong></li></ul><p>As stated previously, the number of partitions of the data must match the value specified in the numPartitions option. Accordingly, the script below first checks and then applies this condition. It uses the coalesce() function to reduce the number of partitions and the repartition() function to increase the number of partitions, as needed.</p><pre>#Partition modification<br>if int(final_df.rdd.getNumPartitions()) &gt; numPartitions:<br>  final_df = final_df.coalesce(numPartitions)<br>  print(&quot;partitions are reduced&quot;)<br>elif int(final_df.rdd.getNumPartitions()) &lt; numPartitions:<br>  final_df= final_df.repartition(numPartitions)<br>  print(&quot;partitions are increased&quot;)<br>else:<br>  print(&quot;partitions are kept the same&quot;)<br><br># Write<br>final_df.write \<br>        .format(&quot;jdbc&quot;) \<br>        .option(&quot;driver&quot;, driver) \<br>        .option(&quot;url&quot;, jdbc_url.replace(db_name, &quot;TnTemp&quot;)) \<br>        .option(&quot;dbtable&quot;, table_name + &quot;_USER_GROUPED_TEMP&quot;) \<br>        .option(&quot;numPartitions&quot;, numPartitions) \<br>        .option(&quot;isolationLevel&quot;, &quot;NONE&quot;) \<br>        .option(&quot;rewriteBatchedStatements&quot;, &quot;true&quot;) \<br>        .option(&quot;batchsize&quot;, batchsize) \<br>        .mode(&quot;overwrite&quot;) \<br>        .save()<br><br>end_time = time.time()<br>print(&quot;Execution time with partitioning: {} s&quot;. format(round(end_time - start_time, 4)))</pre><p>The written data consists of two columns, username and count, and includes 984,869 rows. A screenshot from MSSQL illustrating this data is shown in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*flGKwk_mJIyZqbyx4kHsNg.png" /><figcaption>Figure 2.</figcaption></figure><p>As specified in Figure 3, when partitioning is employed, there are 10 parallel processes executing concurrently, as depicted in Figure 3(a). Conversely, when partitioning is not utilized during database reading, only a single executor is active, thereby limiting the parallelism process. This case is clearly illustrated in Figure 3(b).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r175T57mO58yEBzJNqs2pg.png" /><figcaption><strong>Figure 3.</strong> Comparison of Job Execution with and without Partitioning Technique in Apache Spark</figcaption></figure><p>The execution time with partitioning is 127.1743 seconds, compared to 312.8392 seconds without partitioning. This demonstrates a reduction in execution time by approximately 246%. It is important to note that this enhancement ratio is observed with a data size of 2135.398 MB, and the ratio increases as the data size grows.</p><h3><strong>V. CONCLUSION</strong></h3><p>In conclusion, this tutorial covered the essential variables and partitioning technique used in reading from and writing to MSSQL using PySpark. Additionally, it included a case study demonstrating these operations. The case study compared the execution times of operations with and without partitioning, revealing a reduction of approximately 246% in execution time when partitioning was employed. For future work, various scenarios and different dataset types can be explored, and more transformations can be applied to the dataset to further analyze the effects of partitioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wq0rP9RT1uUO05F0P_yu9g.gif" /></figure><h3><strong>VI. REFERENCES</strong></h3><p><strong>[1]</strong> Apache Spark(n.d). JDBC To Other Databases. Accessed on [15.06.2024]. Retrieved from: <a href="https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html">https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html</a></p><p><strong>[2]</strong> Microsoft (2022). Understanding isolation levels. Accessed on [19.06.2024]. Retrieved from: <a href="https://learn.microsoft.com/en-us/sql/connect/jdbc/understanding-isolation-levels?view=sql-server-ver16">https://learn.microsoft.com/en-us/sql/connect/jdbc/understanding-isolation-levels?view=sql-server-ver16</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1207c07af864" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/how-to-read-and-write-from-mssql-using-pyspark-in-python-1207c07af864">How to Read and Write from MSSQL Using Pyspark in Python</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Overview of Apache NiFi]]></title>
            <link>https://medium.com/turknettech/an-overview-of-apache-nifi-44819186674c?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/44819186674c</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 08:40:13 GMT</pubDate>
            <atom:updated>2024-04-02T08:40:13.498Z</atom:updated>
            <content:encoded><![CDATA[<p>The capabilities of Apache NiFi for managing data flows are examined, providing an in-depth exploration of its essential concepts, features, services, and user interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IB0je0Fk3Tia_i7DJTJBSA.jpeg" /><figcaption>Image Generated by DALL-E 3 Inspired by The Logo of Apache NiFi.</figcaption></figure><h3>I. INTRODUCTION</h3><p>In recent years, the data processing tasks within various organizations have become increasingly complex , necessitating the use of advanced data flow tools. Consequently, this article aims to provide a comprehensive overview of Apache NiFi inculding its definition, performance expectations, characteristics, and features. Additionally, the core concepts of Apache NiFi are explained in detail, with a significant emphasis on its User Interface (UI). This ensures that all important information about Apache NiFi is thoroughly covered. By using Apache NiFi, several advantages can be obtained, such as reducing system failures, processing data simultaneously, eliminating data processing boundaries, enhancing data management, and increasing security <strong>[1]</strong>.</p><blockquote><em>Please refer to the “</em><a href="https://medium.com/@alkhanafseh/installation-of-apache-nifi-2856fca5bbdc">Installation of Apache NiFi</a><em>” article </em><strong><em>[2]</em></strong><em> for a complete installation guide of latest version of Apache NiFi “1.25.0”.</em></blockquote><blockquote><em>Please refer to the “</em><a href="https://medium.com/@alkhanafseh/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f">Executing Python Scripts and SQL Quries In Apache NiFi</a><em>” article </em><strong><em>[3]</em></strong><em> for a detailed guide on how to run Python scripts and SQL queries within Apache NiFi.</em></blockquote><h3>II. UNDERSTANDING APACHE NIFI: DEFINITION, PERFORMANCE EXPECTATIONS, CHARACTERISTICS, AND FEATURES.</h3><p>Apache NiFi is a highly valuable data flow system designed to automate and schedule various data operations across different processes <strong>[1]</strong>. It is capable of handling both simple and complex systems effectively. The performance of Apache NiFi, in terms of IO, CPU, and RAM utilization, is spectacular as well. Apache NiFi has numerous characteristics and features <strong>[1]</strong>, which can be highlighted in the following points:</p><ul><li>Flow management</li><li>Ease of use</li><li>Security</li><li>Extensible architecture</li><li>and flexible scaling model</li></ul><h3>III. THE CORE CONCEPTS OF APACHE NIFI</h3><p>The most important concepts realted to Apache NiFi are DataFlow Manager (DFM), FlowFile, Bulletin, Processor, Connection, Controller Service, Flow Controller, and Process Group.<strong> </strong>The significance of each concept is as follows <strong>[4]</strong>:</p><ul><li><strong>DFM</strong></li></ul><p>In simple words, a DFM is the user who has different permissions and complete management of Apache NiFi.</p><ul><li><strong>FlowFile</strong></li></ul><p>The FlowFile concept refers to each object moving through the system. It has two main parts: attributes and content. The first one stores metadata about the FlowFile, such as its filename, size, etc. The standard attributes that each FlowFile contains are <em>Universally Unique Identifier (UUID)</em>, used to distinguish the FlowFile; <em>filename</em>, used to represent the name of the output of the data when it is written to some path; and <em>path</em>, used to define the path of the filename. On the other hand, the content part is responsible for storing the actual data of the FlowFile.</p><ul><li><strong>Bulletin</strong></li></ul><p>It can be considered a service reporter tool that is available for each service. It provides critical information such as rolling statistics, current status, and severity levels such as Debug, Info, Warning, and Error, giving insights into the current situation of the component.</p><ul><li><strong>Processor</strong></li></ul><p>A Processor, as the name suggests, is responsible for performing the actual work, such as data routing and transformation, as it has direct access to the attributes and the content of the given FlowFile. For example, processor has the ability to read data files, execute different software languages such as Python scripts, and executing different SQL quries. An example of Apache NiFi processor is shown in Figure 1.</p><ul><li><strong>Connection</strong></li></ul><p>A Connection can be regarded as the link between different processors, see Figure 1. It manages the interaction between the processors by establishing queues. A queue in Apache NiFi refers to a buffer that has the ability to store FlowFiles. Therefore, the output of one processor, the FlowFile, is stored in a queue until the other processor is ready to process it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*hEoe0hYoPe-PGAl_Pjbo2g.png" /><figcaption><strong>Figure 1.</strong> Detailed View of a Processor and Connection in Apache NiFi.</figcaption></figure><ul><li><strong>Controller Service</strong></li></ul><p>When a Controller Service is created, it will automatically start up every time Apache NiFi is initiated. Several essential services can be specified in the Controller Service section. For instance, DBCPConnectionPool and CSVRecordSetWriter are utilized to establish connections between various database systems and Apache NiFi, and to enable writing CSV files to the local system, respectively. Once defined, these services can be utilized by any processors.</p><ul><li><strong>Flow Controller</strong></li></ul><p>The Flow Controller, in short, manages the threads used by different processes. As a result, it can dynamically adjust the number of threads used for each flow, increasing or decreasing as necessary.</p><ul><li><strong>Process Group</strong></li></ul><p>Grouping a set of processors and their connections is called a Process Group. The input and output data of this Process Group can be managed by Input Ports and Output Ports, respectively. An example of a Porcess Group is obviously shown in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*fhS7mYdkjHFi-PcrITAFYQ.png" /><figcaption><strong>Figure 2. </strong>Detailed View of a Process Group in Apache NiFi.</figcaption></figure><h3>IV. APACHE NIFI USER INTERFACE</h3><p>Specifically, Apache NiFi UI comprises six main elements: the components toolbar, status bar, navigation palette, operation palette, global menu, and canvas <strong>[6]</strong>. The example of the Apache NiFi interface is clearly illustrated in Figure 3 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GAhZRq7zui6xb5Jqnc7xVg.png" /><figcaption><strong>Figure 3.</strong> Apache NiFi Main Page.</figcaption></figure><h4>Components Toolbar</h4><p>The components toolbar, as shown in Figure 4, comprises seven main elements: Processor, Input &amp; Output ports, Process Group, Remote Process Group, Funnel, Template, and Label. The first three components have been discussed in the preceding section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0SP58bwydDgK16jK-FYYbA.png" /><figcaption><strong>Figure 4.</strong> Components Toolbar in Apache NiFi.</figcaption></figure><ul><li><strong>Funnel</strong></li></ul><p>Generally, Funnel is utilized when data from several Connections need to be merged into a single Connection. When placed on the canvas, it appears as the symbol shown in Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*kXxzGGC2vSe1sQw2wUQaZQ.png" /><figcaption><strong>Figure 5.</strong> Funnel Symbol in Apache NiFi UI.</figcaption></figure><ul><li><strong>Remote Process Group</strong></li></ul><p>A Remote Process Group in NiFi is used to facilitate the transfer of data between different NiFi systems.</p><ul><li><strong>Template</strong></li></ul><p>Apache NiFi enables DFMs to create templates with unique names for their dataflows. These templates can be found inside the Template icon in the Components Toolbar. This feature allows them to easily reuse specific parts of the flow by simply selecting the corresponding template they have created.</p><ul><li><strong>Label</strong></li></ul><p>In summary, Apache NiFi offers the capability to add extra labels to the canvas, enabling users to incorporate additional information, comments, or advices. This can be achieved by dragging the label icon onto the canvas. Once positioned on the canvas, it will resemble the shape displayed in Figure 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*8aSFicIgsIven7caJvlpPQ.png" /><figcaption><strong>Figure 6.</strong> Label in Apache NiFi</figcaption></figure><h4>Status Bar</h4><p>As depicted in Figure 7, the Status Bar plays a crucial role in displaying vital information about the currently running dataflow. It consists of 15 different components. The first one is <em>active threads</em>, which is the number of total threads being used by the current dataflow. The second one is <em>total queued data</em>, which represents the amount of data waiting to be processed by other processors. The third and fourth components are <em>transmitting and not transmitting remote process groups</em>, which show the count of the currently running groups for each state, respectively. The next four components are related to the status of components, indicating if they are <em>running</em>, <em>stopped</em>, <em>invalid</em>, or <em>disabled</em>, respectively. On the other hand, the subsequent five components are responsible for version states. The <em>Up to date versioned process group</em> shows the latest flow version. Moreover, the <em>Locally modified versioned process group</em> indicates the number of local changes that have been made. The <em>Stale versioned process group</em> component informs if there is a new available version of the flow. The <em>Locally modified and Stale versioned process group</em> occurs when the combination of the previous two components occurs simultaneously. The <em>Sync failure versioned process group</em> indicates if there is a conflict in synchronizing the flow with the registry [5]. Additionally, the <em>Last refresh time</em> component displays the last refresh time of the current dataflow. Finally, the <em>search bar</em> is used to make general searches in Apache NiFi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_vCTx7iJ0v6zTCDqJVOz0Q.png" /><figcaption><strong>Figure 7.</strong> Detailed View of Status Bar in Apache NiFi.</figcaption></figure><h4>Navigate palette, Bird’s eye view, Breadcrumbs, and Canvas</h4><p>The Navigate Palette enables DFMs to navigate around the canvas and to zoom in and out. The Bird’s Eye View is a part of the Navigate Palette that allows the required components to be displayed on the canvas within one frame. The Breadcrumbs section shows the depth of the current dataflow and process groups, which can be used to navigate around them. Eventually, Canvas is the home screen of Apache NiFi which is supposed to carry different services or porcessors.</p><h4>Operate Palette</h4><p>The Operate Palette is home to the main buttons of the flow, see Figure 8, providing DFMs flexibility to execute different actions. These include configuring system settings, such as modifying general flow information or adding new Controller Services. Additionally, the Operate Palette allows for a range of operations on the current flow, like enabling, disabling, running, stopping, copying, pasting, grouping, changing colors, and deleting components. It also features two buttons dedicated to creating a template for the current flow or uploading a previously saved template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*YlWaKPcSPMlld_fCzeSlaw.png" /><figcaption><strong>Figure 8.</strong> Detailed View of Operate Palette in Apache NiFi.</figcaption></figure><h4>Global Menu</h4><p>Specifically, the Global Menu is employed to make modifications to the existing components in the dataflow. It also facilitates access to the flow configuration history, node status history, controller settings, bulletin board, and available templates. Additionally, the “Help” section within the Global Menu provides further information about the components. Details regarding the current version of Apache NiFi can be found in the “About” section as well. The Global Menu of Apache NiFi is shown in Figure 9, below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*DfwyGZBnzOcIHXV_2punOA.png" /><figcaption><strong>Figure 9.</strong> Global Menu in Apache NiFi .</figcaption></figure><h3>V. CONCLUSION</h3><p>In conclusion, the data flow management capabilities of Apache NiFi are explained through a comprehensive understanding of its essential definitions, concepts, features, services, and UI, all of which are presented with clear explanations and illustrative examples. Each component of the UI is thoroughly explained in detail. Consequently, this article can be regarded as the first step for learning Apache NiFi.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Y-W6EVF2acXimZUzq1QCA.gif" /></figure><h3>VI. REFERENCES</h3><p><strong>[1]</strong> Apache NiFi Team (n.d). Apache NiFi Overview. Accessed on [15.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/html/overview.html">https://nifi.apache.org/docs/nifi-docs/html/overview.html</a></p><p><strong>[2] </strong>Alkhanafseh, Y. (2024). Installation of Apache NiFi. Accessed on [29.03.2024]. Retrieved from: <a href="https://medium.com/@alkhanafseh/installation-of-apache-nifi-2856fca5bbdc">https://medium.com/@alkhanafseh/installation-of-apache-nifi-2856fca5bbdc</a></p><p><strong>[3] </strong>Alkhanafseh, Y. (2024). Executing Python Scripts and SQL Quries In Apache NiFi. Accessed on [29.03.2024]. Retrieved from: <a href="https://medium.com/@alkhanafseh/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f">https://medium.com/@alkhanafseh/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f</a></p><p><strong>[4]</strong> Apache NiFi Team (n.d). Apache NiFi User Guide. Accessed on [13.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html">https://nifi.apache.org/docs/nifi-docs/html/user-guide.html</a></p><p><strong>[5]</strong> Trainingcred (n.d). Apache Nifi UI Components. Accessed on [14.03.2024]. Retrieved from: <a href="https://trainingcred.com/apache-nifi-ui-components">https://trainingcred.com/apache-nifi-ui-components</a></p><p><strong>[6]</strong> Cloudera (n.d). USING APACHE NIFI. Accessed on [15.03.2024]. Retrieved from: <a href="https://docs.cloudera.com/cfm/2.0.4/nifi-user-guide/topics/nifi-managing_local_changes.html">https://docs.cloudera.com/cfm/2.0.4/nifi-user-guide/topics/nifi-managing_local_changes.html</a>?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44819186674c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/an-overview-of-apache-nifi-44819186674c">An Overview of Apache NiFi</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installation of Apache NiFi]]></title>
            <link>https://medium.com/turknettech/installation-of-apache-nifi-2856fca5bbdc?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2856fca5bbdc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[apache-nifi]]></category>
            <category><![CDATA[dataflow]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 08:39:55 GMT</pubDate>
            <atom:updated>2024-04-02T08:39:55.712Z</atom:updated>
            <content:encoded><![CDATA[<p>Step-by-Step guide for installing Apache NiFi 1.25.0 and other essential dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ToUsAOBZ0iPmTtJPWzor0g.png" /><figcaption>Image Generated by DALL-E 3 Illuserating Apache NiFi Installation Process.</figcaption></figure><h3>I. INTRODUCTION</h3><p>Installing software can often be a challenging and complex process, fraught with potential difficulties and conflicts. This is particularly true when step-by-step instructions are lacking. Recognizing the importance of clear and detailed guidance, this tutorial is designed to provide an extensive installation guide for Apache NiFi 1.25.0, covering all necessary prerequisites such as Java, as well as essential configurations and security settings. It’s important to note that this tutorial focuses only on the standalone mode of Apache NiFi.</p><h3>II. APACHE NIFI</h3><p>Apache NiFi can be considered as a workflow tool equipped with an interactive user interface. It has high ability to automate cybersecurity, observability, event streams, and generative AI data pipelines and distribution for thousands of companies worldwide across every industry <strong>[1]</strong>. The interface of Apache NiFi provides several useful information related to each processecor such as Queued, In, read/write, and Out quantities. Refer to the <strong>“</strong><a href="https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c"><em>An Overview of Apache NiFi</em></a><strong>” </strong>article <strong>[2] </strong>for more information about Apache NiFi. Additionally, Apache NiFi is highly optimized for efficient utilization of IO, CPU, and RAM resources, ensuring optimal performance. It offers the flexibility to operate in either standalone or clustered modes, providing manageability, availability, extensibility, and tolerance <strong>[3]</strong>.</p><h3>III. SYSTEM REQUIRMENTS</h3><p>Apache NiFi, generally, can be installed on only one node or machine and at the same time it can be installed on a cluster that consists of master node(s) and slave node(s) so that it can handle the pipleines more tolurent and more effictively. Based on that, the architecture of Apache NiFi is completely depending on its goal. If the pipleines that will run on Apache NiFi are simple and small, then standalone mode is enough, otherwise, cluster mode must be selected. Disk size is also an important feature that must be taken into consideration as the running pipleins will use the machine disk <strong>[4]</strong>.</p><p>The minimum requirments of Apache NiFi are as follow:</p><ul><li><strong>Java</strong>:</li></ul><p>Java 8 or Java 11</p><ul><li><strong>Web Browsers:</strong></li></ul><p>Microsoft Edge, Mozilla FireFox, Google Chrome, or Safari</p><ul><li><strong>Operating Systems:</strong></li></ul><p>Linux, Unix, Windows, or macOS</p><p>In this tutorial, as a Java; Java 8 will be installed, as a web broswer Google Chrom will be used, and as an operating system Linux Ubunutu will be used. Aditionally, some other packages such as unzip must be installed.</p><h3>IV. INSTALLATIONS</h3><p>A thorough explanation of the installation processes for Java8, the Unzip package, and Apache NiFi 1.25.0 is supposed to be provided in this section.</p><h4><strong>A. Java 8</strong></h4><p>The installation process for Java involves the following sequential steps <strong>[5]</strong></p><p>1. <strong>Update the package lists</strong></p><pre>sudo apt-get update</pre><p>2. <strong>Install Java 8</strong></p><pre>sudo apt install openjdk-8-jdk</pre><p>3. <strong>Prepare the enviroment variables for Java 8</strong></p><p>3.1. Go to main directory and open .bashrc file</p><pre>cd &amp;&amp; nano .bashrc</pre><p>3.2. Put the following lines at the end of the file, you can go to the end of the file using (alt+/) shortcut.</p><p><em>export JAVA_HOME=/lib/jvm/java-8-openjdk-amd64<br>export PATH=$PATH:$JAVA_HOME/bin</em></p><p>3.3. Apply changes to .basrch file</p><pre>source .bashrc</pre><p>3.4. Ensure that the JAVA_HOME variable is correctly configured</p><pre>echo $JAVA_HOME</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*_3JbETFAbU68P6Q0.png" /><figcaption><strong>Figure 2.</strong> Obtained result from “echo $JAVA_HOME” command.</figcaption></figure><p>4. Ensure that you have the correct Java version</p><pre>java -version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/0*1g8TS15p8ZT_eoa7.png" /><figcaption><strong>Figure 3.</strong> Obtained result from “java -version” command.</figcaption></figure><h4>B. Unzip:</h4><p>The installation of unzip package can be easily achived using the following two commands:</p><ol><li><strong>Install Unzip package</strong></li></ol><pre>sudo apt install unzip</pre><p><strong>2. Ensure that you have the installed Unzip package correctly</strong></p><pre>unzip --version</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*h7ftQnRCGOAQd_yZe_K7iw.png" /><figcaption><strong>Figure 4.</strong> Obtained result from “unzip — version” command.</figcaption></figure><h4>C. Apache NiFi 1.25.0</h4><p><strong>1. </strong>Install ِA]pache NiFi 1.25.0</p><pre>cd &amp;&amp; wget https://dlcdn.apache.org/nifi/1.25.0/nifi-1.25.0-bin.zip</pre><p>2. Uncompress the installed Apache NiFi 1.25.0 package</p><pre><br>unzip nifi-1.25.0-bin.zip</pre><p>3. Move Apache NiFi folders to /opt/nifi</p><p>3.1<strong> </strong>Make the directory of <em>/opt/nifi</em></p><pre>mkdir -p /opt/nifi</pre><p>3.2 Move all Apache NiFi folders to <em>/opt/nifi </em>directory</p><pre>sudo cp -r nifi-1.25.0/* /opt/nifi/</pre><p>4. Modify the permissions for the Apache NiFi directory and its bin subdirectory to ensure proper access control.</p><p>4.1 Change the permissions of sudo for <strong>/opt/nifi</strong> folder</p><pre>sudo chmod -R 777 /opt/nifi/</pre><p>4.2 Change the permisions for <strong>/opt/nifi/bin/nifi.sh</strong> file</p><pre>sudo chmod u+x /opt/nifi/bin/nifi.sh</pre><p>4.3 Enhance the accessibility of the <em>nifi.sh</em> file by enabling its direct execution using the nifi command. It can be achived by appending the Apache NiFi bin directory to the PATH variable in the .bashrc file</p><p>4.3.1 Go to main directroy and open <strong>.bashrc</strong> file</p><pre>cd &amp;&amp; nano .bashrc</pre><p>4.3.2 Add the following line to the end of <strong>.bashrc</strong> file</p><pre>alias nifi=&quot;/opt/nifi/bin/nifi.sh&quot;</pre><p>4.3.3 Apply changes to .basrch file</p><pre>soruce .bashrc</pre><p>5. Set the JAVA_HOME environment variable within the Apache NiFi enviroment file.</p><p>5.1 Go to Apache NiFi bin Directory</p><pre>cd /opt/nifi/bin</pre><p>5.2 Update the nifi-env.sh file by uncommenting the <em>export JAVA_HOME</em> line and setting its value to /lib/jvm/java-8-openjdk-amd64.</p><pre>nano nifi-env.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2gjIHkWGjFZvunq5-2m6hQ.png" /><figcaption><strong>Figure 5. </strong>Defining “JAVA_HOME” inside “nifi-env.sh” file</figcaption></figure><p>6. Open port 8443 to allow the Apache NiFi interface to utilize it.</p><pre>sudo ufw allow 8443/tcp</pre><p>7. Verify that the nifi command is functioning properly.</p><pre>nifi status</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0V7k4Yv3Zm9WBjaDZMI2VA.png" /><figcaption><strong>Figure 6.</strong> Obtained result from “nifi status” command.</figcaption></figure><p>8. Generate a new user and password. This is important in order to avoid the generated user and password by Apache NiFi which lists them under logs/nifi-app.log</p><p>8.1 Go to Apache NiFi bin directory</p><pre>cd nifi-1.25.0/bin</pre><p>8.2 Run the following command where:</p><p>admin is the username and admin 111222333 is the password.</p><p>Feel free to change them based on your requiremnts</p><pre>./nifi.sh set-single-user-credentials admin admin111222333</pre><p>9. Start Apache NiFi service</p><pre>nifi start</pre><p>10. Check Apache NiFi status</p><pre>nifi status</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z41vgy4qswCGm1s7DzOWiA.png" /><figcaption><strong>Figure 7. </strong>Obtained result from “nifi status” command.</figcaption></figure><p>11. Access the Apache NiFi interface by pasting the following link into Google Chrome: <a href="https://127.0.0.1:8443/nifi/">https://127.0.0.1:8443/nifi/</a>. The page must look as the one in Figure 8. Then, enter the username and password created in step 8.2. In this case, they are:</p><p><strong>* username</strong>: <em>admin</em></p><p><strong>* password</strong>: <em>admin111222333</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JdPrgEJWqlypZlV5rolk_A.png" /><figcaption><strong>Figure 8.</strong> Apache NiFi login interface</figcaption></figure><h3>V. CONCLUSION</h3><p>In conclusion, this tutorial has been carefully structured to provide a detailed, step-by-step guide for the installation of Apache NiFi and its essential accompanying software. Additionally, the necessary configurations and security permissions have been explained to ensure a smooth and secure setup. Furthermore, we have provided an overview of Apache NiFi, including its performance and minimum requirements, to offer a comprehensive understanding of this powerful workflow management tool. For more information about Apache NiFi, please refer to the “<a href="https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c"><em>An Overview of Apache NiFi</em></a>” article <strong>[2]</strong>. By following this tutorial, any user can confidently establish Apache NiFi and be equipped with a solid foundation of knowledge about the system. For case studies that involve executing Python scripts and SQL queries, please refer to the “<a href="https://medium.com/@alkhanafseh/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f"><em>Executing Python Scripts and SQL Quries In Apache NiFi</em></a>” article <strong>[6]</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*8VBm-wnwimxvOzhWOzFNzw.gif" /></figure><h3>VI. REFRENCES</h3><p><strong>[1]</strong> Apache NiFi (n.d.). An easy to use, powerful, and reliable system to process and distribute data. Accessed on [14.03.2024]. Retrieved from: <a href="https://nifi.apache.org/">https://nifi.apache.org/</a></p><p><strong>[2]</strong> Alkhanafseh, Y. (2024). An Overview of Apache NiFi. Accessed on [17.03.2024]. Retrieved from: <a href="https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c">https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c</a></p><p><strong>[3]</strong> Apache NiFi Team (n.d.). Apache NiFi Overview. Accessed on [18.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/html/overview.html">https://nifi.apache.org/docs/nifi-docs/html/overview.html</a></p><p><strong>[4]</strong> Apache NiFi Team (n.d.). NiFi System Administrator’s Guide. Accessed on [17.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_requirements">https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_requirements</a></p><p><strong>[5]</strong> Alkhanafseh, Y. (2024). Real-Time Stream Processing Using Flink and Kafka with Scala and Zeppelin (Part 1): Installations. Accessed on [19.03.2024]. Retrieved from: <a href="https://medium.com/turknettech/real-time-stream-processing-using-flink-and-kafka-with-scala-and-zeppelin-part-1-installations-f2bd906f47fb">https://medium.com/turknettech/real-time-stream-processing-using-flink-and-kafka-with-scala-and-zeppelin-part-1-installations-f2bd906f47fb</a></p><p><strong>[6]</strong> Alkhanafseh, Y. (2024). Executing Python Scripts and SQL Quries In Apache NiFi. Accessed on [19.03.2024]. Retrieved from: <a href="https://medium.com/@alkhanafseh/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f">https://medium.com/@alkhanafseh/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2856fca5bbdc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/installation-of-apache-nifi-2856fca5bbdc">Installation of Apache NiFi</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Executing Python Scripts and SQL Quries In Apache NiFi]]></title>
            <link>https://medium.com/turknettech/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f?source=rss-b06f7ab6709b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4eb49945846f</guid>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[apache-nifi]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Yousef Alkhanafseh]]></dc:creator>
            <pubDate>Tue, 02 Apr 2024 08:39:41 GMT</pubDate>
            <atom:updated>2024-04-02T08:39:41.200Z</atom:updated>
            <content:encoded><![CDATA[<p>It discovers how to execute Python scripts and SQL queries in Apache NiFi , covering essential steps and configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rJWZoyzjrDGQhr9NBklMqg.png" /><figcaption>Image generated by DALL-E 3.</figcaption></figure><h3>I. INTRODUCTION</h3><p>Generally, the field of data science is rich with a variety of tools and programming languages. Among these, Python and SQL are two of the most commonly employed in data science roles <strong>[1]</strong>. In addition, one of the most powerful dataflow tools is Apache NiFi. Consequently, this article is dedicated to providing a detailed demonstration of executing Python scripts and SQL queries within the Apache NiFi. The article also shows their logging methods, providing insight into the process of capturing and recording data in Apache NiFi. It thoroughly outlines the necessary prerequisites, step-by-step execution processes, and relevant information for each service, ensuring a smooth and easy understanding for the reader.</p><h3>II. PRE-PREPARATIONS</h3><p>1.Refer to “<a href="https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c"><em>An Overview of Apache NiFi</em></a>” article [2] for more information about Apach NiFi and its crictical concepts. [Optional]</p><p>2. Follow the steps of “<a href="https://medium.com/@alkhanafseh/installation-of-apache-nifi-2856fca5bbdc"><em>Installation of Apache NiFi</em></a>” article [3] to install Apache NiFi on your machine.</p><p>3. In the /opt/nifi directory, create the following subdirectories: drivers, python/scripts, python/logs, and sql/logs.</p><p>3.1 Go to /opt/nifi/ folder</p><pre>cd /opt/nifi/</pre><p>3.2 Create the needed directories using the following command</p><pre>mkdir drivers &amp;&amp; mkdir -p python/scripts &amp;&amp; mkdir -p python/logs &amp;&amp; mkdir -p sql/logs</pre><p>4. Create a simple python script that only prints “HELLO WORLD!”</p><p>4.1 Ensure that the ‘nano’ text editor is installed on your system; if it is not, proceed with its installation</p><pre>sudo apt-get install nano</pre><p>4.2 Create a file named test.py and insert the following code:</p><pre>nano /opt/nifi/python/scripts/test.py</pre><pre>print(&quot;HELLO WORLD!&quot;)</pre><p>5. Download Microsoft JDBC Driver to enable the connection between Apache NiFi and SQL Server</p><p>5.1 Go to <em>/opt/nifi/drivers</em> directory.</p><pre>cd /opt/nifi/drivers</pre><p>5.2 Download the <a href="https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.2.0.jre11/mssql-jdbc-12.2.0.jre11.jar">dbc-12.2.0.jre11.jar</a> dependency</p><pre>wget https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.2.0.jre11/mssql-jdbc-12.2.0.jre11.jar</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*3nh7FYvcihBmhC8RbqgEpw.png" /></figure><p>6. Start Apache NiFi service.</p><pre>nifi start</pre><h3>III. PREPARING APACHE NIFI CONTROLLER SERVICES</h3><h4><strong>* DBCPConnectionPool</strong></h4><p>It is one of the controller services provided by Apache NiFi, used to establish a connection between Apache NiFi processors and the specified database <strong>[4]</strong>. Throughout this tutorial, the utilization of the DBCPConnectionPool for establishing a connection between Apache NiFi and MSSQL is extensively explained.</p><p>Follow the steps below to create and configure the DBCPConnectionPool controller service:</p><ol><li>Go to <a href="https://127.0.0.1:8443/nifi/">https://127.0.0.1:8443/nifi/</a> and sign in.</li><li>Expand the Operate box, then, expand the ‘Operate’ panel, then click on the settings icon.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ZPDVAP8-9zzSnskbOyPqw.png" /></figure><p>3. Alter the name of the Process Group, apply the changes, and navigate to the ‘Controller Services’ section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MIQQPfmMxtu39Rnxqav4Fw.png" /></figure><p>4. Click on the ‘+’ symbol on the right, enter ‘DBCPConnectionPool’ in the search box, select it, and then click the ‘Add’ button below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0qmjTI90yAp_gfJvMAe4Q.png" /></figure><p>5. Click on the settings symbol of the DBCPConnectionPool, navigate to the ‘SETTINGS’ section and rename it to ‘MSSQLConnection’, then proceed to the ‘PROPERTIES’ section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzoyWIu0W5de0_gWtY5VRw.png" /></figure><p>6. Adjust the following parameters as specified below</p><p><em>— </em><strong><em>Database Connection URL:</em></strong><em><br></em><strong> </strong>jdbc:sqlserver://<strong>db_ip</strong>;port=1433;database=<strong>db_name</strong>;encrypt=true;trustServerCertificate=true</p><p>Note: Substitute ‘db_ip’ and ‘db_name’ with the host IP of the database and the database name, respectively.</p><p><strong><em>— Database Driver Class Name:</em><br></strong>com.microsoft.sqlserver.jdbc.SQLServerDriver</p><p><em>— </em><strong><em>Database Driver Location(s):</em></strong><em><br></em>/opt/nifi/drivers/mssql-jdbc-12.2.0.jre11.jar</p><p><em>— </em><strong><em>Database User:<br></em></strong>Change it with the username that is able to connect to the database.<br>your_username</p><p><em>— </em><strong><em>Password:<br></em></strong>Change it with the password of the username above.<br>your_password</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HPiqG0lhDpIAI5XRGDw_LA.png" /></figure><p>7. Activate the created DBCPConnectionPool service by clicking on the ‘Enable’ icon, then click on the ‘ENABLE’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NARfAVIstivwy3P5T_v2Kg.png" /></figure><p>It should execute without encountering any errors, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QurQR49kfuvJcHLb9fgjw.png" /></figure><h4><strong>* CSVRecordSetWriter</strong></h4><p>Another important Controller Service is CSVRecordSetWriter. As its name suggests, this service is responsible for enabiling the dataflow to write output files in CSV format <strong>[5]</strong>.</p><p>During this tutorial, the usage of CSVRecordSetWriter is discussed step-by-step. The CSVRecordSetWriter Controller Service can be added by following the steps below:</p><ol><li>After navegating to the settings of Operator and clicking on the ‘Controller Services’ section, click on the ‘+’ symbol on the right, type ‘CSVRecordSetWriter’ into the search box, select it, and then click on the ‘Add’ button below.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6ly_f8tb43S2uzy2sOk-A.png" /></figure><p>2. Activate the service by clicking on the ‘Enable’ icon, then click on the ‘ENABLE’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uh78BA0e9mJP4ogQ9Ytt6A.png" /></figure><p>It is expected to run without encountering any errors, as illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hD2zwwmxd07Bke4nc-kdFg.png" /></figure><h3>IV. APACHE NIFI PROCESSORS</h3><p>This section provides detailed explanations of six key Apache NiFi processors which are ExecuteSQL, ExecuteSQLRecord, PutFile, UpdateAttribute, LOGAttribute, LOGMessage.<strong> </strong>All the processors mentioned, except for ExecuteSQL, are utilized in the case study section.</p><ul><li><strong>ExecuteSQL</strong></li></ul><p>It is a processor used in Apache NiFi to execute SQL queries. The obtained result can only be in the specified format from Apache NiFi itself, which is Avro format <strong>[6]</strong>. Therefore, this type of processor is usually preferred when the output file format is not critical.</p><ul><li><strong>ExecuteSQLRecord</strong></li></ul><p>Similar to ExecuteSQL, it can be considered a method used to execute SQL queries. Nevertheless, this method provides more flexibility, as the output file format, such as CSV, Parquet, TXT, etc., can be specified by the user <strong>[7]</strong>.</p><ul><li><strong>PutFile</strong></li></ul><p>It is a processor capable of writing the contents of a FlowFile to a local file directory <strong>[8]</strong>.</p><ul><li><strong>UpdateAttribute</strong></li></ul><p>It offers DataFlow Managers (DFMs) enhanced control over the FlowFile, allowing for further modifications to the flow, such as renaming the output file. Additionally, it enables the addition or deletion of various attributes based on regular expressions <strong>[9]</strong>.</p><ul><li><strong>LOGAttribute:</strong></li></ul><p>This is a logging method used to capture general information about the process. It supports various logging levels, such as info, warn, error, and debug. When set to the info level, it records both the FlowFile properties and the FlowFile Attribute Map Content <strong>[10]</strong>. In the first section, the execution start datetime and output file size are displayed. In the other section, details such as the executed command type, executed command argument, and output log file name are included.</p><ul><li><strong>LOGMessage</strong></li></ul><p>Similar to LOGAttribute, this can be considered a logging method. However, it is used specifically to display custom log messages defined by the user <strong>[11]</strong>. For example, at the end of the flow, print ‘Process is successfully done.’</p><h3>V. CASE STUDIES</h3><p>In this section, two distinct explanations related to the execution of Python scripts and SQL queries are intended to be significantly displayed along with their respective output logs.</p><h4>Executing Python Scripts</h4><p>In this particular subsection, a detailed explanation of how Python scripts are executed within Apache NiFi, covering the necessary configurations, step-by-step procedures, is supposed to be provided.</p><p>The necessary steps to complete the process are outlined as follows:</p><ol><li>Add ExecuteProcess Processor to the canvas:</li></ol><p>1.1 Drag and drop one Processor from Components Toolbar onto the canvas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*YxzKe4CiSo4soenqomOEZw.gif" /></figure><p>1.2 Search for “ExecuteProcess” processor, then add it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-iTBm-ONqrhwsHTN-M4jLg.png" /></figure><p>1.3 Double-click on the processor to open its configuration, then go to “PROPERTIES” section, and fill the information as in the Figure, below.</p><p><strong>Command</strong>: python3<br><strong>Command Arguments</strong>: /opt/nifi/python/scripts/test.py<br><strong>Working Directory</strong>: /opt/nifi/python/scripts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNukrV1yqyVcmEkebQhv8g.png" /></figure><p>It’s crucial to understand that the command python3 may alternatively be python, depending on the method you employ to execute Python scripts on your system</p><p>2. Add UpdateAttribute Processor.<br>2.1. Repeat the steps outlined in 1. However, this time search for the ‘UpdateAttribute’ processor and proceed to add it to your workflow</p><p>2.2 Double-click on the processor to open its configuration, then go to “PROPERTIES” section, add new property and name it as filename</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5Rs4HNenIOaeTbS-DPs1A.png" /></figure><p>2.3 Modify the ‘filename’ property and assign it the value ‘test_script.log’, which will capture the output of the ‘test.py’ script specified in the ExecuteProcess processor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlVZtkThBTnTvx9-wqESjg.png" /></figure><p>3. Establish a connection between the previously created ExecuteProcessand UpdateAttribute processors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*XyBXXVktNX9lATdQ7U1clw.gif" /></figure><p>4. Add the PutFile processor to the canvas and establish a connection between it and the preceding UpdateAttribute processor.</p><p>4.1 Double-click on the processor to open its configuration, and change the “Directory” property to:</p><p><em>Directory</em>: /opt/nifi/python/logs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v4T3W69vqcV5uxJXUFbbuQ.png" /></figure><p>5. Add LogAttribute processor to the canvas, and make a connection between it and the previousPutFile processor.</p><p>5.1 Double-click on the processor to open its configuration, then go to POPERTIES section and chnage Log prefix to any prefix you prefer.</p><p><em>Log prefix: </em>— Logging the Execution of Python Scripts —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28QGGvR_RMOLdEgZh6z1fw.png" /></figure><p>6. Add LogMessage processor to the canvas, and make a connection between it and the previousLogAttribute processor.</p><p>6.1 Double-click on the processor to open its configuration, then go to “POPERTIES” section and chnage Log prefix to any prefix you prefer. This will print a private message in the log file.</p><p><em>Log prefix</em>: \n — Executing Python script is successfully done — \n</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*820Cje-wEtY9jo-mEXwOvA.png" /></figure><p>7. Navigate to its “RELATIONSHIP” section and check “terminate” box so that it terminates when it is done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*niM1OClXUNzmTbc6Tzb6CA.png" /></figure><p>8. To change the color of each processor, click on the processor, then select the Brush icon, choose a color, and click “APPLY”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_7PXXuCFi01dzKV7ex-RA.png" /></figure><p>9. Add a label for each processor to provide a brief explanation above each one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*k5jUeObN9XMZGJbafeOSzA.gif" /></figure><p>The final flow should be as the one depicted in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bMMB8trqrz45Bceo4vTmpw.png" /></figure><p>10. To observe the progress, run each processor only once by right-clicking on each processor and selecting “Run Once”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXpf5GT8Xcs2Hg35a2WRPQ.png" /></figure><p>Each time, refresh the canvas by right-clicking on it and then clicking the “Refresh” button. The queue section of the connection between each processor should display a number indicating the queued data, which should be 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLnyp5GQa6e1upXHKwevdQ.png" /></figure><p>To view the output of the ExecuteProcess, which is executing a Python script in this case, navigate to the test_script.log file located in the /opt/nifi/python/logs directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*2irSJEkaXryGFxB1MHL01g.png" /></figure><p>To see the output of both LogAttribute and LogMessage processors, navigate to nifi-app.log file under /opt/nifi/logs directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zRvysb3ezrW3JOZRAv5b9g.png" /></figure><h4>Executing SQL Quries</h4><p>The execution of SQL queries in Apache NiFi can be accomplished using the ExecuteSQLRecord processor. In this case, the same architecture used in the Executing Python Scripts subsection will be applied here. However, the ExecuteSQLRecord processor will be used instead of the ExecuteProcess processor.</p><p>Therefore, drag and drop a processor onto the canvas, search for ExecuteSQLRecord, and add it to the canvas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*brk2Tc_Bpl2cjJFfxYskzA.png" /></figure><p>Then, double-click on the processor to open its configuration, then go to properties section, and fill the information as follows:</p><ul><li><strong>Database Connection Pooling Service</strong>: MSSQLConnection<br><em>Note: </em><em>MSSQLConnection controller serviceis prepared in III.DBCPConnectionPool subsection</em></li><li><strong>SQL select query: </strong>&lt;your query&gt;</li><li><strong>Record Writer:</strong> CSVRecordSetWriter<br><em>Note: </em><em>CSVRecordSetWritercontroller serviceis prepared in III.CSVRecordSetWriter subsection.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sx9VEhJnn6hlwfMUDZrjZg.png" /></figure><p>Furthermore, adjust the configurations of the subsequent processors as detailed below:</p><ul><li><strong>UpdateAttribute </strong>&gt; filename: sql_query.log</li><li><strong>PutFile </strong>&gt; Directory: /opt/nifi/sql/logs</li><li><strong>LogAttribute</strong> &gt; Log perfix: -Logging the Execution of SQL Query-</li><li><strong>LogMessage </strong>&gt; Log Message: ************** LOG MESSAGE **************</li><li><strong>LogMessage </strong>&gt; Log perfix: Executing SQL Query is successfully done</li></ul><p>Eventually, the final flow must be as in the figure, below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w037W8WjrYHEcetNW1a2BA.png" /></figure><p>To see the output of ExecuteSQLRecord processor, navigate to sql_query.log file under /opt/nifi/sql/logs directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*it2CXzfgz7P9nU7_2GjYbw.png" /></figure><p>To see the output of both LogAttribute and LogMessage processors, navigate to nifi-app.log file under /opt/nifi/logs directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_iZ0pg_2_FoPaU0s_Utvfg.png" /></figure><h3>VI. CONCLUSION</h3><p>In summary, this article provides a thorough and comprehensive guide on executing Python scripts and SQL queries within Apache NiFi. It meticulously details the execution processes, delves into the prerequisites, and explores logging methods. This detailed guide ensures that readers can easily grasp the process of capturing and recording data in Apache NiFi, making it an invaluable resource for both beginners and experienced DFMs<strong> </strong>in the field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PLAINg7IMRn7t-HdkUzVAg.gif" /></figure><h3>VII. REFERENCES</h3><p><strong>[1]</strong> Saltz, J., Armour, F., &amp; Sharda, R. (2018). Data science roles and the types of data science programs. <em>Communications of the Association for Information Systems</em>, <em>43</em>(1), 33.</p><p><strong>[2] </strong>Alkhanafseh, Y. (2024). An Overview of Apache NiFi. Accessed on [17.03.2024]. Retrieved from: <a href="https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c">https://medium.com/@alkhanafseh/an-overview-of-apache-nifi-44819186674c</a></p><p><strong>[3] </strong>Alkhanafseh, Y. (2024). Installation of Apache NiFi. Accessed on [17.03.2024]. Retrieved from: <a href="https://medium.com/@alkhanafseh/installation-of-apache-nifi-2856fca5bbdc">https://medium.com/@alkhanafseh/installation-of-apache-nifi-2856fca5bbdc</a></p><p><strong>[4]</strong> Apache NiFi (n.d.). DBCPConnectionPool. Accessed on [21.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.24.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.24.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html</a></p><p><strong>[5] </strong>Apache NiFi (n.d.). CSVRecordSetWriter. Accessed on [21.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.23.2/org.apache.nifi.csv.CSVRecordSetWriter/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.23.2/org.apache.nifi.csv.CSVRecordSetWriter/index.html</a></p><p><strong>[6]</strong> Apache NiFi (n.d.). ExecuteSQL. Accessed on [22.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ExecuteSQL/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.ExecuteSQL/index.html</a></p><p><strong>[7]</strong> Apache NiFi (n.d.). ExecuteSQLRecord. Accessed on [23.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apache.nifi.processors.standard.ExecuteSQLRecord/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apache.nifi.processors.standard.ExecuteSQLRecord/index.html</a></p><p><strong>[8]</strong> Apache NiFi (n.d.). PutFile. Accessed on [23.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apache.nifi.processors.standard.PutFile/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apache.nifi.processors.standard.PutFile/index.html</a></p><p><strong>[9]</strong> Apache NiFi (n.d.). UpdateAttribute. Accessed on [23.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.24.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-update-attribute-nar/1.24.0/org.apache.nifi.processors.attributes.UpdateAttribute/index.html</a></p><p><strong>[10]</strong> Apache NiFi (n.d.). LogAttribute. Accessed on [24.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apache.nifi.processors.standard.LogAttribute/index.html">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apache.nifi.processors.standard.LogAttribute/index.html</a></p><p><strong>[11]</strong> Apache NiFi (n.d.). LogMessage. Accessed on [24.03.2024]. Retrieved from: <a href="https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.LogMessage/">https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.LogMessage/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4eb49945846f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/executing-python-scripts-and-sql-quries-in-apache-nifi-4eb49945846f">Executing Python Scripts and SQL Quries In Apache NiFi</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>