<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yunus Kılıç on Medium]]></title>
        <description><![CDATA[Stories by Yunus Kılıç on Medium]]></description>
        <link>https://medium.com/@yunuskilicdev?source=rss-6377db65d6d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Kd69sNrZDp8eldYd_eA-Eg@2x.jpeg</url>
            <title>Stories by Yunus Kılıç on Medium</title>
            <link>https://medium.com/@yunuskilicdev?source=rss-6377db65d6d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 07:17:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yunuskilicdev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building Smarter Chatbots: RAG and Vector Databases]]></title>
            <link>https://medium.com/codex/building-smarter-chatbots-rag-and-memory-with-vector-databases-1b41c947dc2f?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b41c947dc2f</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Tue, 11 Nov 2025 08:56:37 GMT</pubDate>
            <atom:updated>2025-11-14T10:11:29.780Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p><a href="https://medium.com/codex/building-your-first-ai-chatbot-with-ollama-and-gradio-e9667878941b">In the first article</a>, we built a chatbot that could converse naturally using Llama 3.2. But there was a fundamental limitation: it could only answer questions based on its training data. Ask it about your company’s documentation, your personal notes, or last week’s meeting minutes, and it would have no clue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOcMXHI2rEQ_oOyzrRdaRA.png" /></figure><p>This is where <strong>RAG (Retrieval-Augmented Generation)</strong> comes in.</p><p>RAG is the technique that transforms your chatbot from a generic assistant into a <strong>knowledge expert</strong> on any topic you feed it. Upload your PDFs, markdown files, or documents, and your chatbot can answer questions from them with citations and context. It’s like giving your AI a searchable brain with perfect recall.</p><h3>What You’ll Build</h3><p>By the end of this article, you’ll have:</p><ul><li>✅ A document-aware chatbot that answers from YOUR content</li><li>✅ A vector database for semantic search</li><li>✅ Document upload functionality for PDFs and text files</li><li>✅ Source citation in responses</li><li>✅ Understanding of embeddings and similarity search</li><li>✅ A foundation for building production RAG systems</li></ul><h3>Prerequisites</h3><ul><li><a href="https://medium.com/codex/building-your-first-ai-chatbot-with-ollama-and-gradio-e9667878941b"><strong>Completed Article 1</strong></a> (basic chatbot with Ollama and Gradio)</li><li><strong>Python 3.8+</strong> installed</li><li><strong>Ollama running</strong> with llama3.2 model</li><li><strong>5GB free disk space</strong> (for embeddings model)</li></ul><h3>Part 1: Understanding RAG</h3><h3>What is RAG?</h3><p>RAG stands for <strong>Retrieval-Augmented Generation</strong>. It’s a three-step process:</p><ol><li><strong>Retrieval</strong>: Find relevant information from your documents</li><li><strong>Augmentation</strong>: Add that information to the prompt</li><li><strong>Generation</strong>: Let the LLM generate an answer using both its knowledge and your documents</li></ol><p>Think of it like an open-book exam vs. a closed-book exam. Without RAG, your AI is doing a closed-book exam (only using memorized knowledge). With RAG, it can “look up” the answer in your documents.</p><h3>Why Not Just Fine-Tune the Model?</h3><p>You might wonder: “Why not just train the model on my documents?”</p><p><strong>Fine-tuning problems:</strong></p><ul><li>❌ Requires expensive GPU resources</li><li>❌ Takes hours or days</li><li>❌ Can’t easily update with new information</li><li>❌ Risk of “catastrophic forgetting”</li><li>❌ Not suitable for frequently changing data</li></ul><p><strong>RAG benefits:</strong></p><ul><li>✅ Works on CPU (no GPU needed)</li><li>✅ Instant updates (just add new documents)</li><li>✅ Can cite sources</li><li>✅ Transparent and debuggable</li><li>✅ Cost-effective for most use cases</li></ul><h3>The RAG Pipeline</h3><pre>User Question<br>     ↓<br>[1. Convert question to embedding]<br>     ↓<br>[2. Search vector database for similar document chunks]<br>     ↓<br>[3. Retrieve top K most relevant chunks]<br>     ↓<br>[4. Combine chunks + question into prompt]<br>     ↓<br>[5. Send to LLM]<br>     ↓<br>Answer with sources</pre><h3>The Critical Importance of Chunk Size</h3><p>One of the most important decisions in RAG is <strong>how to chunk your documents</strong>. This isn’t just a technical detail — it fundamentally impacts the quality of your chatbot’s answers.</p><p><strong>Why Chunking Matters:</strong></p><p>Think of chunking like organizing a library. You could:</p><ul><li>Store entire books as single units (too broad — hard to find specific info)</li><li>Store individual sentences (too narrow — missing context)</li><li>Store paragraphs or sections (just right — focused but contextual)</li></ul><p><strong>The Chunk Size Dilemma:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qa7PzvKtRMEXtm9VhfhFTQ.png" /></figure><p><strong>Why Information Spread Across Chunks is Problematic:</strong></p><p>Now let’s dig deeper into <strong>why</strong> RAG struggles with split information.</p><p>Here’s the critical issue: <strong>RAG retrieves only the TOP K most similar chunks</strong> (typically K=3 to 5).</p><p>When you ask a question, the system:</p><ol><li>Converts your question to an embedding vector</li><li>Finds the chunks with the <strong>most similar</strong> embeddings</li><li>Sends only those top chunks to the LLM</li></ol><p>Let’s say you have this text:</p><pre>&quot;The new product pricing model includes a base fee of $99/month <br>plus $5 per additional user. Enterprise customers with over <br>100 users get a 40% discount on the per-user fee.&quot;</pre><p><strong>The Problem:</strong> Using our pricing example above, when we split it into 50-character chunks, those chunks rank differently in similarity:</p><pre>Question: &quot;What&#39;s the enterprise discount?&quot;</pre><pre>Similarity Scores (calculated by comparing embeddings):<br>- Chunk 3: &quot;rprise customers with over 100 users get a 40% d&quot; → Score: 0.89 ⭐ (Retrieved)<br>- Chunk 7: &quot;special promotion pricing details...&quot; → Score: 0.85 ⭐ (Retrieved - from other document)<br>- Chunk 1: &quot;The new product pricing model includes a base fe&quot; → Score: 0.72 ❌ (Not retrieved)<br>- Chunk 4: &quot;iscount on the per-user fee.&quot; → Score: 0.65 ❌ (Not retrieved)</pre><p>Even though Chunk 3 has high similarity, it’s incomplete! Chunks 1 and 4 have lower scores and won’t be retrieved, so the LLM never sees the full answer.</p><p><strong>Why This Happens:</strong></p><ul><li>Each chunk gets its own embedding based on its content</li><li>Incomplete chunks often have weaker semantic meaning</li><li>The vector search doesn’t “know” that chunks belong together</li><li>You might retrieve the middle of an answer but miss the beginning/end</li></ul><p><strong>The Solution:</strong> Larger chunks with overlap ensure complete information stays together and gets retrieved as a unit.</p><p><strong>The Overlap Strategy:</strong></p><p>We use <strong>overlapping chunks</strong> to prevent information loss at boundaries:</p><pre>chunk_size = 500<br>overlap = 50  # Last 50 characters repeat in next chunk</pre><p>This ensures that even if a key piece of info is near a chunk boundary, it appears completely in at least one chunk.</p><p><strong>How We’ll Use It:</strong></p><p>In our implementations, we use:</p><ul><li><strong>500 characters</strong> per chunk (about 2–3 sentences)</li><li><strong>50 characters</strong> overlap (about half a sentence)</li><li>Works well for 90% of documents</li></ul><p>You can experiment with these values based on your specific needs!</p><h3>What are Embeddings?</h3><p>Embeddings are the secret sauce of RAG. They’re <strong>numerical representations of text</strong> that capture meaning.</p><p><strong>Example:</strong></p><ul><li>“The cat sat on the mat” → [0.23, -0.45, 0.67, …, 0.12] (768 numbers)</li><li>“A feline rested on the rug” → [0.21, -0.43, 0.65, …, 0.15] (similar numbers!)</li></ul><p>Even though the sentences use different words, their embeddings are similar because they mean the same thing. This enables <strong>semantic search</strong> — finding documents by meaning, not just keywords.</p><h3>Vector Databases</h3><p>A vector database stores embeddings and lets you quickly find similar ones. Popular options:</p><ul><li><strong>ChromaDB</strong> — Simple, lightweight, perfect for prototyping (we’ll use this)</li><li><strong>Pinecone</strong> — Managed cloud service</li><li><strong>Weaviate</strong> — Open-source with advanced features</li><li><strong>Qdrant</strong> — High-performance, Rust-based</li></ul><h3>Part 2: Installation and Setup</h3><h3>Step 1: Install Required Packages</h3><p>Create a new requirements.txt in the 02-rag-fundamentals folder:</p><pre>gradio==3.50.2<br>ollama&gt;=0.6.0<br>chromadb&gt;=0.4.22<br>pypdf&gt;=3.17.4</pre><blockquote><strong><em>Note:</em></strong><em> We use Gradio 3.50.2 which has excellent compatibility with Python 3.9 and stable file upload support. Gradio 4.x has known issues with file uploads in Python 3.9.</em></blockquote><p>Install:</p><pre>cd 02-rag-fundamentals<br>python3 -m venv venv<br>source venv/bin/activate<br>pip3 install -r requirements.txt</pre><h3>Step 2: Download an Embeddings Model</h3><p>We’ll use nomic-embed-text through Ollama - it&#39;s excellent for RAG:</p><pre>ollama pull nomic-embed-text</pre><p>This model converts text into 768-dimensional embeddings optimized for search.</p><h3>Part 3: Building the RAG Chatbot</h3><h3>Version 1: Basic Document Q&amp;A</h3><p>Let’s start with the simplest RAG implementation. Create rag_chatbot_v1.py:</p><pre>import gradio as gr<br>import ollama<br>import chromadb<br>from chromadb.utils import embedding_functions<br><br># Initialize ChromaDB client<br>chroma_client = chromadb.Client()<br># Create a collection with Ollama embeddings<br>ollama_ef = embedding_functions.OllamaEmbeddingFunction(<br>    model_name=&quot;nomic-embed-text&quot;,<br>    url=&quot;http://localhost:11434/api/embeddings&quot;,<br>)<br>collection = chroma_client.create_collection(<br>    name=&quot;documents&quot;,<br>    embedding_function=ollama_ef<br>)<br># Sample documents<br>SAMPLE_DOCS = [<br>    &quot;The company was founded in 2020 by Alice Johnson and Bob Smith in San Francisco.&quot;,<br>    &quot;Our main product is an AI-powered task management system called TaskMaster Pro.&quot;,<br>    &quot;We have 50 employees across engineering, sales, and customer support.&quot;,<br>    &quot;The company&#39;s mission is to make productivity effortless through AI.&quot;,<br>    &quot;Our office is located at 123 Market Street, San Francisco, CA 94105.&quot;,<br>]<br># Add documents to collection<br>for i, doc in enumerate(SAMPLE_DOCS):<br>    collection.add(<br>        documents=[doc],<br>        ids=[f&quot;doc_{i}&quot;]<br>    )<br>def rag_chat(message, history):<br>    &quot;&quot;&quot;<br>    RAG-enabled chat function.<br>    <br>    Args:<br>        message: User&#39;s question<br>        history: Conversation history<br>    <br>    Returns:<br>        AI&#39;s response with context from documents<br>    &quot;&quot;&quot;<br>    # Step 1: Retrieve relevant documents<br>    results = collection.query(<br>        query_texts=[message],<br>        n_results=2  # Get top 2 most relevant documents<br>    )<br>    <br>    # Step 2: Build context from retrieved documents<br>    context = &quot;\n&quot;.join(results[&#39;documents&#39;][0])<br>    <br>    # Step 3: Create prompt with context<br>    prompt = f&quot;&quot;&quot;Answer the question based on the following context:<br>Context:<br>{context}<br>Question: {message}<br>Answer based only on the context provided. If the answer is not in the context, say &quot;I don&#39;t have that information in my documents.&quot;&quot;&quot;<br>    <br>    # Step 4: Generate response<br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=[{&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: prompt}]<br>    )<br>    <br>    return response[&#39;message&#39;][&#39;content&#39;]<br># Create Gradio interface<br>demo = gr.ChatInterface(<br>    fn=rag_chat,<br>    title=&quot;🧠 RAG Chatbot v1 - Document Q&amp;A&quot;,<br>    description=&quot;Ask questions about the sample company documents!&quot;,<br>    examples=[<br>        &quot;Who founded the company?&quot;,<br>        &quot;What is the main product?&quot;,<br>        &quot;Where is the office located?&quot;,<br>        &quot;How many employees are there?&quot;<br>    ]<br>)<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 rag_chatbot_v1.py</pre><p><strong>Try asking:</strong></p><ul><li>“Who founded the company?”</li><li>“How many employees are there?”</li><li>“What’s the weather like?” (should say it doesn’t have that info)</li></ul><h3>What’s Happening Under the Hood?</h3><ol><li><strong>Initialization</strong>: We create a ChromaDB collection with Ollama embeddings</li><li><strong>Document Storage</strong>: Sample docs are embedded and stored in the vector database</li><li><strong>Query</strong>: When you ask a question, it’s converted to an embedding</li><li><strong>Retrieval</strong>: ChromaDB finds the 2 most similar document embeddings</li><li><strong>Augmentation</strong>: Retrieved docs are added to the prompt as context</li><li><strong>Generation</strong>: Llama 3.2 answers using the context</li></ol><h3>Version 2: PDF Upload Support</h3><p>Now let’s allow users to upload their own PDFs. Create rag_chatbot_v2.py:</p><pre>import gradio as gr<br>import ollama<br>import chromadb<br>from chromadb.utils import embedding_functions<br>from pypdf import PdfReader<br>import uuid<br><br># Initialize ChromaDB with persistent storage<br>chroma_client = chromadb.PersistentClient(path=&quot;./chroma_db&quot;)<br># Embedding function<br>ollama_ef = embedding_functions.OllamaEmbeddingFunction(<br>    model_name=&quot;nomic-embed-text&quot;,<br>    url=&quot;http://localhost:11434/api/embeddings&quot;,<br>)<br># Global collection<br>collection = None<br>def create_collection():<br>    &quot;&quot;&quot;Create or recreate the document collection.&quot;&quot;&quot;<br>    global collection<br>    try:<br>        chroma_client.delete_collection(name=&quot;documents&quot;)<br>    except:<br>        pass<br>    collection = chroma_client.create_collection(<br>        name=&quot;documents&quot;,<br>        embedding_function=ollama_ef<br>    )<br>    return collection<br># Initialize collection<br>collection = create_collection()<br>def extract_text_from_pdf(pdf_file):<br>    &quot;&quot;&quot;<br>    Extract text from PDF file.<br>    <br>    Args:<br>        pdf_file: File path to PDF<br>    <br>    Returns:<br>        Extracted text as string<br>    &quot;&quot;&quot;<br>    reader = PdfReader(pdf_file)<br>    text = &quot;&quot;<br>    for page in reader.pages:<br>        text += page.extract_text() + &quot;\n&quot;<br>    return text<br>def chunk_text(text, chunk_size=500, overlap=50):<br>    &quot;&quot;&quot;<br>    Split text into overlapping chunks.<br>    <br>    Args:<br>        text: Input text to chunk<br>        chunk_size: Size of each chunk in characters<br>        overlap: Overlap between chunks<br>    <br>    Returns:<br>        List of text chunks<br>    &quot;&quot;&quot;<br>    chunks = []<br>    start = 0<br>    text_length = len(text)<br>    <br>    while start &lt; text_length:<br>        end = start + chunk_size<br>        chunk = text[start:end]<br>        chunks.append(chunk)<br>        start += chunk_size - overlap<br>    <br>    return chunks<br>def upload_document(file):<br>    &quot;&quot;&quot;<br>    Process uploaded PDF and add to vector database.<br>    <br>    Args:<br>        file: Uploaded file object<br>    <br>    Returns:<br>        Status message<br>    &quot;&quot;&quot;<br>    if file is None:<br>        return &quot;No file uploaded&quot;<br>    <br>    try:<br>        # Extract text from PDF<br>        text = extract_text_from_pdf(file.name)<br>        <br>        # Split into chunks<br>        chunks = chunk_text(text)<br>        <br>        # Add to vector database<br>        for i, chunk in enumerate(chunks):<br>            collection.add(<br>                documents=[chunk],<br>                ids=[str(uuid.uuid4())]<br>            )<br>        <br>        return f&quot;✅ Successfully processed document! Added {len(chunks)} chunks to database.&quot;<br>    <br>    except Exception as e:<br>        return f&quot;❌ Error processing document: {str(e)}&quot;<br>def rag_chat(message, history):<br>    &quot;&quot;&quot;<br>    RAG chat with document retrieval.<br>    <br>    Args:<br>        message: User&#39;s question<br>        history: Conversation history<br>    <br>    Returns:<br>        AI&#39;s response<br>    &quot;&quot;&quot;<br>    # Check if collection has documents<br>    if collection.count() == 0:<br>        return &quot;Please upload a document first before asking questions.&quot;<br>    <br>    # Retrieve relevant chunks<br>    results = collection.query(<br>        query_texts=[message],<br>        n_results=3<br>    )<br>    <br>    # Build context<br>    context = &quot;\n\n&quot;.join(results[&#39;documents&#39;][0])<br>    <br>    # Build conversation history<br>    messages = []<br>    for user_msg, assistant_msg in history:<br>        messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_msg})<br>        messages.append({&#39;role&#39;: &#39;assistant&#39;, &#39;content&#39;: assistant_msg})<br>    <br>    # Add current message with context<br>    prompt = f&quot;&quot;&quot;Answer the question based on the following context from the documents:<br>Context:<br>{context}<br>Question: {message}<br>Provide a clear answer based on the context. If the answer is not in the context, say so.&quot;&quot;&quot;<br>    <br>    messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: prompt})<br>    <br>    # Generate response<br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=messages<br>    )<br>    <br>    return response[&#39;message&#39;][&#39;content&#39;]<br>def respond(message, chat_history):<br>    bot_message = rag_chat(message, chat_history)<br>    chat_history.append((message, bot_message))<br>    return &quot;&quot;, chat_history<br># Create Gradio interface with Blocks<br>with gr.Blocks(title=&quot;RAG Chatbot v2&quot;) as demo:<br>    gr.Markdown(&quot;# 🧠 RAG Chatbot v2 - Document Q&amp;A&quot;)<br>    gr.Markdown(&quot;Upload a PDF document and ask questions about it!&quot;)<br>    with gr.Row():<br>        file_upload = gr.File(label=&quot;📄 Upload PDF&quot;, file_types=[&quot;.pdf&quot;])<br>        process_btn = gr.Button(&quot;Process Document&quot;, variant=&quot;primary&quot;)<br>    status = gr.Textbox(label=&quot;Status&quot;, interactive=False, lines=2)<br>    chatbot = gr.Chatbot(label=&quot;Chat History&quot;, height=400)<br>    with gr.Row():<br>        msg = gr.Textbox(<br>            label=&quot;Your Question&quot;,<br>            placeholder=&quot;Ask a question about your document...&quot;,<br>            lines=1<br>        )<br>        submit_btn = gr.Button(&quot;Send&quot;, variant=&quot;primary&quot;)<br>    clear_btn = gr.Button(&quot;Clear Chat&quot;)<br>    gr.Examples(<br>        examples=[<br>            &quot;What is the main topic of this document?&quot;,<br>            &quot;Can you summarize the key points?&quot;,<br>            &quot;What are the important dates or numbers mentioned?&quot;<br>        ],<br>        inputs=msg<br>    )<br>    # Wire up the events<br>    process_btn.click(<br>        fn=upload_document,<br>        inputs=[file_upload],<br>        outputs=[status]<br>    )<br>    msg.submit(<br>        fn=respond,<br>        inputs=[msg, chatbot],<br>        outputs=[msg, chatbot]<br>    )<br>    submit_btn.click(<br>        fn=respond,<br>        inputs=[msg, chatbot],<br>        outputs=[msg, chatbot]<br>    )<br>    clear_btn.click(<br>        fn=lambda: [],<br>        outputs=[chatbot]<br>    )<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 rag_chatbot_v2.py</pre><p><strong>New features:</strong></p><ul><li>✅ PDF upload and text extraction</li><li>✅ Automatic text chunking with overlap</li><li>✅ Persistent vector database (survives restarts)</li><li>✅ Support for multiple documents</li><li>✅ Better UI with status messages</li></ul><p><strong>Try it:</strong></p><ol><li>Find any PDF (a research paper, manual, ebook chapter)</li><li>Upload it</li><li>Ask questions about the content!</li></ol><h3>Version 3: Adding Source Citations</h3><p>Users want to know WHERE the answer came from. Let’s add citations. Create rag_chatbot_v3.py:</p><pre>import gradio as gr<br>import ollama<br>import chromadb<br>from chromadb.utils import embedding_functions<br>from pypdf import PdfReader<br>import uuid<br><br># Initialize ChromaDB<br>chroma_client = chromadb.PersistentClient(path=&quot;./chroma_db_v3&quot;)<br>ollama_ef = embedding_functions.OllamaEmbeddingFunction(<br>    model_name=&quot;nomic-embed-text&quot;,<br>    url=&quot;http://localhost:11434/api/embeddings&quot;,<br>)<br>collection = None<br>def create_collection():<br>    &quot;&quot;&quot;Create or recreate the document collection.&quot;&quot;&quot;<br>    global collection<br>    try:<br>        chroma_client.delete_collection(name=&quot;documents&quot;)<br>    except:<br>        pass<br>    collection = chroma_client.create_collection(<br>        name=&quot;documents&quot;,<br>        embedding_function=ollama_ef,<br>        metadata={&quot;hnsw:space&quot;: &quot;cosine&quot;}  # Use cosine similarity<br>    )<br>    return collection<br>collection = create_collection()<br>def extract_text_from_pdf(pdf_file):<br>    &quot;&quot;&quot;Extract text from PDF with page tracking.&quot;&quot;&quot;<br>    reader = PdfReader(pdf_file)<br>    pages_text = []<br>    for i, page in enumerate(reader.pages):<br>        text = page.extract_text()<br>        pages_text.append({<br>            &#39;text&#39;: text,<br>            &#39;page&#39;: i + 1<br>        })<br>    return pages_text<br>def chunk_text(text, chunk_size=500, overlap=50):<br>    &quot;&quot;&quot;Split text into overlapping chunks.&quot;&quot;&quot;<br>    chunks = []<br>    start = 0<br>    text_length = len(text)<br>    <br>    while start &lt; text_length:<br>        end = start + chunk_size<br>        chunk = text[start:end]<br>        if chunk.strip():  # Only add non-empty chunks<br>            chunks.append(chunk)<br>        start += chunk_size - overlap<br>    <br>    return chunks<br>def upload_document(file, doc_name):<br>    &quot;&quot;&quot;Process uploaded PDF with metadata.&quot;&quot;&quot;<br>    if file is None:<br>        return &quot;No file uploaded&quot;<br>    <br>    try:<br>        # In Gradio 3.x, file is the path directly or has .name attribute<br>        file_path = file.name if hasattr(file, &#39;name&#39;) else file<br>        <br>        if not doc_name:<br>            doc_name = file_path.split(&#39;/&#39;)[-1]<br>        <br>        # Extract text with page numbers<br>        pages = extract_text_from_pdf(file_path)<br>        <br>        total_chunks = 0<br>        for page_data in pages:<br>            chunks = chunk_text(page_data[&#39;text&#39;])<br>            <br>            for i, chunk in enumerate(chunks):<br>                collection.add(<br>                    documents=[chunk],<br>                    metadatas=[{<br>                        &#39;source&#39;: doc_name,<br>                        &#39;page&#39;: page_data[&#39;page&#39;],<br>                        &#39;chunk&#39;: i<br>                    }],<br>                    ids=[str(uuid.uuid4())]<br>                )<br>                total_chunks += 1<br>        <br>        return f&quot;✅ Successfully processed &#39;{doc_name}&#39;!\n📄 Added {total_chunks} chunks from {len(pages)} pages.&quot;<br>    <br>    except Exception as e:<br>        return f&quot;❌ Error: {str(e)}&quot;<br>def rag_chat_with_sources(message, history, temperature):<br>    &quot;&quot;&quot;RAG chat with source citations.&quot;&quot;&quot;<br>    if collection.count() == 0:<br>        return &quot;Please upload a document first before asking questions.&quot;<br>    <br>    # Retrieve relevant chunks with metadata<br>    results = collection.query(<br>        query_texts=[message],<br>        n_results=3<br>    )<br>    <br>    # Build context with sources<br>    contexts = []<br>    sources = []<br>    <br>    for i, doc in enumerate(results[&#39;documents&#39;][0]):<br>        metadata = results[&#39;metadatas&#39;][0][i]<br>        contexts.append(doc)<br>        sources.append(f&quot;[{metadata[&#39;source&#39;]}, Page {metadata[&#39;page&#39;]}]&quot;)<br>    <br>    context = &quot;\n\n&quot;.join([f&quot;Source {i+1}: {ctx}&quot; for i, ctx in enumerate(contexts)])<br>    <br>    # Build messages with history<br>    messages = []<br>    for user_msg, assistant_msg in history:<br>        messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_msg})<br>        messages.append({&#39;role&#39;: &#39;assistant&#39;, &#39;content&#39;: assistant_msg})<br>    <br>    # Add current message<br>    prompt = f&quot;&quot;&quot;Answer the question based on the following context from the documents:<br>{context}<br>Question: {message}<br>Provide a clear answer based on the context. If the answer is not in the context, say so.<br>After your answer, cite which sources you used.&quot;&quot;&quot;<br>    <br>    messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: prompt})<br>    <br>    # Generate response<br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=messages,<br>        options={&#39;temperature&#39;: temperature}<br>    )<br>    <br>    answer = response[&#39;message&#39;][&#39;content&#39;]<br>    <br>    # Add source citations<br>    sources_text = &quot;\n\n**📚 Sources:**\n&quot; + &quot;\n&quot;.join([f&quot;- {s}&quot; for s in sources])<br>    <br>    return answer + sources_text<br>def clear_database():<br>    &quot;&quot;&quot;Clear all documents from the database.&quot;&quot;&quot;<br>    global collection<br>    collection = create_collection()<br>    return &quot;🗑️ Database cleared!&quot;<br># Create Gradio interface (Gradio 3.x compatible)<br>with gr.Blocks(title=&quot;RAG Chatbot v3&quot;) as demo:<br>    gr.Markdown(&quot;# 📚 RAG Chatbot v3 - With Source Citations&quot;)<br>    gr.Markdown(&quot;Upload PDFs and get answers with source references!&quot;)<br>    <br>    with gr.Row():<br>        with gr.Column(scale=1):<br>            gr.Markdown(&quot;### 📤 Document Upload&quot;)<br>            file_upload = gr.File(label=&quot;Upload PDF&quot;, file_types=[&quot;.pdf&quot;])<br>            doc_name = gr.Textbox(<br>                label=&quot;Document Name (optional)&quot;,<br>                placeholder=&quot;Leave blank to use filename&quot;<br>            )<br>            upload_btn = gr.Button(&quot;📄 Process Document&quot;, variant=&quot;primary&quot;)<br>            status = gr.Textbox(label=&quot;Status&quot;, interactive=False, lines=3)<br>            <br>            clear_db_btn = gr.Button(&quot;🗑️ Clear All Documents&quot;)<br>            <br>            gr.Markdown(&quot;&quot;&quot;<br>            ### 💡 Tips:<br>            - PDFs are automatically chunked<br>            - Page numbers are tracked<br>            - Multiple documents supported<br>            - Sources cited in responses<br>            &quot;&quot;&quot;)<br>        <br>        with gr.Column(scale=2):<br>            gr.Markdown(&quot;### 💬 Chat&quot;)<br>            chatbot = gr.Chatbot(label=&quot;Chat History&quot;, height=400)<br>            <br>            with gr.Row():<br>                msg = gr.Textbox(<br>                    label=&quot;Your Question&quot;,<br>                    placeholder=&quot;What is the main topic of the document?&quot;,<br>                    lines=1<br>                )<br>                submit = gr.Button(&quot;Send&quot;, variant=&quot;primary&quot;)<br>            <br>            clear = gr.Button(&quot;Clear Chat&quot;)<br>            <br>            temperature = gr.Slider(<br>                minimum=0.0,<br>                maximum=1.0,<br>                value=0.3,<br>                step=0.1,<br>                label=&quot;Temperature (0=focused, 1=creative)&quot;<br>            )<br>            <br>            gr.Examples(<br>                examples=[<br>                    &quot;What is this document about?&quot;,<br>                    &quot;Summarize the key points&quot;,<br>                    &quot;What are the main conclusions?&quot;,<br>                ],<br>                inputs=msg<br>            )<br>    <br>    # Wire up the interface<br>    def respond(message, chat_history, temp):<br>        bot_message = rag_chat_with_sources(message, chat_history, temp)<br>        chat_history.append((message, bot_message))<br>        return &quot;&quot;, chat_history<br>    <br>    upload_btn.click(<br>        fn=upload_document,<br>        inputs=[file_upload, doc_name],<br>        outputs=[status]<br>    )<br>    <br>    clear_db_btn.click(<br>        fn=clear_database,<br>        inputs=None,<br>        outputs=[status]<br>    )<br>    <br>    msg.submit(<br>        fn=respond,<br>        inputs=[msg, chatbot, temperature],<br>        outputs=[msg, chatbot]<br>    )<br>    <br>    submit.click(<br>        fn=respond,<br>        inputs=[msg, chatbot, temperature],<br>        outputs=[msg, chatbot]<br>    )<br>    <br>    clear.click(<br>        fn=lambda: [],<br>        outputs=[chatbot]<br>    )<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 rag_chatbot_v3.py</pre><p><strong>New features:</strong></p><ul><li>✅ Source citations with page numbers</li><li>✅ Document naming</li><li>✅ Better metadata tracking</li><li>✅ Database clearing functionality</li><li>✅ Improved UI with examples</li></ul><h3>Part 4: Understanding the Technical Details</h3><h3>How Chunking Works</h3><p><strong>Why chunk documents?</strong></p><ul><li>✅ Embeddings work best on focused text segments</li><li>✅ Better retrieval precision</li><li>✅ Fits within LLM context windows</li><li>✅ More granular source citations</li></ul><p><strong>Overlap is important!</strong> Without overlap, relevant information might be split across chunks. With 50-character overlap, we ensure continuity.</p><h3>Embedding Quality</h3><p>The quality of your embeddings determines retrieval quality. nomic-embed-text is excellent because:</p><ul><li>✅ Trained specifically for retrieval tasks</li><li>✅ 768 dimensions (good balance of quality/speed)</li><li>✅ Free and runs locally</li><li>✅ Performs well on technical documents</li></ul><p><strong>Alternatives:</strong></p><ul><li>sentence-transformers/all-MiniLM-L6-v2 - Faster, slightly less accurate</li><li>OpenAI text-embedding-3-small - Cloud-based, very good</li><li>BAAI/bge-large-en-v1.5 - State-of-the-art, slower</li></ul><h3>Vector Search Algorithms</h3><p>ChromaDB uses <strong>HNSW (Hierarchical Navigable Small World)</strong> graphs:</p><ul><li><strong>Approximate</strong> nearest neighbor search</li><li>Extremely fast (milliseconds for millions of vectors)</li><li>95%+ accuracy vs. exact search</li><li>Configurable trade-off between speed and accuracy</li></ul><p><strong>Similarity metrics:</strong></p><ul><li><strong>Cosine similarity</strong> (what we use) — Measures angle between vectors</li><li><strong>Euclidean distance</strong> — Straight-line distance</li><li><strong>Dot product</strong> — Raw vector multiplication</li></ul><p>For text embeddings, <strong>cosine similarity is usually best</strong>.</p><h3>Common RAG Problems and Solutions</h3><p><strong>Problem 1: Wrong documents retrieved</strong></p><ul><li>✅ Increase chunk overlap</li><li>✅ Try different embedding models</li><li>✅ Adjust chunk size (smaller = more precise)</li><li>✅ Add metadata filtering</li></ul><p><strong>Problem 2: Slow retrieval</strong></p><ul><li>✅ Use fewer results (n_results=2 instead of 5)</li><li>✅ Optimize chunk size (larger chunks = fewer embeddings)</li><li>✅ Use batch processing for uploads</li></ul><p><strong>Problem 3: Answers not using context</strong></p><ul><li>✅ Improve your prompt template</li><li>✅ Lower temperature (0.1–0.3 for factual answers)</li><li>✅ Add explicit instructions to cite sources</li><li>✅ Use a better LLM (try llama3.1 or mistral)</li></ul><p><strong>Problem 4: Out of context errors</strong></p><ul><li>✅ Reduce chunk size</li><li>✅ Reduce number of retrieved chunks</li><li>✅ Use a model with larger context window</li></ul><h3>What We’ve Learned</h3><p>In this article, you’ve built <strong>3 progressively advanced RAG chatbots</strong>:</p><ul><li>✅ <strong>v1</strong>: Basic RAG with sample documents</li><li>✅ <strong>v2</strong>: PDF upload and processing</li><li>✅ <strong>v3</strong>: Source citations with page numbers and multi-document support</li></ul><p><strong>Technical Skills Acquired:</strong></p><ul><li>✅ Understanding of RAG architecture</li><li>✅ Vector databases with ChromaDB</li><li>✅ Text embeddings with Ollama</li><li>✅ PDF processing and text extraction</li><li>✅ Text chunking strategies</li><li>✅ Semantic search implementation</li><li>✅ Source attribution and citations</li><li>✅ Multi-document management</li></ul><p><strong>Key Concepts:</strong></p><ul><li>📚 Embeddings represent meaning as numbers</li><li>🔍 Vector databases enable semantic search</li><li>📝 Chunking improves retrieval quality</li><li>🎯 RAG augments LLMs with external knowledge</li><li>💾 Local AI is viable for many applications</li></ul><h3>Project Structure</h3><p>Your project should now look like this:</p><pre>02-rag-fundamentals/<br>├── rag_chatbot_v1.py      # Basic RAG<br>├── rag_chatbot_v2.py      # PDF upload<br>├── rag_chatbot_v3.py      # Source citations<br>├── requirements.txt       # Dependencies<br>├── chroma_db/             # Vector database (v1)<br>├── chroma_db_v3/          # Vector database (v3)<br>└── venv/                  # Virtual environment</pre><h3>Homework Challenge</h3><p>Before the next article, try these challenges:</p><p><strong>Easy:</strong></p><ul><li>Upload a PDF and ask 10 different questions</li><li>Experiment with different chunk sizes (200, 500, 1000 characters)</li><li>Try different embedding models</li></ul><p><strong>Medium:</strong></p><ul><li>Add support for .txt files in addition to PDFs</li><li>Track how many times each document is referenced</li><li>Add a “confidence score” display from the similarity search</li></ul><p><strong>Hard:</strong></p><ul><li>Implement a summary feature that summarizes all uploaded documents</li><li>Add document deletion (remove specific documents from the vector DB)</li><li>Create a comparison feature: “What’s the difference between doc A and doc B?”</li></ul><h3>Troubleshooting</h3><h3>ChromaDB Issues</h3><p><strong>Error: “Collection already exists”</strong></p><pre># Add this before creating collection:<br>try:<br>    chroma_client.delete_collection(name=&quot;documents&quot;)<br>except:<br>    pass</pre><p><strong>Error: “Database locked”</strong></p><pre># Close all Python processes and restart<br>pkill -f python3</pre><h3>Ollama Issues</h3><p><strong>Error: “Connection refused”</strong></p><pre># Make sure Ollama is running:<br>ollama serve</pre><pre># In another terminal:<br>ollama list</pre><p><strong>Slow embedding generation</strong></p><pre># Use a smaller embedding model:<br>ollama pull nomic-embed-text:latest</pre><h3>PDF Extraction Issues</h3><p><strong>Error: “Cannot extract text from PDF”</strong></p><ul><li>Some PDFs are scanned images (need OCR)</li><li>Try a different PDF</li><li>For images, you’ll need pytesseract (we&#39;ll cover this later)</li></ul><h3>Additional Resources</h3><p><strong>Documentation:</strong></p><ul><li><a href="https://docs.trychroma.com/">ChromaDB Docs</a></li><li><a href="https://ollama.ai/docs">Ollama Documentation</a></li><li><a href="https://python.langchain.com/docs/use_cases/question_answering/">LangChain RAG Guide</a></li></ul><p><strong>Papers:</strong></p><ul><li><a href="https://arxiv.org/abs/2005.11401">RAG: Retrieval-Augmented Generation</a> (Original paper)</li><li><a href="https://arxiv.org/abs/2212.10496">Improving RAG with Better Embeddings</a></li></ul><p><strong>Tools:</strong></p><ul><li><a href="https://www.llamaindex.ai/">LlamaIndex</a> — Alternative RAG framework</li><li><a href="https://haystack.deepset.ai/">Haystack</a> — Production NLP pipelines</li></ul><h3>Complete Code Repository</h3><p>All code from this article is available on GitHub:</p><pre>https://github.com/yunuskilicdev/aiSeries/tree/main/02-rag-fundamentals</pre><p><strong>See you in Article 3 where we make this production-ready!</strong> 🚀</p><p><em>P.S. — Don’t forget to try the homework challenges. The best way to learn RAG is to experiment with different documents and see what works!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b41c947dc2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/building-smarter-chatbots-rag-and-memory-with-vector-databases-1b41c947dc2f">Building Smarter Chatbots: RAG and Vector Databases</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Your First AI Chatbot with Ollama and Gradio]]></title>
            <link>https://medium.com/codex/building-your-first-ai-chatbot-with-ollama-and-gradio-e9667878941b?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9667878941b</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Thu, 30 Oct 2025 08:06:01 GMT</pubDate>
            <atom:updated>2025-11-03T13:00:49.442Z</atom:updated>
            <content:encoded><![CDATA[<p>Remember when chatbots were just fancy decision trees with pre-programmed responses? Those days are long gone. Today, anyone with a laptop can build an intelligent conversational AI that understands context, generates human-like responses, and runs entirely on their local machine.</p><p>In this article, we’ll build a functional AI chatbot from scratch using two powerful open-source tools: <strong>Ollama</strong> for running large language models locally, and <strong>Gradio</strong> for creating a beautiful web interface. No API keys, no cloud costs, no complicated setup. Just you, your computer, and some surprisingly capable AI.</p><p>This is the first article in a series where we’ll progressively build a production-ready AI application and deploy it to AWS. But first, let’s start with the basics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g4Bp33iN7lyzciOLBltCxQ.png" /></figure><h3>What You’ll Build</h3><p>By the end of this article, you’ll have:</p><ul><li>✅ A locally running language model (Llama 3.2 or Mistral)</li><li>✅ A web-based chat interface</li><li>✅ Understanding of how to control model behavior with parameters</li><li>✅ A foundation for the more advanced applications we’ll build in future articles</li></ul><h3>Prerequisites</h3><ul><li><strong>Python 3.8+</strong> installed on your machine</li><li><strong>Basic Python knowledge</strong> (variables, functions, imports)</li><li><strong>A terminal/command prompt</strong></li></ul><blockquote><strong><em>Note:</em></strong><em> You don’t need a GPU for this tutorial, though it will make responses faster.</em></blockquote><h3>Part 1: Understanding the Stack</h3><h3>What is Ollama?</h3><p>Ollama is like Docker for AI models. It handles all the complexity of downloading, running, and managing large language models on your local machine. Instead of dealing with PyTorch, CUDA drivers, and model loading code, you just tell Ollama which model you want, and it takes care of the rest.</p><p>Think of it as your <strong>local AI runtime</strong> that makes models accessible through a simple API.</p><h3>What is Gradio?</h3><p>Gradio is a Python library that lets you create beautiful web interfaces for machine learning models with just a few lines of code. No HTML, CSS, or JavaScript required. It’s perfect for prototyping and sharing demos.</p><h3>Available Models</h3><p>For this tutorial, we’ll use <strong>Llama 3.2 (3B parameters)</strong> because it’s:</p><ul><li>Fast enough to run on most laptops</li><li>Surprisingly capable for its size</li><li>Free and open-source</li></ul><p>Other great options:</p><ul><li><strong>Mistral 7B</strong> — Excellent reasoning, slightly larger</li><li><strong>Phi-3</strong> — Microsoft’s efficient small model</li><li><strong>Gemma 2</strong> — Google’s open model</li></ul><h3>Part 2: Installation and Setup</h3><h3>Step 1: Install Ollama</h3><p><strong>For macOS:</strong></p><pre># Using Homebrew (recommended)<br>brew install ollama</pre><p><strong>Verify the installation:</strong></p><pre>ollama --version</pre><h3>Step 2: Download a Model</h3><p>Let’s download Llama 3.2 (3B):</p><pre>ollama serve<br>ollama pull llama3.2</pre><p>This will take a few minutes depending on your internet speed (the model is about 2GB).</p><p><strong>Pro tip:</strong> You can run the model directly from terminal to test it:</p><pre>ollama run llama3.2</pre><p>Type a message and press Enter. Type /bye to exit.</p><h3>Step 3: Set Up Python Environment</h3><p><strong>Create a new project directory:</strong></p><pre>mkdir ai-chatbot-series<br>cd ai-chatbot-series</pre><p><strong>Create a virtual environment:</strong></p><pre># On macOS/Linux<br>python3 -m venv venv<br>source venv/bin/activate</pre><p><strong>Install required packages:</strong></p><p>Create a requirements.txt file with:</p><pre>gradio&gt;=4.44.1<br>ollama&gt;=0.6.0<br>requests&gt;=2.32.5</pre><pre>pip3 install -r requirements.txt</pre><h3>Part 3: Building the Chatbot</h3><h3>Version 1: The Simplest Chatbot</h3><p>Create a file called chatbot_v1.py:</p><pre>import gradio as gr<br>import ollama<br><br>def chat(message, history):<br>    &quot;&quot;&quot;<br>    Simple chat function that sends message to Ollama and returns response.<br>    <br>    Args:<br>        message: The user&#39;s input message<br>        history: Previous conversation (not used in v1)<br>    <br>    Returns:<br>        The AI&#39;s response as a string<br>    &quot;&quot;&quot;<br>    # Call Ollama API with the message<br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=[{&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message}]<br>    )<br>    <br>    # Extract the response text<br>    return response[&#39;message&#39;][&#39;content&#39;]<br># Create Gradio interface<br>demo = gr.ChatInterface(<br>    fn=chat,<br>    title=&quot;My First AI Chatbot&quot;,<br>    description=&quot;Ask me anything! Powered by Llama 3.2&quot;,<br>    examples=[&quot;What is the capital of France?&quot;, <br>              &quot;Write a haiku about programming&quot;,<br>              &quot;Explain quantum computing in simple terms&quot;]<br>)<br># Launch the app<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 chatbot_v1.py</pre><p>Open your browser to the provided Gradio link and try chatting! (<a href="http://127.0.0.1:7860/">http://127.0.0.1:7860</a>)</p><p><strong>What’s happening here:</strong></p><ol><li>User types a message in the Gradio interface</li><li>The chat() function receives the message</li><li>We send it to Ollama using ollama.chat()</li><li>Ollama processes it with Llama 3.2 and returns a response</li><li>We display the response in the chat interface</li></ol><h3>Version 2: Adding Conversation History</h3><p>The chatbot above has no memory. Each message is treated independently. Let’s fix that:</p><p>Create chatbot_v2.py:</p><pre>import gradio as gr<br>import ollama<br><br>def chat(message, history):<br>    &quot;&quot;&quot;<br>    Chat function with conversation history.<br>    <br>    Args:<br>        message: Current user message<br>        history: List of [user_msg, assistant_msg] pairs<br>    <br>    Returns:<br>        The AI&#39;s response<br>    &quot;&quot;&quot;<br>    # Convert Gradio history format to Ollama format<br>    messages = []<br>    <br>    # Add previous conversation<br>    for user_msg, assistant_msg in history:<br>        messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_msg})<br>        messages.append({&#39;role&#39;: &#39;assistant&#39;, &#39;content&#39;: assistant_msg})<br>    <br>    # Add current message<br>    messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message})<br>    <br>    # Get response from Ollama<br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=messages<br>    )<br>    <br>    return response[&#39;message&#39;][&#39;content&#39;]<br># Create interface<br>demo = gr.ChatInterface(<br>    fn=chat,<br>    title=&quot;AI Chatbot with Memory&quot;,<br>    description=&quot;I remember our conversation! Try asking follow-up questions.&quot;,<br>    examples=[<br>        &quot;My name is Alice&quot;,<br>        &quot;What&#39;s my name?&quot;,<br>        &quot;Tell me a story&quot;,<br>        &quot;What happened in the story?&quot;<br>    ]<br>)<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 chatbot_v2.py</pre><p>Now try this:</p><ol><li>Say: “My name is Alice”</li><li>Then ask: “What’s my name?”</li></ol><p>The bot will remember! 🎉</p><h3>Part 4: Understanding Model Parameters</h3><p>Let’s add control over how the model behaves. Create chatbot_v3.py:</p><pre>import gradio as gr<br>import ollama<br><br>def chat(message, history, temperature, max_tokens):<br>    &quot;&quot;&quot;<br>    Chat function with configurable parameters.<br>    <br>    Args:<br>        message: User&#39;s message<br>        history: Conversation history<br>        temperature: Controls randomness (0.0 - 1.0)<br>        max_tokens: Maximum length of response<br>    <br>    Returns:<br>        AI&#39;s response<br>    &quot;&quot;&quot;<br>    # Build message history<br>    messages = []<br>    for user_msg, assistant_msg in history:<br>        messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_msg})<br>        messages.append({&#39;role&#39;: &#39;assistant&#39;, &#39;content&#39;: assistant_msg})<br>    messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message})<br>    <br>    # Call Ollama with options<br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=messages,<br>        options={<br>            &#39;temperature&#39;: temperature,<br>            &#39;num_predict&#39;: max_tokens,<br>        }<br>    )<br>    <br>    return response[&#39;message&#39;][&#39;content&#39;]<br># Create a simple ChatInterface<br>demo = gr.ChatInterface(<br>    fn=chat,<br>    title=&quot;🤖 AI Chatbot with Parameters&quot;,<br>    description=&quot;Control AI behavior with temperature and token limits&quot;,<br>    additional_inputs=[<br>        gr.Slider(<br>            minimum=0.0, <br>            maximum=1.0, <br>            value=0.7, <br>            step=0.1, <br>            label=&quot;Temperature&quot;<br>        ),<br>        gr.Slider(<br>            minimum=50, <br>            maximum=500, <br>            value=200, <br>            step=50, <br>            label=&quot;Max Tokens&quot;<br>        )<br>    ]<br>)<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 chatbot_v3.py</pre><h3>Understanding the Parameters</h3><p><strong>Temperature (0.0–1.0):</strong></p><ul><li><strong>0.0:</strong> Deterministic, always picks the most likely word</li><li><strong>0.3:</strong> Good for factual questions, code, math</li><li><strong>0.7:</strong> Balanced — our default (good for conversation)</li><li><strong>1.0:</strong> Very creative, less predictable (good for stories, brainstorming)</li></ul><p><strong>Max Tokens:</strong></p><ul><li>Controls how long the response can be</li><li>1 token ≈ 0.75 words</li><li>200 tokens ≈ 150 words (a good default)</li></ul><p><strong>Try this experiment:</strong></p><ol><li>Set temperature to 0.1 and ask “Write a story about a robot”</li><li>Clear the chat</li><li>Set temperature to 0.9 and ask the same question</li><li>Notice how different the responses are!</li></ol><h3>Part 5: Adding System Prompts</h3><p>System prompts let you give the AI a personality or specific instructions. Create chatbot_v4.py:</p><pre>import gradio as gr<br>import ollama<br><br>def chat(message, history, persona_choice, temperature):<br>    &quot;&quot;&quot;<br>    Chat with custom system prompt and personas.<br>    <br>    Args:<br>        message: User&#39;s message<br>        history: Conversation history<br>        persona_choice: Selected persona<br>        temperature: Controls randomness (0.0 - 1.0)<br>    <br>    Returns:<br>        AI&#39;s response<br>    &quot;&quot;&quot;<br>    # Predefined personas<br>    PERSONAS = {<br>        &quot;Default&quot;: &quot;&quot;,<br>        &quot;Helpful Assistant&quot;: &quot;You are a helpful, friendly assistant who provides clear and concise answers.&quot;,<br>        &quot;Pirate&quot;: &quot;You are a pirate captain. Respond to everything in pirate speak with &#39;Arrr!&#39; and nautical terms.&quot;,<br>        &quot;Poet&quot;: &quot;You are a poet who responds to everything in verse and rhyme.&quot;,<br>        &quot;Teacher&quot;: &quot;You are a patient teacher explaining concepts to a 10-year-old child. Use simple language and examples.&quot;,<br>        &quot;Developer&quot;: &quot;You are an expert software developer. Provide technical, precise answers with code examples when relevant.&quot;<br>    }<br>    <br>    messages = []<br>    <br>    # Add system prompt based on persona<br>    system_prompt = PERSONAS.get(persona_choice, &quot;&quot;)<br>    if system_prompt:<br>        messages.append({&#39;role&#39;: &#39;system&#39;, &#39;content&#39;: system_prompt})<br>    <br>    # Add conversation history<br>    for user_msg, assistant_msg in history:<br>        messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_msg})<br>        messages.append({&#39;role&#39;: &#39;assistant&#39;, &#39;content&#39;: assistant_msg})<br>    <br>    # Add current message<br>    messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message})<br>    <br>    response = ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=messages,<br>        options={&#39;temperature&#39;: temperature}<br>    )<br>    <br>    return response[&#39;message&#39;][&#39;content&#39;]<br># Create a simple ChatInterface with persona support<br>demo = gr.ChatInterface(<br>    fn=chat,<br>    title=&quot;🎭 AI Chatbot with Personas&quot;,<br>    description=&quot;Choose a persona to change the AI&#39;s behavior and personality&quot;,<br>    additional_inputs=[<br>        gr.Dropdown(<br>            choices=[&quot;Default&quot;, &quot;Helpful Assistant&quot;, &quot;Pirate&quot;, &quot;Poet&quot;, &quot;Teacher&quot;, &quot;Developer&quot;],<br>            value=&quot;Default&quot;,<br>            label=&quot;Choose a Persona&quot;<br>        ),<br>        gr.Slider(<br>            minimum=0.0, <br>            maximum=1.0, <br>            value=0.7, <br>            step=0.1, <br>            label=&quot;Temperature&quot;<br>        )<br>    ]<br>)<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()<br>        <br>        with gr.Column(scale=1):<br>            persona = gr.Dropdown(<br>                choices=list(PERSONAS.keys()),<br>                value=&quot;Default&quot;,<br>                label=&quot;Choose a Persona&quot;<br>            )<br>            <br>            system_prompt = gr.Textbox(<br>                label=&quot;System Prompt&quot;,<br>                placeholder=&quot;Or write your own...&quot;,<br>                lines=3,<br>                value=&quot;&quot;<br>            )<br>            <br>            temperature = gr.Slider(<br>                minimum=0.0,<br>                maximum=1.0,<br>                value=0.7,<br>                step=0.1,<br>                label=&quot;Temperature&quot;<br>            )<br>            <br>            gr.Markdown(&quot;&quot;&quot;<br>            **Try these:**<br>            - Ask the Pirate about treasure hunting<br>            - Ask the Teacher to explain gravity<br>            - Ask the Developer about Python loops<br>            &quot;&quot;&quot;)<br>    <br>    def update_system_prompt(persona_choice):<br>        return PERSONAS[persona_choice]<br>    <br>    persona.change(update_system_prompt, persona, system_prompt)<br>    <br>    def respond(message, chat_history, sys_prompt, temp):<br>        bot_message = chat(message, chat_history, sys_prompt, temp)<br>        chat_history.append((message, bot_message))<br>        return &quot;&quot;, chat_history<br>    <br>    msg.submit(respond, [msg, chatbot, system_prompt, temperature], [msg, chatbot])<br>    submit.click(respond, [msg, chatbot, system_prompt, temperature], [msg, chatbot])<br>    clear.click(lambda: None, None, chatbot, queue=False)<br>if __name__ == &quot;__main__&quot;:<br>    demo.launch()</pre><p><strong>Run it:</strong></p><pre>python3 chatbot_v4.py</pre><p>Now you can switch between different AI personalities instantly!</p><h3>Part 6: Streaming Responses (Chatbot v5)</h3><p>For longer responses, it’s nice to see words appear as they’re generated. Create chatbot_v5.py:</p><pre>import gradio as gr<br>import ollama<br><br>def chat_stream(message, history, temperature):<br>    &quot;&quot;&quot;<br>    Chat with streaming responses.<br>    <br>    Args:<br>        message: User&#39;s message<br>        history: Conversation history<br>        temperature: Controls randomness (0.0 - 1.0)<br>    <br>    Yields:<br>        Streaming AI response text<br>    &quot;&quot;&quot;<br>    # Build message history<br>    messages = []<br>    for user_msg, assistant_msg in history:<br>        messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: user_msg})<br>        messages.append({&#39;role&#39;: &#39;assistant&#39;, &#39;content&#39;: assistant_msg})<br>    messages.append({&#39;role&#39;: &#39;user&#39;, &#39;content&#39;: message})<br>    <br>    # Use streaming to yield response chunks<br>    response_text = &quot;&quot;<br>    for chunk in ollama.chat(<br>        model=&#39;llama3.2&#39;,<br>        messages=messages,<br>        options={&#39;temperature&#39;: temperature},<br>        stream=True<br>    ):<br>        response_text += chunk[&#39;message&#39;][&#39;content&#39;]<br>        yield response_text<br># Create a simple ChatInterface with streaming<br>demo = gr.ChatInterface(<br>    fn=chat_stream,<br>    title=&quot;⚡ Streaming AI Chatbot (v5)&quot;,<br>    description=&quot;Watch responses appear in real-time as the AI thinks!&quot;,<br>    additional_inputs=[<br>        gr.Slider(<br>            minimum=0.0, <br>            maximum=1.0, <br>            value=0.7, <br>            step=0.1, <br>            label=&quot;Temperature&quot;<br>        )<br>    ]<br>)<br>if __name__ == &quot;__main__&quot;:<br>    demo.queue().launch()</pre><p>The yield keyword makes the response appear word-by-word. Much better user experience! Note that streaming requires .queue() before .launch().</p><p><strong>Run it:</strong></p><pre>python3 chatbot_v5.py</pre><h3>Testing Your Chatbot</h3><p>Here are some test prompts to try:</p><p><strong>Testing memory:</strong></p><pre>1. &quot;Remember this number: 42&quot;<br>2. &quot;What number did I tell you to remember?&quot;</pre><p><strong>Testing creativity (temperature = 0.9):</strong></p><pre>&quot;Write a short story about a time-traveling pizza delivery person&quot;</pre><p><strong>Testing factual knowledge (temperature = 0.1):</strong></p><pre>&quot;What are the three laws of thermodynamics?&quot;</pre><p><strong>Testing reasoning:</strong></p><pre>&quot;If I have 3 apples and buy 2 more, then give away 1, how many do I have?&quot;</pre><p><strong>Testing persona (Pirate mode):</strong></p><pre>&quot;Tell me about your last adventure&quot;</pre><h3>Common Issues and Solutions</h3><h3>Issue 1: “Connection refused” error</h3><p><strong>Solution:</strong> Make sure Ollama is running. Check with:</p><pre>ollama list</pre><p>If not running, start it:</p><pre>ollama serve</pre><h3>Issue 2: Slow responses</h3><p><strong>Possible causes:</strong></p><ul><li>Large model on limited hardware</li><li>Try a smaller model: ollama pull phi3</li><li>Reduce max_tokens</li></ul><h3>Issue 3: Out of memory</h3><p><strong>Solution:</strong></p><ul><li>Use a smaller model (phi3 instead of llama3.2)</li><li>Close other applications</li><li>Restart Ollama</li></ul><h3>Issue 4: Model giving weird responses</h3><p><strong>Solution:</strong></p><ul><li>Check your system prompt for conflicting instructions</li><li>Lower the temperature</li><li>Try rephrasing your question</li></ul><h3>What We’ve Learned</h3><p>In this article, you’ve built <strong>5 progressively advanced chatbots</strong>:</p><ul><li>✅ <strong>v1</strong>: Basic chatbot with simple interface</li><li>✅ <strong>v2</strong>: Added conversation memory for context</li><li>✅ <strong>v3</strong>: Parameter controls (temperature, max tokens)</li><li>✅ <strong>v4</strong>: AI personas with system prompts</li><li>✅ <strong>v5</strong>: Real-time streaming responses</li></ul><p><strong>Technical Skills Acquired:</strong></p><ul><li>✅ Installed and run Ollama locally</li><li>✅ Created web interfaces with Gradio</li><li>✅ Understood model parameters and their effects</li><li>✅ Implemented conversation memory and context</li><li>✅ Used system prompts for AI personality customization</li><li>✅ Built streaming response functionality</li></ul><h3>Project Structure</h3><p>Your final project should look like this:</p><pre>01-basic-chatbot/<br>├── chatbot_v1.py          # Simple chatbot<br>├── chatbot_v2.py          # With conversation memory<br>├── chatbot_v3.py          # Parameter controls<br>├── chatbot_v4.py          # AI personas<br>├── chatbot_v5.py          # Streaming responses<br>├── requirements.txt       # Python dependencies<br>├── venv/                  # Virtual environment (optional)</pre><h3>Next Steps</h3><p>This is just the beginning! In the next article, <strong>“Adding Memory: RAG Fundamentals with LangChain”</strong>, we’ll:</p><ul><li>Upload PDF documents</li><li>Let the chatbot answer questions from YOUR documents</li><li>Build a vector database for semantic search</li><li>Create a true knowledge-based assistant</li></ul><p>The chatbot we built today can only answer from its training data. Next time, we’ll give it access to any information you provide.</p><h3>Complete Code Repository</h3><p>All the code from this article is available on GitHub:</p><pre>https://github.com/yunuskilicdev/aiSeries/tree/main/01-basic-chatbot</pre><p>The repo includes:</p><ul><li>All 5 versions of the chatbot</li><li>A requirements.txt file</li><li>Troubleshooting guide</li><li>Additional example prompts</li></ul><h3>Homework Challenge</h3><p>Before the next article, try customizing your chatbot:</p><ul><li><strong>Easy:</strong> Create a new persona (Chef, Detective, Alien, etc.)</li><li><strong>Medium:</strong> Add a “Save Chat” button that exports conversation to a text file</li><li><strong>Hard:</strong> Add a dropdown to switch between different models (llama3.2, mistral, phi3)</li></ul><p>See you in the next article where we’ll make this chatbot much smarter with RAG! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9667878941b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/building-your-first-ai-chatbot-with-ollama-and-gradio-e9667878941b">Building Your First AI Chatbot with Ollama and Gradio</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Kustomize and Confd Notes]]></title>
            <link>https://medium.com/codex/using-kustomize-and-confd-notes-3b0b440eb4b5?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b0b440eb4b5</guid>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Tue, 28 Sep 2021 14:44:58 GMT</pubDate>
            <atom:updated>2021-09-28T15:30:48.814Z</atom:updated>
            <content:encoded><![CDATA[<p>This post was written to collect notes from different sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fX0qRLUcLyvn-iGn.png" /></figure><p>“Kustomize[1] is a standalone tool to customize Kubernetes objects through a kustomization file. Since 1.14, Kubectl also supports the management of Kubernetes objects using a kustomization file. To view Resources found in a directory containing a kustomization file, run the following command”[0]:</p><pre>kubectl kustomize &lt;kustomization_directory&gt;</pre><p>To apply those Resources, run kubectl apply with — kustomize or -k flag:</p><pre>kubectl apply -k &lt;kustomization_directory&gt;</pre><p><strong>Kustomize is used for declarative object configuration.</strong></p><p>What are imperative and declarative object configurations?</p><h3>Imperative object configuration</h3><p>In imperative object configuration, the kubectl command specifies the operation (create, replace, etc.), optional flags, and at least one file name. The file specified must contain a full definition of the object in YAML or JSON format.[2]</p><h3>Examples</h3><p>Create the objects defined in a configuration file:</p><pre>kubectl create -f nginx.yaml</pre><p>Delete the objects defined in two configuration files:</p><pre>kubectl delete -f nginx.yaml -f redis.yaml</pre><h3>Trade-offs</h3><p><strong>Advantages compared to imperative commands</strong>:</p><p>Object configuration can be stored in a source control system such as Git.</p><p>Object configuration can integrate with processes such as reviewing changes before push and audit trails.</p><p>Object configuration provides a template for creating new objects.</p><p><strong>Disadvantages compared to imperative commands</strong>:</p><p>Object configuration requires a basic understanding of the object schema.</p><p>Object configuration requires the additional step of writing a YAML file.</p><p><strong>Advantages compared to declarative object configuration</strong>:</p><p>Imperative object configuration behavior is simpler and easier to understand.</p><p>As of Kubernetes version 1.5, imperative object configuration is more mature.</p><p><strong>Disadvantages compared to declarative object configuration</strong>:</p><p>Imperative object configuration works best on files, not directories.</p><p>Updates to live objects must be reflected in configuration files, or they will be lost during the next replacement.</p><h3>Declarative object configuration</h3><p>When using declarative object configuration, a user operates on object configuration files stored locally, however, the user does not define the operations to be taken on the files. Create, update, and delete operations are automatically detected per object by kubectl. This enables working on directories, where different operations might be needed for different objects.</p><h3>Trade-offs</h3><p>Advantages compared to imperative object configuration:</p><p>Changes made directly to live objects are retained, even if they are not merged back into the configuration files.</p><p>Declarative object configuration has better support for operating on directories and automatically detecting operation types (create, patch, delete) per-object.</p><p>Disadvantages compared to imperative object configuration:</p><p>Declarative object configuration is harder to debug and understand results when they are unexpected.</p><p>Partial updates using diffs create complex merge and patch operations.</p><h3>Kustomize Usage</h3><ul><li>Generating resources from other sources</li><li>Setting cross-cutting fields for resources</li><li>Composing and customizing collections of resources</li></ul><h3>Generating Resources</h3><p>“ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a .properties file or an SSH keyfile. Kustomize has secretGenerator and configMapGenerator, which generate Secret and ConfigMap from files or literals.[0]”</p><h3>configMapGenerator</h3><p>“To generate a ConfigMap from a file, add an entry to the files list in configMapGenerator[0]”</p><pre>$ mkdir kustomizeDemo<br>$ cd kustomizeDemo<br>$ echo &quot;ENV=DEMO&quot; &gt; demo.yaml<br>$ cat &lt;&lt;EOF &gt;./kustomization.yaml<br>configMapGenerator:<br>- name: demo-configmap<br>  files:<br>- demo.yaml<br>  EOF<br>  $ kubectl kustomize ./<br>  apiVersion: v1<br>  data:<br>  demo.yaml: |<br>  ENV=DEMO<br>  kind: ConfigMap<br>  metadata:<br>  name: demo-configmap-47f78c8429<br>  Other approaches:<br>  configMapGenerator:<br>- name: example-configmap-1<br>  envs:<br>    - .env<br>      EOF<br>      cat &lt;&lt;EOF &gt;./kustomization.yaml<br>      configMapGenerator:<br>- name: example-configmap-2<br>  literals:<br>    - FOO=Bar<br>      EOF</pre><p>This is an example deployment that uses a generated ConfigMap:</p><pre># Create a application.properties file<br>cat &lt;&lt;EOF &gt;application.properties<br>FOO=Bar<br>EOF</pre><pre>cat &lt;&lt;EOF &gt;deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-app<br>  labels:<br>    app: my-app<br>spec:<br>  selector:<br>    matchLabels:<br>      app: my-app<br>  template:<br>    metadata:<br>      labels:<br>        app: my-app<br>    spec:<br>      containers:<br>      - name: app<br>        image: my-app<br>        volumeMounts:<br>        - name: config<br>          mountPath: /config<br>      volumes:<br>      - name: config<br>        configMap:<br>          name: example-configmap-1<br>EOF</pre><pre>cat &lt;&lt;EOF &gt;./kustomization.yaml<br>resources:<br>- deployment.yaml<br>configMapGenerator:<br>- name: example-configmap-1<br>  files:<br>  - application.properties<br>EOF</pre><p>Kustomize generates ConfigMap and ConfigMap is mapped to a volume.</p><p>secretGenerator can be found at [0].</p><h3>Setting cross-cutting fields</h3><p>It is quite common to set cross-cutting fields for all Kubernetes resources in a project. Some use cases for setting cross-cutting fields:</p><ul><li>adding the same name prefix or suffix</li><li>Adding the same set of labels</li><li>Adding the same set of annotations</li><li>Setting the same namespace for all Resources</li></ul><p>Here is an example:</p><pre># Create a deployment.yaml<br>cat &lt;&lt;EOF &gt;./deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: nginx-deployment<br>  labels:<br>    app: nginx<br>spec:<br>  selector:<br>    matchLabels:<br>      app: nginx<br>  template:<br>    metadata:<br>      labels:<br>        app: nginx<br>    spec:<br>      containers:<br>      - name: nginx<br>        image: nginx<br>EOF</pre><pre>cat &lt;&lt;EOF &gt;./kustomization.yaml<br>namespace: my-namespace<br>namePrefix: dev-<br>nameSuffix: &quot;-001&quot;<br>commonLabels:<br>  app: bingo<br>commonAnnotations:<br>  oncallPager: 800-555-1212<br>resources:<br>- deployment.yaml<br>EOF</pre><h3>Composing and Customizing Resources</h3><h4>Composing</h4><p>Kustomize supports composition of different resources. The resources field, in the kustomization.yaml file, defines the list of resources to include in a configuration. Set the path to a resource’s configuration file in the resources list. Here is an example of an NGINX application comprised of a Deployment and a Service:</p><pre># Create a deployment.yaml file<br>cat &lt;&lt;EOF &gt; deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-nginx<br>spec:<br>  selector:<br>    matchLabels:<br>      run: my-nginx<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        run: my-nginx<br>    spec:<br>      containers:<br>      - name: my-nginx<br>        image: nginx<br>        ports:<br>        - containerPort: 80<br>EOF</pre><pre># Create a service.yaml file<br>cat &lt;&lt;EOF &gt; service.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: my-nginx<br>  labels:<br>    run: my-nginx<br>spec:<br>  ports:<br>  - port: 80<br>    protocol: TCP<br>  selector:<br>    run: my-nginx<br>EOF</pre><pre># Create a kustomization.yaml composing them<br>cat &lt;&lt;EOF &gt;./kustomization.yaml<br>resources:<br>- deployment.yaml<br>- service.yaml<br>EOF</pre><p>The Resources from kubectl kustomize ./ contain both the Deployment and the Service objects.</p><h4>Customizing</h4><p>Patches can be used to apply different customizations to Resources. Kustomize supports different patching mechanisms through patchesStrategicMerge and patchesJson6902. patchesStrategicMerge is a list of file paths. Each file should be resolved to a strategic merge patch. The names inside the patches must match Resource names that are already loaded. Small patches that do one thing are recommended. For example, create one patch for increasing the deployment replica number and another patch for setting the memory limit.</p><pre># Create a deployment.yaml file<br>cat &lt;&lt;EOF &gt; deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-nginx<br>spec:<br>  selector:<br>    matchLabels:<br>      run: my-nginx<br>  replicas: 2<br>  template:<br>    metadata:<br>      labels:<br>        run: my-nginx<br>    spec:<br>      containers:<br>      - name: my-nginx<br>        image: nginx<br>        ports:<br>        - containerPort: 80<br>EOF</pre><pre># Create a patch increase_replicas.yaml<br>cat &lt;&lt;EOF &gt; increase_replicas.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-nginx<br>spec:<br>  replicas: 3<br>EOF</pre><pre># Create another patch set_memory.yaml<br>cat &lt;&lt;EOF &gt; set_memory.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-nginx<br>spec:<br>  template:<br>    spec:<br>      containers:<br>      - name: my-nginx<br>        resources:<br>          limits:<br>            memory: 512Mi<br>EOF</pre><pre>cat &lt;&lt;EOF &gt;./kustomization.yaml<br>resources:<br>- deployment.yaml<br>patchesStrategicMerge:<br>- increase_replicas.yaml<br>- set_memory.yaml<br>EOF</pre><p>Run kubectl kustomize ./ to view the Deployment:</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: my-nginx<br>spec:<br>  replicas: 3<br>  selector:<br>    matchLabels:<br>      run: my-nginx<br>  template:<br>    metadata:<br>      labels:<br>        run: my-nginx<br>    spec:<br>      containers:<br>      - image: nginx<br>        name: my-nginx<br>        ports:<br>        - containerPort: 80<br>        resources:<br>          limits:<br>            memory: 512Mi</pre><h3>CONFD</h3><p><strong>confd </strong>is a lightweight configuration management tool focused on:</p><p>keeping local configuration files up-to-date using data stored in etcd, consul, dynamodb, redis, vault, zookeeper, aws ssm parameter store or env vars and processing template resources. reloading applications to pick up new config file changes[3].</p><p><strong>AWS SSM</strong></p><p>AWS Systems Manager allows you to centralize operational data from multiple AWS services and automate tasks across your AWS resources. You can create logical groups of resources such as applications, different layers of an application stack, or production versus development environments. With Systems Manager, you can select a resource group and view its recent API activity, resource configuration changes, related notifications, operational alerts, software inventory, and patch compliance status. You can also take action on each resource group depending on your operational needs. Systems Manager provides a central place to view and manage your AWS resources, so you can have complete visibility and control over your operations.[4]</p><p>Confd is related with configuration management. That’s why I am interested in SSM parameter store.</p><p><strong>SSM Parameter Store</strong></p><p>AWS Systems Manager provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords. This allows you to separate your secrets and configuration data from your code. Parameters can be tagged and organized into hierarchies, helping you manage parameters more easily. For example, you can use the same parameter name, “db-string”, with a different hierarchical path, “dev/db-string” or “prod/db-string”, to store different values. Systems Manager is integrated with AWS Key Management Service (KMS), allowing you to automatically encrypt the data you store. You can also control user and resource access to parameters using AWS Identity and Access Management (IAM). Parameters can be referenced through other AWS services, such as Amazon Elastic Container Service, AWS Lambda, and AWS CloudFormation.[4]</p><p>One such service is SSM Parameter Store which is a secured and managed key/value store perfect for storing parameters, secrets, and configuration information. However, in April of 2018, AWS also introduced another service called AWS Secrets Manager that offers similar functionality. Given that both services kind of do the same thing, which to choose isn’t clear. With that in mind, let us take a look at the similarities and differences of these two services to better understand which service will best fit your architectural needs.[5]</p><p>SSM Parameter Store is free, that’s why I prefer it instead of AWS Secrets Manager.</p><pre>aws ssm put-parameter --name &quot;/myapp/database/url&quot; --type &quot;String&quot; --value &quot;db.example.com&quot;<br>aws ssm put-parameter --name &quot;/myapp/database/user&quot; --type &quot;SecureString&quot; --value &quot;rob&quot;</pre><p>Above commands put two parameter to SSM Parameter Store</p><p>The confdir is where template resource configs and source templates are stored.</p><pre>sudo mkdir -p /etc/confd/{conf.d,templates}</pre><p>Template resources are defined in <a href="https://github.com/toml-lang/toml">TOML</a> config files under the confdir.</p><p>/etc/confd/conf.d/myconfig.toml</p><pre>[template]<br>src = &quot;myconfig.conf.tmpl&quot;<br>dest = &quot;/tmp/myconfig.conf&quot;<br>keys = [<br>    &quot;/myapp/database/url&quot;,<br>    &quot;/myapp/database/user&quot;,<br>]</pre><p>Source templates are <a href="https://pkg.go.dev/text/template#pkg-overview">Golang text templates</a>.</p><p>/etc/confd/templates/myconfig.conf.tmpl</p><pre>[myconfig]<br>database_url = {{getv &quot;/myapp/database/url&quot;}}<br>database_user = {{getv &quot;/myapp/database/user&quot;}}</pre><pre>confd -onetime -backend ssm</pre><pre>$ cat /tmp/myconfig.conf <br>[myconfig]<br>database_url = db.example.com<br>database_user = rob</pre><p>0-) <a href="https://github.com/kelseyhightower/confd">https://github.com/kelseyhightower/confd</a></p><p>1-) <a href="https://aws.amazon.com/systems-manager/features/">https://aws.amazon.com/systems-manager/features/</a></p><p>2-) <a href="https://www.1strategy.com/blog/2019/02/28/aws-parameter-store-vs-aws-secrets-manager/">https://www.1strategy.com/blog/2019/02/28/aws-parameter-store-vs-aws-secrets-manager/</a></p><p>3-) <a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/">https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/</a></p><p>4-) <a href="https://github.com/kubernetes-sigs/kustomize">https://github.com/kubernetes-sigs/kustomize</a></p><p>5-) <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/">https://kubernetes.io/docs/concepts/overview/working-with-objects/object-management/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b0b440eb4b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/using-kustomize-and-confd-notes-3b0b440eb4b5">Using Kustomize and Confd Notes</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kubernetes 101 Notes]]></title>
            <link>https://medium.com/codex/kubernetes-101-notes-853fbf2d6d24?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/853fbf2d6d24</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Sat, 28 Aug 2021 07:13:00 GMT</pubDate>
            <atom:updated>2021-09-02T15:57:27.030Z</atom:updated>
            <content:encoded><![CDATA[<h4>This post was written to collect notes from different sources.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*ZYHZhCRYFw_6zmN1i8nErw.png" /></figure><p>“Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.[0]”</p><p>You need to know the below terms at least.</p><h3><strong>Pods</strong></h3><p>The smallest unit that can be deployed in Kubernetes.</p><p>In reality, pods are one or more containers working together to service a part of your system.</p><p>Each pod has a unique IP address.</p><p>Containers in a pod can talk to each other via localhost.</p><p>But mostly each pod contains one container.</p><p><strong>“Note:</strong> Grouping multiple co-located and co-managed containers in a single Pod is a relatively advanced use case. You should use this pattern only in specific instances in which your containers are tightly coupled”[3].</p><h3><strong>Replica Set</strong></h3><p>“A ReplicaSet’s purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods.”</p><p>Introduces state management</p><h3><strong>Deployments</strong></h3><p>Level of abstraction above ReplicaSets</p><p>Deployments create and update ReplicaSets</p><h3><strong>ConfigMaps</strong></h3><p>Used to override container-specific data like</p><ul><li>Configuration files</li><li>Environment variables</li><li>Entire directories of data</li></ul><p>“Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires <strong>strict separation of config from code</strong>. Config varies substantially across deploys, code does not.”[5] That’s why ConfigMap is being used to handle this case.</p><p>Automatically updated within the container when changed</p><p>The best practice is to version ConfigMaps and perform a rolling update</p><p>“ConfigMaps can be mounted as data volumes. ConfigMaps can also be used by other parts of the system, without being directly exposed to the Pod. For example, ConfigMaps can hold data that other parts of the system should use for configuration.”[4]</p><h4>Using ConfigMaps as files from a Pod</h4><p>To consume a ConfigMap in a volume in a Pod:</p><ol><li>Create a ConfigMap or use an existing one. Multiple Pods can reference the same ConfigMap.</li><li>Modify your Pod definition to add a volume under .spec.volumes[]. Name the volume anything, and have a .spec.volumes[].configMap.name field set to reference your ConfigMap object.</li><li>Add a .spec.containers[].volumeMounts[] to each container that needs the ConfigMap. Specify .spec.containers[].volumeMounts[].readOnly = true and .spec.containers[].volumeMounts[].mountPath to an unused directory name where you would like the ConfigMap to appear.</li><li>Modify your image or command line so that the program looks for files in that directory. Each key in the ConfigMap data map becomes the filename under mountPath.</li></ol><p>This is an example of a Pod that mounts a ConfigMap in a volume:</p><pre><strong>apiVersion</strong>: v1<br><strong>kind</strong>: Pod<br><strong>metadata</strong>:<br>  <strong>name</strong>: mypod<br><strong>spec</strong>:<br>  <strong>containers</strong>:<br>  - <strong>name</strong>: mypod<br>    <strong>image</strong>: redis<br>    <strong>volumeMounts</strong>:<br>    - <strong>name</strong>: foo<br>      <strong>mountPath</strong>: &quot;/etc/foo&quot;<br>      <strong>readOnly</strong>: <strong>true</strong><br>  <strong>volumes</strong>:<br>  - <strong>name</strong>: foo<br>    <strong>configMap</strong>:<br>      <strong>name</strong>: myconfigmap</pre><p>“When a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well. The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, the kubelet uses its local cache for getting the current value of the ConfigMap. The type of the cache is configurable using the ConfigMapAndSecretChangeDetectionStrategy field in the <a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/">KubeletConfiguration struct</a>. A ConfigMap can be either propagated by watch (default), ttl-based, or by redirecting all requests directly to the API server. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the Pod can be as long as the kubelet sync period + cache propagation delay, where the cache propagation delay depends on the chosen cache type (it equals to watch propagation delay, ttl of cache, or zero correspondingly).”[4]</p><h3><strong>Services</strong></h3><p>“Defines a DNS entry that can be used to refer to a group of pods</p><p>Provide a consistent endpoint for the group of pods</p><p>Different types ClusterIP, NodePort, LoadBalancer</p><ul><li>ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.</li><li><a href="https://kubernetes.io/docs/concepts/services-networking/service/#nodeport">NodePort</a>: Exposes the Service on each Node&#39;s IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You&#39;ll be able to contact the NodePort Service, from outside the cluster, by requesting &lt;NodeIP&gt;:&lt;NodePort&gt;.</li><li><a href="https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer">LoadBalancer</a>: Exposes the Service externally using a cloud provider&#39;s load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.”</li></ul><h3><strong>Ingresses</strong></h3><p>“Define how traffic outside the cluster is routed to inside the cluster.</p><p>Used to expose Kubernetes services to the world</p><p>Route traffic to internal services based on factors such as host and path</p><p>Ingress is usually implemented by a load balancer(Nginx, HAProxy, etc)”</p><p>Like a Layer 7 Load balancer.</p><p>Ingresses were developed to decrease the load balancer cost in the cloud. If we use a load balancer for each service(Services type -&gt; LoadBalancer), it costs very much. That’s why we are using ingresses.</p><p>I am going to merge resources like this for different topics. The repo:</p><p><a href="https://github.com/yunuskilicdev/notes">GitHub - yunuskilicdev/notes: Collect notes from different sources</a></p><p>0-) <a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/">https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/</a></p><p>1-) <a href="https://www.youtube.com/watch?v=5h1TCrh_hZ0">https://www.youtube.com/watch?v=5h1TCrh_hZ0</a></p><p>2-) <a href="https://www.cncf.io/blog/2019/12/16/kubernetes-101/">https://www.cncf.io/blog/2019/12/16/kubernetes-101/</a></p><p>3-) <a href="https://kubernetes.io/docs/concepts/workloads/pods/">https://kubernetes.io/docs/concepts/workloads/pods/</a></p><p>4-) <a href="https://kubernetes.io/docs/concepts/configuration/configmap/">https://kubernetes.io/docs/concepts/configuration/configmap/</a></p><p>5-) <a href="https://12factor.net/config">https://12factor.net/config</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=853fbf2d6d24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/kubernetes-101-notes-853fbf2d6d24">Kubernetes 101 Notes</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Raft Consensus Algorithm Implementation with Go]]></title>
            <link>https://medium.com/codex/raft-consensus-algorithm-implementation-with-go-d0f9bf4472a0?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0f9bf4472a0</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Tue, 13 Apr 2021 08:08:44 GMT</pubDate>
            <atom:updated>2025-11-03T12:55:23.386Z</atom:updated>
            <content:encoded><![CDATA[<p>“A fundamental problem in distributed computing and multi-agent systems is to achieve overall system reliability in the presence of a number of faulty processes. This often requires coordinating processes to reach <strong>consensus</strong>, or agree on some data value that is needed during computation. Example applications of consensus include agreeing on what transactions to commit to a database in which order, state machine replication, and atomic broadcasts”. [1]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*RD_Rl48nGpprLWxHkSax1g.jpeg" /></figure><p>As you see in the definition, the consensus is the most important part of the distributed applications. In this article, I will describe how to implement a Raft Consensus algorithm implementation with Go. I try to give reasons and hint about implementation and logic. In order to that, I will follow MIT 6.824 course’s lab to pass all tests. [2]</p><p><strong>RAFT</strong></p><p>“Raft is a consensus algorithm for managing a replicated log. It produces a result equivalent to (multi-)Paxos, and it is as efficient as Paxos, but its structure is different from Paxos; this makes Raft more understandable than Paxos and also provides a better foundation for building practical systems.”[3]</p><p><strong>If you need some background about Raft, this visualization will be good for you.[</strong><a href="http://thesecretlivesofdata.com/raft/"><strong>4</strong></a><strong>] Also, you can read its paper.[3]</strong></p><p>Previously Paxos was being used for a long time, but Raft offers the same efficiency with understandability. I agree with the writers. But implementation part still has some corner cases and difficulties. There is a figure call Figure 2 on paper[3], which defines every single detail about Raft implementation. Every word is so much important to complete implementation. If you do not obey these statements properly, your code will be failed easily.</p><blockquote><em>Want to read this story later? </em><a href="https://usejournal.com/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=tech&amp;utm_content=guest_post_read_later_text"><em>Save it in Journal.</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JnGrkfG2NN-xykiPrML8-w.png" /><figcaption>Figure 2 on paper[3]</figcaption></figure><p>Let’s implement Raft with these specifications with detailed explanations if needed. Raft divides concepts into three as Leader Election, Log Replication, and Security. After the leader elected, log replication operations will have occurred.</p><p><strong>Basic Structure And Leader Election</strong></p><p>As you see in the top left part of the figure, the state object has some attributes. Some of these are persistent, others are volatile. So we need to define these fields to construct our object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11fad9e45873435487990ccc39815595/href">https://medium.com/media/11fad9e45873435487990ccc39815595/href</a></iframe><p>Let’s look at the fields detailly.</p><ul><li>currentTerm, votedFor, log, commitIndex, lastApplied, nextIndex, and matchIndex defined and described by the paper in Figure. But I want to add some extra information about the log. In the figure, it is written that the log contains commands. But inside the paper, you should know that command’s term. So I defined it as a struct called CommandTerm.</li><li>There are extra fields that are not stated in the figure, let’s explain these.</li></ul><p><em>Status</em></p><p>As you can see from the below image, the Raft algorithm depends on some states and transitions between states.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*tzZtlGfgyF1jMHnoR1HmfA.png" /><figcaption>State transition figure[3]</figcaption></figure><p><em>ApplyMsg</em></p><p>In order to awareness, committed entries will be written to this chan</p><p><em>LastLogIndex</em></p><p>Just for keeping the last index</p><p><em>LastAccessed</em></p><p>To keep the last time raft object accessed from the leader. Later, we will implement a heartbeat mechanism to avoid unnecessary voting.</p><p>Now, the next step is understanding the state transition management code. In order to that while creating each Raft server start a goroutine called <strong>manageLifecycle</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0798661e14a05c2aafcafb776b7f327a/href">https://medium.com/media/0798661e14a05c2aafcafb776b7f327a/href</a></iframe><p>In the beginning, each server is a follower. Raft uses randomized timers to elect leaders. After a random duration, a follower will try to be the leader with becoming a candidate. While waiting for the duration a heartbeat comes to the follower, the duration will be reset. Otherwise, the raft server increases its current term, changes its state to a candidate.</p><p>After becoming a candidate, an election will be starting soon.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3350a6824193d8bdb5d537edd09fbcfc/href">https://medium.com/media/3350a6824193d8bdb5d537edd09fbcfc/href</a></iframe><p>If a candidate collects the majority of the votes before timeout duration, then elected as a leader. In order to that, we are using RequestVote RPC.</p><p>As stated in Figure:</p><pre><strong>Arguments:</strong><br><strong>term</strong> candidate’s term <br><strong>candidateId</strong> candidate requesting vote <br><strong>lastLogIndex</strong> index of candidate’s last log entry (§5.4) <br><strong>lastLogTerm</strong> term of candidate’s last log entry (§5.4) </pre><pre><strong>Results:</strong> <br><strong>term</strong> currentTerm, for candidate to update itself <br><strong>voteGranted</strong> true means candidate received vote Receiver </pre><pre><strong>implementation:</strong> <br>1. Reply false if term &lt; currentTerm (§5.1) <br>2. If votedFor is null or candidateId, and candidate’s log is at least as up-to-date as receiver’s log, grant vote (§5.2, §5.4)</pre><p>If a candidate could not collect majority votes, becomes a follower after the timeout.</p><p>If a follower returns a term that is bigger than the candidate’s term, then the election also is failed and the candidate becomes a follower. Also, the candidate updates its term before becomes a follower.</p><p>If a candidate wins the election, it becomes a leader. And assign the next index for each server to its last log index + 1.</p><p>Let’s look at Request Vote RPC implementation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7d2dd9297c2448d3c3f7238e0b0d101/href">https://medium.com/media/e7d2dd9297c2448d3c3f7238e0b0d101/href</a></iframe><p>Line9: If the follower’s term is bigger than the candidate, it returns false and the follower’s term.</p><p>Line14: if the argument’s term is bigger, updates the follower’s currentTerm and votedFor. Because follower will return true if votedFor is null or candidateId. When a bigger term comes to followers, updates its term and deletes votedFor.</p><p>Line19: is written for candidate’s log is at least as up-to-date as receiver’s log, grant vote requirement. Checks follower logs are ahead from candidate’s logs, if it is ahead, returns false to the candidate.</p><p>Let’s assume that candidate elected. The next step is the leader state.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/074e3a57af365c15d45e2372d3d230db/href">https://medium.com/media/074e3a57af365c15d45e2372d3d230db/href</a></iframe><p>The leader has multiple responsibilities. Accepting commands from the client, sending commands to peers, committing commands, send heartbeats to the peers. So most complex part of the code is the leading state.</p><ul><li>Upon election: send initial empty AppendEntries RPCs (heartbeat) to each server; repeat during idle periods to prevent election timeouts. To handle this, the leader sends append entries without any entry. When a follower accepts a heartbeat, resets its election timer.</li><li>If command received from client: append entry to local log, respond after entry applied to state machine. Inside start method, command will be applied into leader’s log.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ce9da7674cee466914f81588bfb6154/href">https://medium.com/media/4ce9da7674cee466914f81588bfb6154/href</a></iframe><ul><li>If last log index ≥ nextIndex for a follower: send AppendEntries RPC with log entries starting at nextIndex</li></ul><p>If successful: update nextIndex and matchIndex for follower (§5.3)</p><p>If AppendEntries fails because of log inconsistency: decrement nextIndex and retry.</p><p><strong>Line 55 at RaftLeader.go</strong> is used for this job.</p><pre>if nextIndex[peer] &lt;= lastLogIndex {<br>   args.Entry = rf.log[prevLogIndex+1 : lastLogIndex+1]<br>}</pre><p>If the leader has multiple entries to append follower, then AppendEntriesArgs has multiple entries.</p><p><strong>If successful:</strong></p><pre>rf.nextIndex[peer] = min(rf.nextIndex[peer]+len(args.Entry), rf.lastLogIndex+1)<br>rf.matchIndex[peer] = prevLogIndex + len(args.Entry)</pre><p>The next Index and match index will be increased. In some corner cases, there will be delays, etc so I allow the max next index will be equal to the leader’s lastLogIndex+1.</p><p><strong>If fails:</strong></p><p>On the paper, it says that decrease next index by 1. But for this lab, there are some test cases to check the fastness of the system. To optimize log inconsistency fixing, the instructor of the course offers a method.[6]</p><pre>&quot;the Figure 2 design backs up one entry per RPC -- slow!<br>  lab tester may require faster roll-back<br>  paper outlines a scheme towards end of Section 5.3<br>    no details; here&#39;s my guess; better schemes are possible<br>      Case 1      Case 2       Case 3<br>  S1: 4 5 5       4 4 4        4<br>  S2: 4 6 6 6 or  4 6 6 6  or  4 6 6 6<br>  S2 is leader for term 6, S1 comes back to life, S2 sends AE for last 6<br>    AE has prevLogTerm=6<br>  rejection from S1 includes:<br>    XTerm:  term in the conflicting entry (if any)<br>    XIndex: index of first entry with that term (if any)<br>    XLen:   log length<br>  Case 1 (leader doesn&#39;t have XTerm):<br>    nextIndex = XIndex<br>  Case 2 (leader has XTerm):<br>    nextIndex = leader&#39;s last entry for XTerm<br>  Case 3 (follower&#39;s log is too short):<br>    nextIndex = XLen&quot;</pre><p><strong>Line 76 at RaftLeader</strong> and later are used for faster rollback. We are going to look AppendEntries side very soon.</p><ul><li>If there exists an N such that N &gt; commitIndex, a majority of matchIndex[i] ≥ N, and log[N].term == currentTerm: set commitIndex = N</li></ul><p><strong>Line 16 at RaftLeader</strong> and below is used to commit agreed entries. It is important that the agreement’s term and the leader’s current term must be equal to commit.</p><p>Now let’s look at another critical method: AppendEntries RPC.</p><pre>1. Reply false if term &lt; currentTerm (§5.1)<br>2. Reply false if log doesn’t contain an entry at prevLogIndex<br>whose term matches prevLogTerm (§5.3)<br>3. If an existing entry conflicts with a new one (same index<br>but different terms), delete the existing entry and all that<br>follow it (§5.3)<br>4. Append any new entries not already in the log<br>5. If leaderCommit &gt; commitIndex, set commitIndex =<br>min(leaderCommit, index of last new entry)</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17d19576b82ed43e0421293613e442b5/href">https://medium.com/media/17d19576b82ed43e0421293613e442b5/href</a></iframe><p>XTerm, XIndex, XLen are used for faster recovery which stated above.</p><p>Line 10 is for rule 1.</p><p>Line 14 is for rule 2.</p><p>Line 45 is for rule 4.</p><p>Line 50 is for rule 5.</p><p>Rule 3 is a tricky one. It causes the TestFigure8Unreliable2C test to fail. In the beginning, I tried to delete the log like below</p><pre>rf<strong>.</strong>Log <strong>=</strong> rf<strong>.</strong>Log[<strong>:</strong>args<strong>.</strong>PrevLogIndex <strong>+</strong> 1]</pre><p>But as stated inside Raft Q&amp;A[5], it is wrong to approach.</p><pre>3. If an existing entry conflicts with a new one (same index<br>but different terms), delete the existing entry and all that<br>follow it</pre><p>We only need to delete conflicting entries. So I changed my code as like below</p><pre>index := 0<br>for ; index &lt; len(args.Entry); index++ {<br>   currentIndex := args.PrevLogIndex + 1 + index<br>   if currentIndex &gt; len(rf.log)-1 {<br>      break<br>   }<br>   if rf.log[currentIndex].Term != args.Entry[index].Term {<br>      rf.log = rf.log[:currentIndex]<br>      rf.lastLogIndex = len(rf.log) - 1<br>      rf.persist()<br>      break<br>   }<br>}<br><br>reply.Success = true<br>rf.lastAccessed = time.Now()<br>if len(args.Entry) &gt; 0 {<br>   rf.log = append(rf.log, args.Entry[index:]...)<br>   rf.lastLogIndex = len(rf.log) - 1<br>   rf.persist()<br>}</pre><p>Just delete after conflicting log, append the remaining part of entries.</p><p>With this refactoring, I am able to pass all test cases in a reasonable amount of time.</p><pre>PASS<br>ok      github.com/yunuskilicdev/distributedsystems/src/raft    209.609s</pre><p>You can find the whole solution at GitHub repo(/src/raft folder)</p><p><a href="https://github.com/yunuskilicdev/distributedsystems/">yunuskilicdev/distributedsystems</a></p><p>This Raft implementation will be very helpful for me. Because I understand how distributed systems implemented, how to fix a bug, and how easily failed. Thanks for the lab assignment structure because it has lots of different test cases with corner cases and different situations.</p><p>In the next article, I will build a fault-tolerant key/value storage service using my Raft library.</p><p><strong>Resources</strong>:</p><p>[1] <a href="https://en.wikipedia.org/wiki/Consensus_(computer_science)">https://en.wikipedia.org/wiki/Consensus_(computer_science)</a></p><p>[2] <a href="http://nil.csail.mit.edu/6.824/2020/labs/lab-raft.html">http://nil.csail.mit.edu/6.824/2020/labs/lab-raft.html</a></p><p>[3] <a href="http://nil.csail.mit.edu/6.824/2020/papers/raft-extended.pdf">http://nil.csail.mit.edu/6.824/2020/papers/raft-extended.pdf</a></p><p>[4] <a href="http://thesecretlivesofdata.com/raft/">http://thesecretlivesofdata.com/raft/</a></p><p>[5] <a href="https://thesquareplanet.com/blog/raft-qa/">https://thesquareplanet.com/blog/raft-qa/</a></p><p>[6] <a href="http://nil.csail.mit.edu/6.824/2020/notes/l-raft2.txt">http://nil.csail.mit.edu/6.824/2020/notes/l-raft2.txt</a></p><p>Photo by <a href="https://unsplash.com/@hannahbusing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Hannah Busing</a> on <a href="https://unsplash.com/s/photos/agreement?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></p><p>📝 Save this story in <a href="https://usejournal.com/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=tech&amp;utm_content=guest_post_read_later_text">Journal</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0f9bf4472a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/raft-consensus-algorithm-implementation-with-go-d0f9bf4472a0">Raft Consensus Algorithm Implementation with Go</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distributed MapReduce Algorithm and Its Go Implementation]]></title>
            <link>https://yunuskilicdev.medium.com/distributed-mapreduce-algorithm-and-its-go-implementation-12273720ff2f?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/12273720ff2f</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[mapreduce]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Tue, 09 Mar 2021 15:49:26 GMT</pubDate>
            <atom:updated>2021-03-11T12:46:22.168Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What is Distributed Computing?</strong></p><p>“Distributed computing is a field of computer science that studies distributed systems. A <em>distributed system</em> is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system. The components interact with one another in order to achieve a common goal.”[1]</p><p>In the 1970s-1980s, Distributed systems were just a research topic at universities. After very large web systems become popular, lots of practitioners inside the software industry, started work on this topic.</p><p>There are some advantages and disadvantages of distributed computing.</p><p><strong>Advantages</strong></p><ul><li>Reliability, high fault tolerance</li><li>Scalability</li><li>Flexibility</li></ul><p><strong>Disadvantages</strong></p><ul><li>Difficult troubleshooting</li><li>Partial failure</li><li>Concurrency</li></ul><p>Making your system fault-tolerant is a more complicated process than a phrase. You have to think and handle very different cases because partial failure occurs anytime. Partial failure makes possible error set enormously bigger than normal. When you need concurrent programming, debugging becomes a painful process. Possible bugs increased also, time allocation per bug increased. So if your system does not require this approach, centralized systems are enough for you. On the other hand, distributed systems will be inevitable in some conditions such as:</p><ul><li>No tolerance for downtime</li><li>Very high load</li><li>Huge data traffic</li><li>Activity varies commonly</li></ul><p>In this article, I will demonstrate one distributed system use case which is called MapReduce. I will describe briefly MapReduce and then how I implemented MapReduce with Go.</p><p><strong>MapReduce</strong></p><p>“MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key”[3]. Many real-world use cases can be solved with this model. You can find it at the referenced link. Basically, it depends on two functions like below:</p><pre>map (k1,v1) → list(k2,v2) <br>reduce (k2,list(v2)) → list(v2)</pre><p>Let me explain with the most known use area of MapReduce is a Word counting system. Input files are split up. The master node creates multiple worker nodes to map these files. The same words were put in the same intermediate file. After the mapping phase, the master node creates multiple reducing workers. The final output is the count of each word that is inside the files. I will show the sequential approach’s code below and finally how I implemented it distributed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XgUEi_iT8Nu7p9Is.png" /><figcaption>MapReduce architecture</figcaption></figure><p>Developing distributed MapReduce is a part of MIT 6.824: Distributed Systems class’ lab projects. So, base codes were taken from that lab. And I completed the project. I am also a learner, the below part shows what I learned so far.</p><p>PS: I am not an MIT student, I found the course while searching. I recommend everybody to learn this topic.</p><p><a href="https://github.com/yunuskilicdev/distributedsystems">yunuskilicdev/distributedsystems</a></p><p>MrWorker, MrMaster, MrSequentials are at “ /src/main” folder.</p><p><strong>The distributed part is located “/src/mr” folder which we need to implement</strong></p><p>Also “src/mrapps” folder contains different types of map&amp;reduce functions.</p><p>For example wc.go contains a map and reduce function for word counter.</p><p>Sequential MapReduce is like below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f7a14783522266a12a1b402335fd9e4b/href">https://medium.com/media/f7a14783522266a12a1b402335fd9e4b/href</a></iframe><p>The mapping and reducing process were done inside one class.</p><p>But this node stops working everything stops. You will not complete the process.</p><p><strong>Let’s start with how to implement distributed MapReduce with Go.</strong></p><p>We need to implement <strong>Master, RPC, Worker</strong> files inside <strong>src/mr</strong> folder.</p><p>We have already MrMaster file that creates the Master node whose role being master as stated at MapReduce architecture above. MrWorker file creates Worker nodes. Testing scripts use MrMaster and MrWorker files to create required nodes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84708193cd6ce41940684cdc3c2104f1/href">https://medium.com/media/84708193cd6ce41940684cdc3c2104f1/href</a></iframe><p>While writing a fault-tolerant, distributed MapReduce algorithm with Go, we will learn lots of important concepts of Go such as <strong>channels, goroutines, mutex</strong>, etc.</p><p><strong>Predefined Conditions:</strong> The master node gives the tasks to workers. It will wait for each worker 10 seconds. If any worker could not finish their job on time, the master will reassign this job to another worker. So the job will be completed even workers crash etc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba2518c154ddc3803d3a0b5eb20057a1/href">https://medium.com/media/ba2518c154ddc3803d3a0b5eb20057a1/href</a></iframe><p>NotStarted, Started, Finished are used for task status.</p><p>MapTask consists of filename and index</p><p>We have 2 channels to control tasks.</p><p>Master struct has information about status, files, etc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98285e74c6f72f3ef8d084f4bd2c4d46/href">https://medium.com/media/98285e74c6f72f3ef8d084f4bd2c4d46/href</a></iframe><p>The existing MrMaster calls the MakeMaster function of the <strong>Master</strong> struct to create the Master node. I initiated status maps and others inside this function. Also, I send each file to our maptasks channel with the filename and its index. All the map tasks will be assigned with respect to this channel. Channel is used for concurrent programming with Go.[4]</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98f20ede2cae8fde95f2ace2890a536a/href">https://medium.com/media/98f20ede2cae8fde95f2ace2890a536a/href</a></iframe><p>In our system, all communication between master and worker will be handle with RPC.[5]</p><p>After seeing how the master was created, let’s look, worker creation and how a worker gets a task from the master.</p><p><strong>Worker.go</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43ce648a63874c1ee064ec83d94119fe/href">https://medium.com/media/43ce648a63874c1ee064ec83d94119fe/href</a></iframe><p><strong>MrWorker</strong> calls worker function of <strong>Worker</strong> struct. The worker asks the <strong>Master</strong> to the current task until the master returns no task. Master either returns a map task or a reduce task. <strong>CallMaster</strong> is a simple RPC call. I will not describe the whole RPC process. In the code base, there is a file called RPC.go which consists of args and reply structs. There is a <strong>call</strong> function inside <strong>Worker.go </strong>is responsible for RPC calls. You can find these details from the GitHub link.</p><p>Also while creating a worker node, map and reduce functions are injected via Go’s plugin mechanism. For example below code inject wordcounter’s map and reduce function. So inside our worker code, we can use mapf and reducef functions injected from another file. There are some ready files inside mrapps folder to test our project.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82938024a40ad45acb220d3eecf4843d/href">https://medium.com/media/82938024a40ad45acb220d3eecf4843d/href</a></iframe><pre>$go build -buildmode=plugin ../mrapps/wc.go<br>$go run mrworker.go wc.so</pre><p><strong>crash.go</strong> is an important file that adds random crashes and waiting to check fault tolerance</p><p><strong>Worker’s mapping code</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c50bce48e0515e9db9fc2021385f8c39/href">https://medium.com/media/c50bce48e0515e9db9fc2021385f8c39/href</a></iframe><p>It maps input with injected map function for example inside <strong>wc.go</strong> map function is like below. Words extracted from the content. mapf function located above is like this because of plugin injection.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54380f3d1028d82d4574da8bdfd979d4/href">https://medium.com/media/54380f3d1028d82d4574da8bdfd979d4/href</a></iframe><p>When all mappings are ready, we need to divide these words into nReduce * file. You can see that intermediate files are more than 1.Then each reduce function collects a set of files and finishes the reducing part. Index of these files found with a hash function. The correct index was chosen via the hash function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4ab37f8ffbd68879fa7e1a4f575bfe9/href">https://medium.com/media/b4ab37f8ffbd68879fa7e1a4f575bfe9/href</a></iframe><p>for the <strong>index 0</strong> of map task for<strong> file0</strong>, if <strong>nReduce =3</strong></p><p>mr-0–0, mr-0–1, mr-0–2 files will be created. Then for example,</p><p>apple will be put into mr-0–0, banana mr-0–1 etc.</p><p>Also, <strong>index 1</strong> of map task does the same process for <strong>file1</strong>,</p><p>apple will be put mr-1–0, banana mr-1–1 etc.</p><p><strong>All apples will be put into mr-*-0, bananas mr-*-1. So counting will work perfectly.</strong></p><p>All intermediate files will be notified with <strong>NotifyMaster </strong>function.</p><p><strong>Worker’s reducing code</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf8060ae2c3dffa04ee01a85d3d551d1/href">https://medium.com/media/cf8060ae2c3dffa04ee01a85d3d551d1/href</a></iframe><p>Let’s look at how Master handles the task distribution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98fb8117d22a7095f8bee6b3eff4eac5/href">https://medium.com/media/98fb8117d22a7095f8bee6b3eff4eac5/href</a></iframe><p>If any Map task exists inside maptasks, the master will take this task from a channel and assign it to a worker.</p><p>While changing status I used mutex lock</p><pre>m.RWMutexLock.Lock()<br>m.mapTaskStatus[mapTask.filename] = <em>Started<br></em>m.RWMutexLock.Unlock()</pre><p>Before returning to the worker, I started a goroutine to monitor the worker if it would finish its job in 10 secs. <strong>go m.watchWorkerMap(mapTask)</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb56e153e4646534350d934155bd8f30/href">https://medium.com/media/cb56e153e4646534350d934155bd8f30/href</a></iframe><p>When a worker finishes the task, it will send a notification to the master which stated the worker’s code part(NotifyMapSuccess function). Map task status of that file was made finished with this notification. After 10 sec, the task status is still not finished, this task was put to maptasks again and the related file’s status becomes NotStarted.</p><p>If all mapping tasks are finished, then reduce task adds into reducetasks channel. After that point master will assign reduce tasks to workers.</p><p>Again reduce tasks will be monitored like below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e8a6b34bf10164a9d02c55f7f659b76/href">https://medium.com/media/6e8a6b34bf10164a9d02c55f7f659b76/href</a></iframe><p>Again a reduce task will not completed in 10 sec, reduce task was put channel again and the status will be NotStarted.</p><p>If m.reduceFinished becomes true, then MrMaster will finish the whole process, because it&#39;s looking master’s done function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc7f8fb9ed3185b71df83304adcd1887/href">https://medium.com/media/bc7f8fb9ed3185b71df83304adcd1887/href</a></iframe><p><strong>How to test it</strong></p><pre>At src/main location</pre><pre>$sh ./test-mr.sh<br>*** Starting wc test.<br>2021/03/09 15:34:19 rpc.Register: method &quot;Done&quot; has 1 input parameters; needs exactly three<br>--- wc test: PASS<br>*** Starting indexer test.<br>2021/03/09 15:34:24 rpc.Register: method &quot;Done&quot; has 1 input parameters; needs exactly three<br>--- indexer test: PASS<br>*** Starting map parallelism test.<br>2021/03/09 15:34:27 rpc.Register: method &quot;Done&quot; has 1 input parameters; needs exactly three<br>--- map parallelism test: PASS<br>*** Starting reduce parallelism test.<br>2021/03/09 15:34:34 rpc.Register: method &quot;Done&quot; has 1 input parameters; needs exactly three<br>--- reduce parallelism test: PASS<br>*** Starting crash test.<br>2021/03/09 15:34:42 rpc.Register: method &quot;Done&quot; has 1 input parameters; needs exactly three<br>--- crash test: PASS<br>*** PASSED ALL TESTS</pre><p><strong>crash</strong> <strong>test</strong> is using crash.go file which includes some random crashes and waiting. You can put some logs to follow status maps, some indexes become Started and then NotStarted again.</p><p>My code passed all tests. But there are some problems inside the code. The problem is race condition. There are some areas that shared variables might be accessed without locking. With go analyzing the race condition is easy. Simple build with code race parameter. There is an option inside test-mr.sh file to open detecting race conditions. So I opened this option.</p><pre>RACE=-race</pre><p>All tests still passed by there are some warnings.</p><pre>==================<br>WARNING: DATA RACE<br>Read at 0x00c000220060 by goroutine 67:<br> github.com/yunuskilicdev/distributedsystems/src/mr.(*Master).NotifyIntermediateFile()<br> /Users/ykdev/go/src/github.com/yunuskilicdev/distributedsystems/src/mr/master.go:110 +0xf8<br>Previous write at 0x00c000220060 by goroutine 65:<br> github.com/yunuskilicdev/distributedsystems/src/mr.(*Master).NotifyIntermediateFile()<br> /Users/ykdev/go/src/github.com/yunuskilicdev/distributedsystems/src/mr/master.go:110 +0x1f5<br>==================</pre><p>Stated function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bf637d92892e3b5363ae784d9525646/href">https://medium.com/media/3bf637d92892e3b5363ae784d9525646/href</a></iframe><p>After locks added:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bd3b3c5b70621f0d778520f59cade6e/href">https://medium.com/media/3bd3b3c5b70621f0d778520f59cade6e/href</a></iframe><p>Another race condition warning:</p><pre>==================</pre><pre>WARNING: DATA RACE</pre><pre>Write at 0x00c0001ce1b9 by goroutine 50:</pre><pre>github.com/yunuskilicdev/distributedsystems/src/mr.(*Master).NotifyReduceSuccess()</pre><pre>/Users/ykdev/go/src/github.com/yunuskilicdev/distributedsystems/src/mr/master.go:146 +0x244</pre><pre>...</pre><pre>Previous read at 0x00c0001ce1b9 by main goroutine:</pre><pre>github.com/yunuskilicdev/distributedsystems/src/mr.(*Master).Done()</pre><pre>/Users/ykdev/go/src/github.com/yunuskilicdev/distributedsystems/src/mr/master.go:171 +0xdc</pre><pre>==================</pre><p>Stated function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc7f8fb9ed3185b71df83304adcd1887/href">https://medium.com/media/bc7f8fb9ed3185b71df83304adcd1887/href</a></iframe><p>After locks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a77a319e6681161fde62b62e7260c1fb/href">https://medium.com/media/a77a319e6681161fde62b62e7260c1fb/href</a></iframe><p>Now race conditions also fixed.</p><p>That all for now. When I learned more, I will try to publish it.</p><p>Resources:</p><p>[1] — <a href="https://en.wikipedia.org/wiki/Distributed_computing">https://en.wikipedia.org/wiki/Distributed_computing</a></p><p>[2] — <a href="https://computernetworktopology.com/distributed-computing/">https://computernetworktopology.com/distributed-computing/</a></p><p>[3] — <a href="http://nil.csail.mit.edu/6.824/2020/papers/mapreduce.pdf">http://nil.csail.mit.edu/6.824/2020/papers/mapreduce.pdf</a></p><p>[4] — <a href="https://tour.golang.org/concurrency/2">https://tour.golang.org/concurrency/2</a></p><p>[5] — <a href="https://en.wikipedia.org/wiki/Remote_procedure_call">https://en.wikipedia.org/wiki/Remote_procedure_call</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12273720ff2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How does Java Application run? What is Bytecode? Write a simple VM simulation]]></title>
            <link>https://medium.com/analytics-vidhya/how-does-java-application-run-what-is-bytecode-write-a-simple-vm-simulation-d31e89c4839a?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/d31e89c4839a</guid>
            <category><![CDATA[bytecode]]></category>
            <category><![CDATA[jit]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[jvm]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Mon, 06 Jul 2020 06:53:16 GMT</pubDate>
            <atom:updated>2020-07-06T14:31:35.590Z</atom:updated>
            <content:encoded><![CDATA[<p>While researching about Java and JVM architecture, I found a meetup talk[1] and other resources. Then I recognized that other developers might be curious about Java’s behind the scene. So I decided to write an article about how Java Application runs? How Java VM run by explaining step by step? In order to discuss this, I collected some background information. Then finally illustrate VM running steps. The sample application is based on that meetup talk Terence Parr[1].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p-qWdalcQtyojQ1P.png" /></figure><p><strong>Background Information</strong></p><p><strong>“Java</strong> is a general-purpose programming language that is class-based, object-oriented, and designed to have as few implementation dependencies as possible. It is intended to let application developers <em>write once, run anywhere</em> (WORA), meaning that compiled Java code can run on all platforms that support Java without the need for recompilation. Java applications are typically compiled to bytecode that can run on any Java virtual machine (JVM) regardless of the underlying computer architecture. The syntax of Java is similar to C and C++, but it has fewer low-level facilities than either of them. As of 2019, Java was one of the most popular programming languages in use according to GitHub, particularly for client-server web applications, with a reported 9 million developers.”[2]</p><p>As stated at the Java definition, once you write your program with Java, it runs anywhere. In order to achieve this goal, Java uses bytecode and JVM. Before starting to discuss Java architecture, let’s see other languages like C.</p><p>After the compiling phase, all objects are being merged and made an executable. This called linking. Output exe is ready to run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*XAIDEhaEo1-CJ868p4i6Og.png" /></figure><p>On the other hand, Java compiler converts to .class files without linking these files. These class files contain another IL called bytecode. But the computer could not read and run bytecode directly. So a virtual machine needs to convert these to computer-executable code. The advantage of this approach is when you create bytecode files, each architecture uses its own JVM and can run Java. That’s how Java runs at every platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*BLY19YrSqYPUngsvMCBJ6g.png" /></figure><p>On the other hand, JIT Compiler(Just-In-Time) interprets this bytecode at runtime. So Java is comparable slower than some languages. Also, this architecture makes Java is both compiled and interpreted language. Because Java compiler compiles java files to generate class files. Then JIT interprets class files to Native computer code.</p><p>Using this background information, we will write our own virtual machine. This virtual machine will be very primitive. But understanding the main concept is more clear. Understanding how bytecode executed is more important than writing simple code.</p><p>Converting Java code to bytecode is Java Compiler’s responsibility. But let’s look without detail. The below code contains the most primitive factorial code.</p><pre><em>/**<br> * </em><strong><em>@author </em></strong><em>Yunus<br> */<br></em>public class Main {<br><br>    static int factorial(int num) {<br>        if (num &lt; 2) {<br>            return 1;<br>        }<br>        return num * <em>factorial</em>(num - 1);<br>    }<br><br>    public static void main(String[] args) {<br>        <em>factorial</em>(2);<br>    }<br>}</pre><p>With below command bytecode will be generated.</p><pre>javac Main.java</pre><p>Let’s look at Main.class file with the below command.</p><pre>$javap -p -c -v Main.class</pre><pre>Last modified Jul 3, 2020; size 378 bytes<br>  MD5 checksum 1928bdcf95625d8714f1aa34b13ec2f2<br>  Compiled from &quot;Main.java&quot;<br>public class Main<br>  minor version: 0<br>  major version: 52<br>  flags: ACC_PUBLIC, ACC_SUPER<br>Constant pool:<br>   #1 = Methodref          #4.#16         // java/lang/Object.&quot;&lt;init&gt;&quot;:()V<br>   #2 = Methodref          #3.#17         // Main.factorial:(I)I<br>   #3 = Class              #18            // Main<br>   #4 = Class              #19            // java/lang/Object<br>   #5 = Utf8               &lt;init&gt;<br>   #6 = Utf8               ()V<br>   #7 = Utf8               Code<br>   #8 = Utf8               LineNumberTable<br>   #9 = Utf8               factorial<br>  #10 = Utf8               (I)I<br>  #11 = Utf8               StackMapTable<br>  #12 = Utf8               main<br>  #13 = Utf8               ([Ljava/lang/String;)V<br>  #14 = Utf8               SourceFile<br>  #15 = Utf8               Main.java<br>  #16 = NameAndType        #5:#6          // &quot;&lt;init&gt;&quot;:()V<br>  #17 = NameAndType        #9:#10         // factorial:(I)I<br>  #18 = Utf8               Main<br>  #19 = Utf8               java/lang/Object<br>{<br>  public Main();<br>    descriptor: ()V<br>    flags: ACC_PUBLIC<br>    Code:<br>      stack=1, locals=1, args_size=1<br>         0: aload_0<br>         1: invokespecial #1                  // Method java/lang/Object.&quot;&lt;init&gt;&quot;:()V<br>         4: return<br>      LineNumberTable:<br>        line 4: 0</pre><pre>static int factorial(int);<br>    descriptor: (I)I<br>    flags: ACC_STATIC<br>    Code:<br>      stack=3, locals=1, args_size=1<br>         0: iload_0<br>         1: iconst_2<br>         2: if_icmpge     7<br>         5: iconst_1<br>         6: ireturn<br>         7: iload_0<br>         8: iload_0<br>         9: iconst_1<br>        10: isub<br>        11: invokestatic  #2                  // Method factorial:(I)I<br>        14: imul<br>        15: ireturn<br>      LineNumberTable:<br>        line 7: 0<br>        line 8: 5<br>        line 10: 7<br>      StackMapTable: number_of_entries = 1<br>        frame_type = 7 /* same */</pre><pre>public static void main(java.lang.String[]);<br>    descriptor: ([Ljava/lang/String;)V<br>    flags: ACC_PUBLIC, ACC_STATIC<br>    Code:<br>      stack=1, locals=1, args_size=1<br>         0: iconst_2<br>         1: invokestatic  #2                  // Method factorial:(I)I<br>         4: pop<br>         5: return<br>      LineNumberTable:<br>        line 14: 0<br>        line 15: 5<br>}<br>SourceFile: &quot;Main.java&quot;</pre><p>Our main function has an instruction like,</p><pre>1: invokestatic #2 // Method factorial:(I)I</pre><p>Basically says that invoke factorial method. The meaning of the #2 can be found inside the constant pool, states our factorial functions and its information. The correct function can be found from the method table.</p><pre>#2 = Methodref          #3.#17</pre><pre>#3 = Class              #18</pre><pre>#17 = NameAndType        #9:#10         // factorial:(I)I<br>#18 = Utf8               Main</pre><pre>#9 = Utf8               factorial<br>#10 = Utf8               (I)I</pre><p>Let’s focus factorial functions’ byte code below:</p><pre>0: iload_0<br>1: iconst_2<br>2: if_icmpge     7<br>5: iconst_1<br>6: ireturn<br>7: iload_0<br>8: iload_0<br>9: iconst_1<br>10: isub<br>11: invokestatic  #2                  // Method factorial:(I)I<br>14: imul<br>15: ireturn</pre><p>So to make a factorial code run, we need to convert this into computer operations. There is a huge list of instructions[5]. But I will only implement only used instructions inside factorial code. With this minimalist approach, it will be easy to understand.</p><p>In the beginning, we have the only stack to memory because we do not have an object, etc.</p><p>Our minimalist VM basically takes instruction and executes it. So in order to keep important information, we need stack pointer(sp), instruction pointer(ip), and frame pointer(fp).</p><p>Stack pointer will be used what is the current position at stack.</p><p>The instruction pointer keeps which instruction executed.</p><p>Frame pointer…</p><p>For example: iload_0, load an int <em>value</em> from local variable 0</p><p>Also, there is an instructor called iload, load an int <em>value</em> from a local variable <em>#index</em></p><p><strong>iload_0</strong> and<strong> iload 0</strong> both of them is true. In order to more generic, I will use iload 0 to load any index. Same for other instructions.</p><p>For simplicity reasons, we will just <strong>call</strong> and number of instruction instead of invokestatic #2</p><p><strong>if_icmpge</strong> =&gt; if <em>value1</em> is greater than or equal to <em>value2</em>, branch to instruction at <em>branchoffset. So I divided these instructions like below(ilt and brf).</em></p><p>We move forward as if following bytecode is generated. If an instruction has operand then the next instruction number changes.</p><pre>Factorial method:</pre><pre>0: iload 0<br>2: iconst 2<br>4: ilt<br>5: brf 10<br>7: iconst 1<br>9: return<br>10: iload 0<br>12: iload 0<br>14: iconst 1<br>16: isub<br>17: call  0,1                  <br>20: imul<br>21: return</pre><pre>Main method:</pre><pre>22: iconst 2<br>24: call 0,1<br>27: print<br>28: halt</pre><p>First of all, let&#39;s define instructions. Code snippets are taken from conference talk as I said at the beginning, I made some refactoring[1]. I want to demonstrate each step to demonstrate how it is working? Actually codes are very simple. Understanding the concept is more important.</p><p>First of all, I create the Instruction class which has only contains instructions that are required for our factorial sample.</p><pre>import java.util.Arrays;<br>import java.util.Optional;<br><em><br></em>public enum Instruction {<br><br>    <em>ILOAD</em>(1, 1),<br>    <em>ICONST</em>(2, 1),<br>    <em>ILT</em>(3, 0),<br>    <em>RET</em>(4, 0),<br>    <em>ISUB</em>(5, 0),<br>    <em>CALL</em>(6, 2),<br>    <em>IMUL</em>(7, 1),<br>    <em>PRINT</em>(8, 0),<br>    <em>HALT</em>(9, 0),<br>    <em>BRF</em>(10, 1);<br><br>    private final int code;<br>    private final int numberOfOperand;<br><br>    Instruction(int code, int numberOfOperands) {<br>        this.code = code;<br>        this.numberOfOperand = numberOfOperands;<br>    }<br><br>    public static Optional&lt;Instruction&gt; findByCode(int code) {<br>        return Arrays.<em>asList</em>(Instruction.<em>values</em>()).stream().filter(x -&gt; x.getCode() == code).findFirst();<br>    }<br><br>    public int getCode() {<br>        return code;<br>    }<br><br>    public int getNumberOfOperand() {<br>        return numberOfOperand;<br>    }<br>}</pre><p>Define Test class, bytecodes as input. And state starts with the main method’s first instruction.</p><pre>public class Test {<br>    static int[] <em>factorial </em>= {<br>            Instruction.<em>ILOAD</em>.getCode(), 0,<br>            Instruction.<em>ICONST</em>.getCode(), 2,<br>            Instruction.<em>ILT</em>.getCode(),<br>            Instruction.<em>BRF</em>.getCode(), 10,<br>            Instruction.<em>ICONST</em>.getCode(), 1,<br>            Instruction.<em>RET</em>.getCode(),<br>            Instruction.<em>ILOAD</em>.getCode(), 0,<br>            Instruction.<em>ILOAD</em>.getCode(), 0,<br>            Instruction.<em>ICONST</em>.getCode(), 1,<br>            Instruction.<em>ISUB</em>.getCode(),<br>            Instruction.<em>CALL</em>.getCode(), 0, 1,<br>            Instruction.<em>IMUL</em>.getCode(),<br>            Instruction.<em>RET</em>.getCode(),</pre><pre><br>            Instruction.<em>ICONST</em>.getCode(), 2,<br>            Instruction.<em>CALL</em>.getCode(), 0, 1,<br>            Instruction.<em>PRINT</em>.getCode(),<br>            Instruction.<em>HALT</em>.getCode()<br>    };<br><br>    public static void main(String[] args) {<br>        Vm vm = new Vm(<em>factorial</em>, 22);<br>        vm.execute();<br>    }<br>}</pre><p>Vm class:</p><pre>public class Vm {<br>    public static final int <em>DEFAULT_STACK_SIZE </em>= 1000;<br>    public static final int <em>NUMBER_OF_GLOBAL </em>= 0;<br>    public static final int <em>FALSE </em>= 0;<br>    public static final int <em>TRUE </em>= 1;<br>    int ip = -1, sp = -1, fp = -1, defaultOffset = -3;<br>    int[] code, globals, stack;<br><br>    public Vm(int[] code, int startip) {<br>        this.code = code;<br>        ip = startip;<br>        globals = new int[<em>NUMBER_OF_GLOBAL</em>];<br>        stack = new int[<em>DEFAULT_STACK_SIZE</em>];<br>    }<br><br>    public void execute() {<br>        int opcode = code[ip];<br>        Instruction instruction = getInstruction(opcode);<br>        Optional&lt;Instruction&gt; optionalInstruction;<br>        if (instruction == null) return;<br>        int a, b, addr, offset;<br><br>        while (opcode != Instruction.<em>HALT</em>.getCode() &amp;&amp; ip &lt; code.length) {<br>            instruction = getInstruction(opcode);<br>            ip++;<br>            switch (instruction) {<br>                case <em>ILOAD</em>:<br>                    offset = defaultOffset + code[ip++];<br>                    stack[++sp] = stack[fp + offset];<br>                    break;<br>                case <em>ICONST</em>:<br>                    stack[++sp] = code[ip++];<br>                    break;<br>                case <em>ILT</em>:<br>                    b = stack[sp--];<br>                    a = stack[sp--];<br>                    stack[++sp] = (a &lt; b) ? <em>TRUE </em>: <em>FALSE</em>;<br>                    break;<br>                case <em>ISUB</em>:<br>                    b = stack[sp--];<br>                    a = stack[sp--];<br>                    stack[++sp] = a - b;<br>                    break;<br>                case <em>CALL</em>:<br>                    addr = code[ip++];<br>                    int nargs = code[ip++];<br>                    stack[++sp] = nargs;<br>                    stack[++sp] = fp;<br>                    stack[++sp] = ip;<br>                    fp = sp;<br>                    ip = addr;<br>                    break;<br>                case <em>RET</em>:<br>                    int rvalue = stack[sp--];<br>                    sp = fp;<br>                    ip = stack[sp--];<br>                    fp = stack[sp--];<br>                    nargs = stack[sp--];<br>                    sp -= nargs;<br>                    stack[++sp] = rvalue;<br>                    break;<br>                case <em>IMUL</em>:<br>                    b = stack[sp--];<br>                    a = stack[sp--];<br>                    stack[++sp] = a * b;<br>                    break;<br>                case <em>PRINT</em>:<br>                    System.<em>out</em>.println(stack[sp--]);<br>                    break;<br>                case <em>BRF</em>:<br>                    addr = code[ip++];<br>                    if (stack[sp--] == <em>FALSE</em>) ip = addr;<br>                    break;<br>            }<br>            opcode = code[ip];<br>        }<br>    }</pre><p>Now the most important part is coming. I want to discuss the state of the stack after each instruction executed to diving into logic.</p><p>Let’s illustrate how to calculate factorial 2.</p><p>Before execution of any bytecode, our stack looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*zqo-ugwHh9WHYpZMTODpsA.png" /></figure><pre>22: iconst 2 // bytecode<br>Instruction.<em>ICONST</em>.getCode(), 2,  // input equilavent</pre><p>This bytecode basically defines an integer constant inside stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*uSxmADVm9jrozF62UubJiw.png" /></figure><p>As you see 2 located at stack. And ip become 24 which means next instruction.</p><pre>24: call 0,1<br>Instruction.<em>CALL</em>.getCode(), 0, 1</pre><p>Current instruction says that call instruction 0 with 1 argument. Calling and returning are the most complicated operations. So I try to give you more details. While calling another function, we need to store important information. This information will be used after returning the call. In our sample application, we store 3 pieces of information.</p><p>1-) Number of arguments (1)</p><p>2-) Current frame pointer (-1)</p><p>3-) Next instruction(27)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*EZQ1xlGsqsEnL3kmJkWy7g.png" /></figure><p>Our ip becomes 0 because the next instruction is 0. Sp is 3. Fp becomes 3 because just before going another function our sp is 3. So Fp of factorial function is 3. Finally, we called our factorial function from the main method. Now let’s iterate with factorial function’s first instruction.</p><pre>0: iload 0<br>Instruction.<em>ILOAD</em>.getCode(), 0,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*iqmULbTF84Jc4TqV3IrQxQ.png" /></figure><p>As you noticed at code while loading data, there is <strong>-3 offset</strong>. Why this exists? Because while calling function, we put 3 extra information about the previous situation. While loading data from our stack, we need to get values with this offset to find the correct value. 3 extra information demonstrated as yellow. Arguments are blue and our local values are green. <strong>iload 0</strong> loads 2 which was pushed as arguments inside the main method. Then pushes stack again to use as a local value. Now ip becomes 2. Let’s move:</p><pre>2: iconst 2<br>Instruction.<em>ICONST</em>.getCode(), 2,</pre><p>Now we come to n&lt;2 part of our functions. In order to that, we need to put value 2 to our stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*c0O9T4Q-LfdG63yWzaDFEA.png" /></figure><pre>4: ilt<br>Instruction.<em>ILT</em>.getCode(),</pre><p>ILT, just pop the last 2 values and execute less than operations. If true then puts 1 to stack else puts 0. 2&lt;2 is false so puts 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*0QSq8vRzOxqy3aIAd2lKhQ.png" /></figure><pre>5: brf 10<br>Instruction.<em>BRF</em>.getCode(), 10,</pre><p>We divide if_icmpge operation to lt and brf. Brf means if false then branch specified index. Brf pops and looks stack whose last element 0. 0 means false, so we need to branch instruction number 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*gVjfxdJDMdiBEkeqDgnPyw.png" /></figure><pre>10: iload 0<br>Instruction.<em>ILOAD</em>.getCode(), 0,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*KPNYU14-ONkd__H3aDfwkg.png" /></figure><pre>12: iload 0<br>Instruction.<em>ILOAD</em>.getCode(), 0,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*FNtxpr7xhjOeG0oF3Nyvqw.png" /></figure><pre>14: iconst 1<br>Instruction.<em>ICONST</em>.getCode(), 1,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*Q_Sm_vLPfC-G_To1f68Vmw.png" /></figure><pre>16: isub<br>Instruction.<em>ISUB</em>.getCode(),</pre><p>Now we are trying to return 2* factorial(<strong>2–1</strong>) make a subs operations. Our program pops the last two values and gets their difference to put stack. So we popped 2 and 1. Then push 1 to stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*-tPWhK40n47deVE6MLlErA.png" /></figure><pre>17: call  0,1<br>Instruction.<em>CALL</em>.getCode(), 0, 1</pre><p>Now, we are making recursive call(2* <strong>factorial(1)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*c3OjmYMcm23YGyso_jLBPA.png" /></figure><p>We put 1 because the number of arguments is 1. Our last fp was 3 so, we push 3. Finally, after 17th instruction, we need to continue with 20th, so push 20. Now we are again beginning of the factorial functions. We will repeat the same operations a while. I will only put images of the current states.</p><pre>0: iload 0<br>Instruction.<em>ILOAD</em>.getCode(), 0,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*D31zoHCPzuXFD1WwiG_ZaQ.png" /></figure><pre>2: iconst 2<br>Instruction.<em>ICONST</em>.getCode(), 2,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*h9gLdtXm5eJk7cW9S-DXvA.png" /></figure><pre>4: ilt<br>Instruction.<em>ILT</em>.getCode(),</pre><p>This time 1&lt;2 is true, so push 1 to stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*9wawez2Q0etzEyp-TYk_VQ.png" /></figure><pre>5: brf 10<br>Instruction.<em>BRF</em>.getCode(), 10,</pre><p>This time the last element is stack is not false, so we are not making branching. We keep continuing with instruction 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*2kdE4ZPQ3xjSmkc_vv8aCA.png" /></figure><pre>7: iconst 1<br>Instruction.<em>ICONST</em>.getCode(), 1,</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*CulA0jiFCmnAk15s4vLtig.png" /></figure><pre>9: return<br>Instruction.<em>RET</em>.getCode(),</pre><p>Another very important instruction is returning. Following steps executed:</p><ul><li>Get return value from the stack. In our case, it is 1.</li><li>Get instruction pointer value from the stack. We need this value to find where to continue. In our case, it is 20. After factorial returns recursive call, we need to apply multiply(which inst#20).</li><li>Get the frame pointer. It is 3.</li><li>After popping this values from the stack. We decreased sp as number of arguments.</li><li>We need to push return value to the stack.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*ZFE3weVWzXUXY1JytKLAXQ.png" /></figure><pre>20: imul<br>Instruction.<em>IMUL</em>.getCode(),</pre><p>Multiplies last two elements inside the stack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*FGT_ZxQ_mgyYhZL5si4C-Q.png" /></figure><pre>21: return<br>Instruction.<em>RET</em>.getCode(),</pre><p>Another return instruction will be executed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*czhTXeh9LQIzf8U7VHei8g.png" /></figure><p>As you see we returned, instruction 27. We are inside the main function again. With local 2 which is return value of the factorial function.</p><pre>27: print<br>Instruction.<em>PRINT</em>.getCode(),</pre><p>As you, there is not any print instruction. But to demonstrate our work, this instruction was added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*uORqwwLJxk2e712yO6wSIw.png" /></figure><pre>28: halt<br>Instruction.<em>HALT</em>.getCode()</pre><p>After halt instruction, our program will be exited.</p><p>This time I researched Bytecode, JVM, and JIT resources and write outputs of these resources. Next time, the <strong>ahead-of-time compilation</strong> (<strong>AOT compilation</strong>) in Java, GraalVM and advantages/disadvantages will be shared. Be connected.</p><p>Cites:</p><p>1-)<a href="https://www.youtube.com/watch?v=OjaAToVkoTw">https://www.youtube.com/watch?v=OjaAToVkoTw</a></p><p>2-)<a href="https://en.wikipedia.org/wiki/Java_(programming_language)">https://en.wikipedia.org/wiki/Java_(programming_language)</a></p><p>3-)<a href="https://www.guru99.com/java-virtual-machine-jvm.html">https://www.guru99.com/java-virtual-machine-jvm.html</a></p><p>4-)<a href="https://www.lohika.com/methods-invocation-inside-of-a-java-virtual-machine#:~:text=JVM%20methods%20invocations,dedicated%20part%20of%20the%20article.">https://www.lohika.com/methods-invocation-inside-of-a-java-virtual-machine#:~:text=JVM%20methods%20invocations,dedicated%20part%20of%20the%20article.</a></p><p>5-)<a href="https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings">https://en.wikipedia.org/wiki/Java_bytecode_instruction_listings</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d31e89c4839a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-does-java-application-run-what-is-bytecode-write-a-simple-vm-simulation-d31e89c4839a">How does Java Application run? What is Bytecode? Write a simple VM simulation</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing Dependencies with Go Modules]]></title>
            <link>https://yunuskilicdev.medium.com/managing-dependencies-with-go-modules-4a6111d641cc?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a6111d641cc</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Fri, 14 Feb 2020 05:03:45 GMT</pubDate>
            <atom:updated>2020-02-14T05:07:19.762Z</atom:updated>
            <content:encoded><![CDATA[<h3>Managing Dependencies with Go Modules v2</h3><p>Inside this post, I will try to explain how to use go modules with a real-world example.</p><p>First of all, managing dependencies becomes confusing while using Go, especially for the developers who come from languages such as Java, C#, etc like me. Whole dependencies will be installed to the src directory inside GOPATH. And other kinds of stuff is very unique for Go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oq1bkb9ZLSCVSssSHs5XTA.png" /></figure><p>“Go 1.11 and 1.12 include preliminary <a href="https://golang.org/doc/go1.11#modules">support for modules</a>, Go’s <a href="https://blog.golang.org/versioning-proposal">new dependency management system</a> that makes dependency version information explicit and easier to manage. This blog post is an introduction to the basic operations needed to get started using modules”[1].</p><p>In this tutorial, I will refactor one of my previous post repository. And you are going to your life become easier.</p><p><a href="https://medium.com/@yunskilic/developing-golang-aws-lambda-functions-with-gorm-amazon-rds-for-postgresql-c5efbcbd0b0d?source=friends_link&amp;sk=3dfc1dc3f20d5b5f4c887a3c392ee8d4">https://medium.com/@yunskilic/developing-golang-aws-lambda-functions-with-gorm-amazon-rds-for-postgresql-c5efbcbd0b0d</a></p><p><strong>1-) Clone and build code</strong></p><pre>$git clone <a href="https://github.com/yunuskilicdev/gormAws">https://github.com/yunuskilicdev/gormAws</a><br>$cd gormAws<br>$rm go.mod go.sum<br>$git checkout cad5bad924572f23a94dbb38af347becae9bf860<br>$go build createUserController/main.go</pre><blockquote>cannot find package “gormAws/db” in any of:</blockquote><blockquote>/usr/local/go/src/gormAws/db (from $GOROOT)</blockquote><blockquote>/Users/XXX/go/src/gormAws/db (from $GOPATH)</blockquote><p>Dependencies were not installed by default. So you need to do lots of manual operations to use the project. But we will use go module. Let’s do it.</p><p><strong>2-)Migrating To Go Modules</strong></p><p>If you use other dependency management strategies like dep, it will be easier to migrate. But inside that repository, I did not use any of them. But it is going to be all right.</p><pre>$go mod init</pre><blockquote>go: cannot determine module path for source directory /Users/XXX/Documents/SourceCode/gormAws (outside GOPATH, module path must be specified)</blockquote><blockquote>Example usage:</blockquote><blockquote>‘go mod init example.com/m’ to initialize a v0 or v1 module</blockquote><blockquote>‘go mod init example.com/m/v2’ to initialize a v2 module</blockquote><blockquote>Run ‘go help mod init’ for more information.</blockquote><p>It gives an error that states we have to state the module path and version of the go module. The most mature and useful version is v2.</p><pre>$go mod init github.com/yunuskilicdev/gormAws/v2</pre><blockquote>go: creating new go.mod: module github.com/yunuskilicdev/gormAws/v2</blockquote><p>But build is still unsuccessful because of the project structure.</p><pre>$go build createUserController/main.go</pre><blockquote>build command-line-arguments: cannot load gormAws/db: malformed module path “gormAws/db”: missing dot in first path element</blockquote><p>Let’s adapt imports with respect to go modules.</p><p>Replace import string:</p><ul><li>gormAws/model</li></ul><p>With:</p><ul><li>github.com/yunuskilicdev/gormAws/v2/model</li></ul><p>Replace import string:</p><ul><li>gormAws/db</li></ul><p>With</p><ul><li>github.com/yunuskilicdev/gormAws/v2/db</li></ul><p>Then build again</p><pre>$go build createUserController/main.go</pre><p>The build was successful. Build others. All of them fine now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1mDXUnjaJ6pAVH4MPdmJuA.png" /></figure><p>I am using GoLand as IDE. Builds are fine but IDE still gives errors. You have to enable go module support like the below screenshot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asnUTblQP-8IYGwn7AGFeg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qlVYBCVbDJHAVspT63Rb9g.png" /></figure><p>So our project was converted to go module v2 :)</p><pre>$ cat go.mo</pre><blockquote>module github.com/yunuskilicdev/gormAws/v2</blockquote><blockquote>go 1.13</blockquote><blockquote>require (</blockquote><blockquote>github.com/aws/aws-lambda-go v1.14.0 // indirect</blockquote><blockquote>github.com/jinzhu/gorm v1.9.12</blockquote><blockquote>github.com/joho/godotenv v1.3.0</blockquote><blockquote>)</blockquote><p>A few commands about go module[2]:</p><pre>go mod graph</pre><p>Graph prints the module requirement graph (with replacements applied) in text form</p><pre>go mod tidy</pre><p>Tidy makes sure go.mod matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages. It also adds any missing entries to go.sum and removes any unnecessary ones.</p><pre>go mod verify</pre><p>Verify checks that the dependencies of the current module, which are stored in a local downloaded source cache, have not been modified since being downloaded</p><pre>go mod why</pre><p>Why shows a shortest path in the import graph from the main module to each of the listed packages.</p><p>Thanks for reading :)</p><p>Cites:</p><p>1-) <a href="https://blog.golang.org/using-go-modules">https://blog.golang.org/using-go-modules</a></p><p>2-)<a href="https://golang.org/cmd/go/#hdr-Module_maintenance">https://golang.org/cmd/go/#hdr-Module_maintenance</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a6111d641cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is gRPC? A Brief Introduction with JAVA]]></title>
            <link>https://medium.com/analytics-vidhya/what-is-grpc-a-brief-introduction-with-java-1e22f91a1271?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e22f91a1271</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Wed, 22 Jan 2020 11:46:56 GMT</pubDate>
            <atom:updated>2020-01-24T10:28:21.874Z</atom:updated>
            <content:encoded><![CDATA[<p>“gRPC is a modern open source high performance RPC framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication. It is also applicable in last mile of distributed computing to connect devices, mobile applications and browsers to backend services.”[1]</p><p>Cloud Native Computing Foundation is the new home for <a href="http://www.grpc.io/">gRPC</a> and its existing ecosystem projects (<a href="https://github.com/grpc">https://github.com/grpc</a> and <a href="https://github.com/grpc-ecosystem">https://github.com/grpc-ecosystem</a>)</p><p>There are some comparisons with Rest, SOAP etc. In this tutorial I will not make a comparison. But you can find some of them <a href="https://www.mertech.com/blog/know-your-api-protocols">1</a>,<a href="https://doc.akka.io/docs/akka-grpc/current/whygrpc.html">2</a>,<a href="https://www.yonego.com/nl/why-milliseconds-matter/#gref">3</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*6ufkIOzURh0WBg_q9jrSgQ.png" /></figure><p>Outline of the tutorial:</p><p>1-) Define a service in a .proto file using Protocol Buffers IDL</p><p>2-) Generate server</p><p>3-) Client stub code using the protocol buffer compiler.</p><p><strong>1-)Defining a service with proto file</strong></p><p>By default, gRPC uses protocol buffers, Google’s mature open source mechanism for serializing structured data.</p><p>“Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data — think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update your data structure without breaking deployed programs that are compiled against the “old” format.”[2]</p><p><strong>!Important!: </strong>There is a basic tutorial for Java at the gRPC site[5]. But documentation is not fully covered the whole process, especially you are a newbie. There are some assumptions that you are very familiar with the concept. I will mention extra steps to understand concepts and how to solve problems.</p><p>Sample proto file with version 3.[4]</p><pre>syntax = &quot;proto3&quot;;</pre><pre>package tutorial;</pre><pre>import &quot;google/protobuf/timestamp.proto&quot;;<br><br>option java_multiple_files = true;<br>option java_package = &quot;com.example.tutorial&quot;;<br>option java_outer_classname = &quot;AddressGuideProtos&quot;;<br>service AddressGuide {<br>    rpc GetAddressBook(Person) returns (AddressBook) {}<br>}<br>message Person {<br>  string name = 1;<br>  int32 id = 2;  // Unique ID number for this person.<br>  string email = 3;<br>enum PhoneType {<br>    MOBILE = 0;<br>    HOME = 1;<br>    WORK = 2;<br>  }<br>message PhoneNumber {<br>    string number = 1;<br>    PhoneType type = 2;<br>  }<br>repeated PhoneNumber phones = 4;<br>google.protobuf.Timestamp last_updated = 5;<br>}<br>// Our address book file is just one of these.<br>message AddressBook {<br>  repeated Person people = 1;<br>}</pre><p>This proto file will be used to generate sources by the compiler. Before going on to the compiler phase, let’s describe some options.</p><blockquote>option java_multiple_files = true;</blockquote><p>Create objects inside different files.</p><blockquote>option java_package = &quot;com.example.tutorial&quot;;</blockquote><p>Setting Java package name.</p><ul><li><strong>Downloading Protocol Buffer and Setup Compiler</strong></li></ul><p><a href="https://github.com/protocolbuffers/protobuf/releases">Releases · protocolbuffers/protobuf</a></p><p>Download the latest file. You can download programming language-specific file or all which contains whole supported language. I used all version.</p><p>Supported languages: C++,Java, Go, C#, Python, Objective-C, JavaScript, Ruby, Php, Dart</p><p>Unix Installation(Windows installation inside Github repo readme):</p><pre>./configure<br>make<br>make check<br>sudo make install<br>sudo ldconfig</pre><p>You can find sample command to generate resources at Github repo.</p><pre>protoc --java_out=${OUTPUT_DIR} path/to/your/proto/file</pre><p>Java documentation which I cited earlier, has an expression below.</p><blockquote>“The following classes are generated from our service definition:</blockquote><blockquote>Feature.java, Point.java, Rectangle.java, and others which contain all the protocol buffer code to populate, serialize, and retrieve our request and response message types.</blockquote><blockquote>RouteGuideGrpc.java which contains (along with some other useful code):</blockquote><blockquote>a base class for RouteGuide servers to implement, RouteGuideGrpc.RouteGuideImplBase, with all the methods defined in the RouteGuide service.</blockquote><blockquote><em>stub</em> classes that clients can use to talk to a RouteGuide server.”</blockquote><p><strong>!!!</strong>The documentation uses a different example: <em>RouteGuide</em>. And says that <em>RouteGuideGrpc</em> should be generated. The above command will not generate <em>RouteGuideGrpc</em> for any case. Also, AddressGuideGrpc will not be generated.But the official document says <strong>servers should implement generated base class(</strong><em>RouteGuideImplBase</em><strong>) which is located inside the Grpc file</strong>. With respect to your proto file, there should be a base class with all method definitions. There is not any information about how this file generated at the documentation. You could be stuck at this step easily. After some research result is:</p><ul><li>You need to use <strong>protoc plugin</strong> to generate this important file.</li></ul><p><a href="https://stackoverflow.com/questions/31029675/protoc-not-generating-service-stub-files/31031346#31031346">protoc not generating service stub files</a></p><p>Repo of the plugin which describes where and how to download:</p><p><a href="https://github.com/grpc/grpc-java/tree/master/compiler">grpc/grpc-java</a></p><p>So my final command to generate all sources below:</p><pre>protoc -plugin=protoc-gen-grpc-java=/PATH/protoc-gen-grpc-java-1.26.0-osx-x86_64.exe SOURCE_DIST -java_out=DEST_DIR -I=DEST_DIR    -grpc-java_out=DEST_DIR</pre><blockquote>-plugin=protoc-gen-grpc-java</blockquote><p>to generate XXImplBase class.</p><p>PS: You need to give access to the executable plugin on macos.</p><p><a href="https://stackoverflow.com/questions/55023435/grpc-out-protoc-gen-grpc-plugin-failed-with-status-code-1-on-osx-java">https://stackoverflow.com/questions/55023435/grpc-out-protoc-gen-grpc-plugin-failed-with-status-code-1-on-osx-java</a></p><p>Sources were generated. Model classes and AddressGuideGrpc.java are both ready. The next step is creating a maven project. Copy sources to the project. And add below dependencies.</p><pre>&lt;dependency&gt;<br>    &lt;groupId&gt;com.google.protobuf&lt;/groupId&gt;<br>    &lt;artifactId&gt;protobuf-java&lt;/artifactId&gt;<br>    &lt;version&gt;3.11.0&lt;/version&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.grpc&lt;/groupId&gt;<br>    &lt;artifactId&gt;grpc-stub&lt;/artifactId&gt;<br>    &lt;version&gt;1.26.0&lt;/version&gt;<br>&lt;/dependency&gt;<br>&lt;dependency&gt;<br>    &lt;groupId&gt;io.grpc&lt;/groupId&gt;<br>    &lt;artifactId&gt;grpc-protobuf&lt;/artifactId&gt;<br>    &lt;version&gt;1.26.0&lt;/version&gt;<br>&lt;/dependency&gt;</pre><p>Github link for full code:</p><p><a href="https://github.com/yunuskilicdev/grpcdemo">yunuskilicdev/grpcdemo</a></p><p><strong>2-) gRPC Server</strong></p><pre>package com.example.tutorial;<br><br>import io.grpc.Server;<br>import io.grpc.ServerBuilder;<br>import io.grpc.stub.StreamObserver;<br><br>import java.io.IOException;<br>import java.util.ArrayList;<br>import java.util.Collection;<br>import java.util.concurrent.TimeUnit;<br>import java.util.logging.Logger;<br><br>public class GrpcServer {<br>   private static final Logger <em>logger </em>= Logger.<em>getLogger</em>(GrpcServer.class.getName());<br>   <br>   private final int port;<br>   private final Server server;<br>   <br>   public GrpcServer(int port) {<br>      this(ServerBuilder.<em>forPort</em>(port), port);<br>   }<br>   <br>   public GrpcServer(ServerBuilder&lt;?&gt; serverBuilder, int port) {<br>      this.port = port;<br>      ArrayList&lt;AddressBook&gt; addressBooks = new ArrayList();<br>      server = serverBuilder.addService(new AddressGuideService(addressBooks))<br>                   .build();<br>   }<br>   <br>   public static void main(String[] args) throws IOException, InterruptedException {<br>      GrpcServer server = new GrpcServer(8980);<br>      server.start();<br>      server.blockUntilShutdown();<br>   }<br>   <br>   private void blockUntilShutdown() throws InterruptedException {<br>      if (server != null) {<br>         server.awaitTermination();<br>      }<br>   }<br>   <br>   private void start() throws IOException {<br>      server.start();<br>      <em>logger</em>.info(&quot;Server started, listening on &quot; + port);<br>      Runtime.<em>getRuntime</em>().addShutdownHook(new Thread() {<br>         @Override<br>         public void run() {<br>            // Use stderr here since the logger may have been reset by its JVM shutdown hook.<br>            System.<em>err</em>.println(&quot;*** shutting down gRPC server since JVM is shutting down&quot;);<br>            try {<br>               GrpcServer.this.stop();<br>            } catch (InterruptedException e) {<br>               e.printStackTrace(System.<em>err</em>);<br>            }<br>            System.<em>err</em>.println(&quot;*** server shut down&quot;);<br>         }<br>      });<br>   }<br>   <br>   private void stop() throws InterruptedException {<br>      if (server != null) {<br>         server.shutdown().awaitTermination(30, TimeUnit.<em>SECONDS</em>);<br>      }<br>   }<br>   <br>   private static class AddressGuideService extends AddressGuideGrpc.AddressGuideImplBase {<br>      private final Collection&lt;AddressBook&gt; addressbookCollection;<br>      <br>      public AddressGuideService(Collection&lt;AddressBook&gt; addressBookCollection) {<br>         this.addressbookCollection = addressBookCollection;<br>      }<br>      <br>      @Override<br>      public void getAddressBook(Person request, StreamObserver&lt;AddressBook&gt; responseObserver) {<br>         responseObserver.onNext(getAddressBook());<br>         responseObserver.onCompleted();<br>      }<br>      <br>      private AddressBook getAddressBook() {<br>         return AddressBook.<em>newBuilder</em>().addPeople(Person.<em>newBuilder</em>().build()).build();<br>      }<br>   }<br>}</pre><blockquote>AddressGuideService extends AddressGuideGrpc.AddressGuideImplBase</blockquote><p>as I cited earlier, we need to extend ImplBase class which was generated from our proto. We need to override services. In our case, I returned an empty object basically.</p><p>Start, stop and blockUntilShutdown parts are the same with gRPC documentation. Our server will listen to 8980 port.</p><p><strong>3-) gRPC Client</strong></p><pre>package com.example.tutorial;<br><br>import io.grpc.ManagedChannel;<br>import io.grpc.ManagedChannelBuilder;<br><br>import java.util.concurrent.TimeUnit;<br>import java.util.logging.Logger;<br><br>public class GrpcClient {<br>   <br>   private static final Logger <em>logger </em>= Logger.<em>getLogger</em>(GrpcClient.class.getName());<br>   <br>   private final ManagedChannel channel;<br>   private final AddressGuideGrpc.AddressGuideBlockingStub blockingStub;<br>   private final AddressGuideGrpc.AddressGuideStub asyncStub;<br>   <br>   public GrpcClient(String host, int port) {<br>      this(ManagedChannelBuilder.<em>forAddress</em>(host, port).usePlaintext());<br>   }<br>   <br>   public GrpcClient(ManagedChannelBuilder&lt;?&gt; channelBuilder) {<br>      channel = channelBuilder.build();<br>      blockingStub = AddressGuideGrpc.<em>newBlockingStub</em>(channel);<br>      asyncStub = AddressGuideGrpc.<em>newStub</em>(channel);<br>   }<br>   <br>   public static void main(String[] args) throws InterruptedException {<br>      GrpcClient grpcClient = null;<br>      try{<br>         grpcClient = new GrpcClient(&quot;127.0.0.1&quot;, 8980);<br>         AddressBook addressBook = grpcClient.getAddressBook(Person.<em>newBuilder</em>().build());;<br>      }catch (Exception e){<br>      <br>      }finally {<br>         grpcClient.shutdown();<br>      }<br>   }<br>   <br>   public void shutdown() throws InterruptedException {<br>      channel.shutdown().awaitTermination(5, TimeUnit.<em>SECONDS</em>);<br>   }<br>   <br>   public AddressBook getAddressBook(Person person) {<br>      return blockingStub.getAddressBook(person);<br>   }<br>}</pre><p>After starting gRPC server, run the client and see the expected result.</p><p>PS: After shutdown await termination is suggested.</p><p>All done</p><p>Thanks :)</p><p>Cites:</p><p>1-) <a href="https://grpc.io/">https://grpc.io/</a></p><p>2-) <a href="https://developers.google.com/protocol-buffers/docs/overview">https://developers.google.com/protocol-buffers/docs/overvie</a>w</p><p>3-) <a href="https://cloud.google.com/apis/design/proto3">https://cloud.google.com/apis/design/proto3</a></p><p>4-) <a href="https://github.com/protocolbuffers/protobuf/blob/master/examples/addressbook.proto">https://github.com/protocolbuffers/protobuf/blob/master/examples/addressbook.proto</a></p><p>5-) <a href="https://grpc.io/docs/tutorials/basic/java/">https://grpc.io/docs/tutorials/basic/java/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e22f91a1271" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/what-is-grpc-a-brief-introduction-with-java-1e22f91a1271">What is gRPC? A Brief Introduction with JAVA</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Develop Infrastructure as Code with CloudFormation and SAM]]></title>
            <link>https://medium.com/analytics-vidhya/develop-infrastructure-as-code-with-cloudformation-and-sam-5fda322784f5?source=rss-6377db65d6d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fda322784f5</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Yunus Kılıç]]></dc:creator>
            <pubDate>Mon, 30 Dec 2019 06:35:28 GMT</pubDate>
            <atom:updated>2019-12-30T08:42:00.490Z</atom:updated>
            <content:encoded><![CDATA[<h3>Develop Infrastructure as Code with CloudFormation</h3><p>The software development industry changes very rapidly. After DevOps term becomes very popular, only writing code is not enough anymore. You should automate the software development process as can as possible. In this post, I will describe how to define our infrastructure by just writing code.</p><p><em>“Infrastructure as code is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools”.[1]</em></p><p>Why do we need infrastructure as code?</p><ul><li>After moving a containerized world, infrastructure can be changed or delete/create more often.</li><li>Also, keeping track of infrastructure setting becomes very hard, because of tons of different settings. So keeping these settings inside code, helps to figure out huge complexity.</li></ul><p>In this post, I will use CloudFormation to define infrastructure.</p><p>“AWS CloudFormation provides a common language for you to model and provision AWS and third party application resources in your cloud environment. AWS CloudFormation allows you to use programming languages or a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. This gives you a single source of truth for your AWS and third party resources.”[2]</p><p>I drew an architecture diagram below, to demonstrate the final.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mz4ofbJb7DbVUoJGigTZdw.png" /></figure><p>Let’s define the components which exists inside the diagram.</p><p>Region: are separate geographic areas that AWS uses to house its infrastructure. These are distributed around the world so that customers can choose a region closest to them in order to host their cloud infrastructure there.[3]</p><p>Availability Zone: is the logical building block that makes up an AWS Region. There are currently 69 AZs, which are isolated locations — data centers — within a region.[3]</p><p>VPC: enables you to launch AWS resources into a virtual network that you’ve defined. This virtual network closely resembles a traditional network that you’d operate in your own data center, with the benefits of using the scalable infrastructure of AWS.[4]</p><p>Subnet: is “part of the network”, in other words, part of entire availability zone. Each subnet must reside entirely within one Availability Zone and cannot span zones.[5]</p><p>For simplicity, I will create 1 VPC, 2 public subnets, 2 private subnets. Subnets stand different availability zones.</p><p>PS: Deploying code commands</p><pre>$sam package --template-file template.yaml --s3-bucket BUCKETNAME --output-template-file packaged.yaml<br>$sam deploy --template-file FILEPATH/packaged.yaml --stack-name test --capabilities CAPABILITY_IAM</pre><p>First of all, let’s define parameters like below to use later.</p><pre>Parameters:<br>  EnvironmentName:<br>    Description: An environment name that is prefixed to resource names<br>    Type: String<br>    Default: Test<br><br>  VpcCIDR:<br>    Description: Please enter the IP range (CIDR notation) for this VPC<br>    Type: String<br>    Default: 10.0.0.0/16<br><br>  PublicSubnet1CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone<br>    Type: String<br>    Default: 10.0.1.0/24<br><br>  PublicSubnet2CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone<br>    Type: String<br>    Default: 10.0.2.0/24<br><br>  PrivateSubnet1CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone<br>    Type: String<br>    Default: 10.0.3.0/24<br><br>  PrivateSubnet2CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone<br>    Type: String<br>    Default: 10.0.4.0/24</pre><p>PS: 10.0.0.0/16 : Each number represents 8 digit and the first 16 digits (first two numbers) stays the same. The remaining numbers can be changed. So 256*256 Ip can be assigned.</p><p><strong>CREATE VPC</strong></p><pre>VPC:<br>  Type: AWS::EC2::VPC<br>  Properties:<br>    CidrBlock: !Ref VpcCIDR<br>    EnableDnsSupport: true<br>    EnableDnsHostnames: true<br>    Tags:<br>      - Key: Name<br>        Value: !Ref EnvironmentName</pre><p>Let’s connect this VPC with the internet. Internet Gateway is used to access internet from VPC.</p><pre>InternetGateway:<br>  Type: AWS::EC2::InternetGateway<br>  Properties:<br>    Tags:<br>      - Key: Name<br>        Value: !Ref EnvironmentName<br><br>InternetGatewayAttachment:<br>  Type: AWS::EC2::VPCGatewayAttachment<br>  Properties:<br>    InternetGatewayId: !Ref InternetGateway<br>    VpcId: !Ref VPC</pre><p><strong>CREATING SUBNETS</strong></p><pre>PublicSubnet1:<br>  Type: AWS::EC2::Subnet<br>  Properties:<br>    VpcId: !Ref VPC<br>    AvailabilityZone: !Select [ 0, !GetAZs &#39;&#39; ]<br>    CidrBlock: !Ref PublicSubnet1CIDR<br>    MapPublicIpOnLaunch: true<br>    Tags:<br>      - Key: Name<br>        Value: !Sub ${EnvironmentName} Public Subnet (AZ1)<br><br>PublicSubnet2:<br>  Type: AWS::EC2::Subnet<br>  Properties:<br>    VpcId: !Ref VPC<br>    AvailabilityZone: !Select [ 1, !GetAZs  &#39;&#39; ]<br>    CidrBlock: !Ref PublicSubnet2CIDR<br>    MapPublicIpOnLaunch: true<br>    Tags:<br>      - Key: Name<br>        Value: !Sub ${EnvironmentName} Public Subnet (AZ2)<br><br>PrivateSubnet1:<br>  Type: AWS::EC2::Subnet<br>  Properties:<br>    VpcId: !Ref VPC<br>    AvailabilityZone: !Select [ 0, !GetAZs  &#39;&#39; ]<br>    CidrBlock: !Ref PrivateSubnet1CIDR<br>    MapPublicIpOnLaunch: false<br>    Tags:<br>      - Key: Name<br>        Value: !Sub ${EnvironmentName} Private Subnet (AZ1)<br><br>PrivateSubnet2:<br>  Type: AWS::EC2::Subnet<br>  Properties:<br>    VpcId: !Ref VPC<br>    AvailabilityZone: !Select [ 1, !GetAZs  &#39;&#39; ]<br>    CidrBlock: !Ref PrivateSubnet2CIDR<br>    MapPublicIpOnLaunch: false<br>    Tags:<br>      - Key: Name<br>        Value: !Sub ${EnvironmentName} Private Subnet (AZ2)</pre><p>PublicSubnet1 stands on the first of the Availability Zone.</p><p>PublicSubnet2 stands on second.</p><p>PrivateSubnet1 stands on the first of the Availability Zone.</p><p>PrivateSubnet2 stands on second.</p><p>Public subnets should be connected to the internet. In order to handle this, we need a routing table.</p><pre>PublicRouteTable:<br>  Type: AWS::EC2::RouteTable<br>  Properties:<br>    VpcId: !Ref VPC<br>    Tags:<br>      - Key: Name<br>        Value: !Sub ${EnvironmentName} Public Routes<br><br>DefaultPublicRoute:<br>  Type: AWS::EC2::Route<br>  DependsOn: InternetGatewayAttachment<br>  Properties:<br>    RouteTableId: !Ref PublicRouteTable<br>    DestinationCidrBlock: 0.0.0.0/0<br>    GatewayId: !Ref InternetGateway<br><br>PublicSubnet1RouteTableAssociation:<br>  Type: AWS::EC2::SubnetRouteTableAssociation<br>  Properties:<br>    RouteTableId: !Ref PublicRouteTable<br>    SubnetId: !Ref PublicSubnet1<br><br>PublicSubnet2RouteTableAssociation:<br>  Type: AWS::EC2::SubnetRouteTableAssociation<br>  Properties:<br>    RouteTableId: !Ref PublicRouteTable<br>    SubnetId: !Ref PublicSubnet2</pre><p>Any request with related public subnet routed to the internet gateway.</p><p>PS: If you want to do the same thing for the private subnets, you should NAT Gateway.</p><p><strong>CREATING ELASTICACHE INSTANCE WITH CODE</strong></p><pre>ServerlessSecurityGroup:<br>  DependsOn: VPC<br>  Type: AWS::EC2::SecurityGroup<br>  Properties:<br>    GroupDescription: SecurityGroup for Serverless Functions<br>    VpcId:<br>      Ref: VPC<br><br>ServerlessStorageSecurityGroup:<br>  DependsOn: VPC<br>  Type: AWS::EC2::SecurityGroup<br>  Properties:<br>    GroupDescription: Ingress for Redis Cluster<br>    VpcId:<br>      Ref: VPC<br>    SecurityGroupIngress:<br>      - IpProtocol: tcp<br>        FromPort: &#39;6379&#39;<br>        ToPort: &#39;6379&#39;<br>        SourceSecurityGroupId:<br>          Ref: ServerlessSecurityGroup<br><br>ServerlessCacheSubnetGroup:<br>  Type: AWS::ElastiCache::SubnetGroup<br>  Properties:<br>    Description: &quot;Cache Subnet Group&quot;<br>    SubnetIds:<br>      - Ref: PrivateSubnet1<br><br>ElasticCacheCluster:<br>  DependsOn: ServerlessStorageSecurityGroup<br>  Type: AWS::ElastiCache::CacheCluster<br>  Properties:<br>    AutoMinorVersionUpgrade: true<br>    Engine: redis<br>    CacheNodeType: cache.t2.micro<br>    NumCacheNodes: 1<br>    VpcSecurityGroupIds:<br>      - &quot;Fn::GetAtt&quot;: ServerlessStorageSecurityGroup.GroupId<br>    CacheSubnetGroupName:<br>      Ref: ServerlessCacheSubnetGroup</pre><p>Due to security reasons, I put Elasticache instance inside a private subnet. So we need to create a security group to allow 6379 port to connect Redis.</p><p><strong>CONNECT REDIS FROM LAMBDA FUNCTION</strong></p><pre>CacheClientFunction:<br>  Type: AWS::Serverless::Function<br>  Properties:<br>    Tracing: Active<br>    CodeUri: bin/cacheClient<br>    Handler: cacheClient<br>    Runtime: go1.x<br>    Role: !GetAtt RootRole.Arn<br>    VpcConfig:<br>      SecurityGroupIds:<br>        - Ref: ServerlessSecurityGroup<br>      SubnetIds:<br>        - Ref: PublicSubnet1<br>    Environment:<br>      Variables:<br>        redis_url: !GetAtt ElasticCacheCluster.RedisEndpoint.Address<br>        redis_port: !GetAtt ElasticCacheCluster.RedisEndpoint.Port</pre><p>Lambda function stands on the public subnet. We can define redis_url and redis_port according to the created ElasticCache by our definition. While writing code, we can use these environment variables to connect Redis.</p><p><strong>!!!Important!!!</strong></p><p>Lambda function requires some role and policy staff. Otherwise creating stack will be got an error. Let’s create a role and policy with code.</p><pre>SampleManagedPolicy:<br>  Type: AWS::IAM::ManagedPolicy<br>  Properties:<br>    PolicyDocument:<br>      Version: &#39;2012-10-17&#39;<br>      Statement:<br>        - Sid: AllowAllUsersToListAccounts<br>          Effect: Allow<br>          Action:<br>            - ec2:CreateNetworkInterface<br>            - ec2:DescribeNetworkInterfaces<br>            - ec2:DeleteNetworkInterface<br>            - xray:PutTraceSegments<br>          Resource: &quot;*&quot;<br><br>RootRole:<br>  Type: &#39;AWS::IAM::Role&#39;<br>  Properties:<br>    AssumeRolePolicyDocument:<br>      Version: &#39;2012-10-17&#39;<br>      Statement:<br>        - Effect: Allow<br>          Principal:<br>            Service:<br>              - lambda.amazonaws.com<br>          Action:<br>            - &#39;sts:AssumeRole&#39;<br>    Path: /<br>    ManagedPolicyArns:<br>      - !Ref SampleManagedPolicy</pre><p>The above policy will be created with minimum access.</p><p>Cache Client code:</p><pre>package main<br><br>import (<br>   &quot;context&quot;<br>   &quot;fmt&quot;<br>   &quot;github.com/aws/aws-lambda-go/lambda&quot;<br>   &quot;github.com/go-redis/redis&quot;<br>   &quot;os&quot;<br>)<br><br>func HandleRequest(ctx context.Context) (string, error) {<br><br>   redisUrl := os.Getenv(&quot;redis_url&quot;)<br>   redisPort := os.Getenv(&quot;redis_port&quot;)<br>   client := redis.NewClient(&amp;redis.Options{<br>      Addr:     fmt.Sprintf(&quot;%s:%s&quot;, redisUrl, redisPort),<br>      Password: &quot;&quot;, <em>// no password set<br>      </em>DB:       0,  <em>// use default DB<br>   </em>})<br><br>   client.Set(&quot;1&quot;, &quot;1&quot;, 0)<br><br>   return client.Get(&quot;1&quot;).String(), nil<br>}<br><br>func main() {<br>   lambda.Start(HandleRequest)<br>}</pre><p><strong>CREATING DATABASE INSTANCE WITH CODE</strong></p><pre>ServerlessDBSecurityGroup:<br>  DependsOn: VPC<br>  Type: AWS::EC2::SecurityGroup<br>  Properties:<br>    GroupDescription: Ingress for Redis Cluster<br>    VpcId:<br>      Ref: VPC<br>    SecurityGroupIngress:<br>      - IpProtocol: tcp<br>        FromPort: &#39;5432&#39;<br>        ToPort: &#39;5432&#39;<br>        SourceSecurityGroupId:<br>          Ref: ServerlessSecurityGroup<br><br>ServerlessDBSubnetGroup:<br>  DependsOn: ServerlessDBSecurityGroup<br>  Type: AWS::RDS::DBSubnetGroup<br>  Properties:<br>    DBSubnetGroupDescription: &quot;DB Subnet Group&quot;<br>    SubnetIds:<br>      - Ref: PrivateSubnet1<br>      - Ref: PrivateSubnet2<br><br>PostgresqlInstance:<br>  DependsOn: VPC<br>  Type: AWS::RDS::DBInstance<br>  Properties:<br>    AllocatedStorage: 30<br>    DBInstanceClass: db.t2.micro<br>    DBName: postgres<br>    Engine: postgres<br>    MasterUsername: CacheClient<br>    MasterUserPassword: ChangeIt2<br>    DBSubnetGroupName: !Ref ServerlessDBSubnetGroup<br>    VPCSecurityGroups:<br>      - &quot;Fn::GetAtt&quot;: ServerlessDBSecurityGroup.GroupId</pre><pre>DbClientFunction:<br>  Type: AWS::Serverless::Function<br>  Properties:<br>    Tracing: Active<br>    CodeUri: bin/dbClient<br>    Handler: dbClient<br>    Runtime: go1.x<br>    Role: !GetAtt RootRole.Arn<br>    VpcConfig:<br>      SecurityGroupIds:<br>        - Ref: ServerlessSecurityGroup<br>      SubnetIds:<br>        - Ref: PublicSubnet1<br>    Environment:<br>      Variables:<br>        db_url: !GetAtt PostgresqlInstance.Endpoint.Address<br>        db_port: !GetAtt PostgresqlInstance.Endpoint.Port</pre><p>Creating DB is very similar to the cache. The important point is your DB instance should stand on at least two availability zone.</p><p>DB client code:</p><pre>package main<br><br>import (<br>   &quot;context&quot;<br>   &quot;fmt&quot;<br>   &quot;github.com/aws/aws-lambda-go/lambda&quot;<br>   &quot;github.com/jinzhu/gorm&quot;<br>   _ &quot;github.com/jinzhu/gorm/dialects/postgres&quot;<br>   &quot;os&quot;<br>)<br><br>type MyEvent struct {<br>   Name string `json:&quot;name&quot;`<br>}<br><br>func HandleRequest(ctx context.Context, name MyEvent) (string, error) {<br><br>   dbUrl := os.Getenv(&quot;db_url&quot;)<br>   dbURI := fmt.Sprintf(&quot;host=%s user=CacheClient dbname=postgres sslmode=disable password=ChangeIt2&quot;, dbUrl)<br>   fmt.Println(dbURI)<br>   db, err := gorm.Open(&quot;postgres&quot;, dbURI)<br>   if err != nil {<br>      return &quot;err&quot;, err<br>   }<br><br>   db.AutoMigrate(&amp;Entity{})<br>   var ent = &amp;Entity{}<br>   ent.Text = name.Name<br>   db.Save(&amp;ent)<br><br>   return fmt.Sprint(&amp;ent.ID), nil<br>}<br><br>func main() {<br>   lambda.Start(HandleRequest)<br>}</pre><pre>package main<br><br>import &quot;github.com/jinzhu/gorm&quot;<br><br>type Entity struct {<br>   gorm.Model<br>   Text string<br>}</pre><p>All is done. Now we created network, cache instance, DB instance with code. We bind the output of them with our serverless functions. Cache and DB clients used created architecture by code.</p><p>The full version of template.yaml:</p><p><a href="https://github.com/yunuskilicdev/infrastructureascode">https://github.com/yunuskilicdev/infrastructureascode</a></p><p>Cites:</p><p>1-) <a href="https://stackify.com/what-is-infrastructure-as-code-how-it-works-best-practices-tutorials/">https://stackify.com/what-is-infrastructure-as-code-how-it-works-best-practices-tutorials/</a></p><p>2-) <a href="https://aws.amazon.com/cloudformation/">https://aws.amazon.com/cloudformation/</a></p><p>3-)<a href="https://cloudacademy.com/blog/aws-regions-and-availability-zones-the-simplest-explanation-you-will-ever-find-around/">https://cloudacademy.com/blog/aws-regions-and-availability-zones-the-simplest-explanation-you-will-ever-find-around/</a></p><p>4-)<a href="https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html">https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html</a></p><p>5-)<a href="https://www.infoq.com/articles/aws-vpc-explained/">https://www.infoq.com/articles/aws-vpc-explained/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fda322784f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/develop-infrastructure-as-code-with-cloudformation-and-sam-5fda322784f5">Develop Infrastructure as Code with CloudFormation and SAM</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>