<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Naveenkumar Karthigayan on Medium]]></title>
        <description><![CDATA[Stories by Naveenkumar Karthigayan on Medium]]></description>
        <link>https://medium.com/@naveenkumarkarthigayan?source=rss-9f62ce466dd3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jHO_y4esh0JCYYR0.jpg</url>
            <title>Stories by Naveenkumar Karthigayan on Medium</title>
            <link>https://medium.com/@naveenkumarkarthigayan?source=rss-9f62ce466dd3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:25:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@naveenkumarkarthigayan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Error Handling in Rust with Actix Web: Best Practices and Strategies]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/error-handling-in-rust-with-actix-web-best-practices-and-strategies-551602086fd3?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/551602086fd3</guid>
            <category><![CDATA[error-handling]]></category>
            <category><![CDATA[actix]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[actix-web]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Wed, 27 Mar 2024 18:03:24 GMT</pubDate>
            <atom:updated>2024-03-27T18:03:24.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*9P8hMlJ_1E-kAnncKohwRg.png" /></figure><p>Error handling is a critical aspect of writing robust and reliable software in Rust. Actix Web, a high-performance web framework for Rust, provides developers with powerful tools and idioms for managing errors effectively. In this guide, we’ll delve into advanced error handling techniques using custom error types and response formatting in Actix Web, ensuring your Rust applications handle errors gracefully.</p><h3>Dependency</h3><pre>[dependencies]<br>thiserror = &quot;1.0.58&quot;<br>serde_json = &quot;1.0.114&quot;<br>tokio =  { version = &quot;1.36.0&quot;, features = [&quot;full&quot;] }<br>bcrypt = &quot;0.15.1&quot;<br>actix = &quot;0.13.3&quot;<br>actix-web = &quot;4.5.1&quot;<br>actix-xml = &quot;0.2.0&quot;<br>quick-xml = { version = &quot;0.31.0&quot;, features = [&quot;serialize&quot;] }<br>serde = { version = &quot;1.0.197&quot;, features = [&quot;derive&quot;] }</pre><h3>Custom Error Types</h3><p>We begin by defining custom error types to encapsulate various error scenarios within our Rust applications. These custom error types enhance clarity and allow for better organization of error handling logic.</p><pre><br>use actix_web::dev::{Service};<br>use bcrypt::BcryptError;<br>use tokio::task::JoinError;<br>use actix_web::http::StatusCode;<br>use actix_web::{HttpResponse};<br>use quick_xml::se::to_string;<br>use serde::{Deserialize, Serialize};<br><br><br>#[derive(thiserror::Error, Debug)]<br>#[error(&quot;...&quot;)]<br>pub enum Error {<br><br>    #[error(&quot;Error parsing ObjectID {0}&quot;)]<br>    ParseObjectID(String),<br><br>    #[error(&quot;{0}&quot;)]<br>    Authenticate(#[from] AuthenticateError),<br><br>    #[error(&quot;{0}&quot;)]<br>    BadRequest(#[from] BadRequest),<br><br>    #[error(&quot;{0}&quot;)]<br>    NotFound(#[from] NotFound),<br><br>    #[error(&quot;{0}&quot;)]<br>    RunSyncTask(#[from] JoinError),<br><br>    #[error(&quot;{0}&quot;)]<br>    HashPassword(#[from] BcryptError),<br>}<br><br>impl Error {<br>    fn get_codes(&amp;self) -&gt; (StatusCode, u16) {<br>        match *self {<br>            // 4XX Errors<br>            Error::ParseObjectID(_) =&gt; (StatusCode::BAD_REQUEST, 40001),<br>            Error::BadRequest(_) =&gt; (StatusCode::BAD_REQUEST, 40002),<br>            Error::NotFound(_) =&gt; (StatusCode::NOT_FOUND, 40003),<br>            Error::Authenticate(AuthenticateError::WrongCredentials) =&gt; (StatusCode::UNAUTHORIZED, 40004),<br>            Error::Authenticate(AuthenticateError::InvalidToken) =&gt; (StatusCode::UNAUTHORIZED, 40005),<br>            Error::Authenticate(AuthenticateError::Locked) =&gt; (StatusCode::LOCKED, 40006),<br><br>            // 5XX Errors<br>            Error::Authenticate(AuthenticateError::TokenCreation) =&gt; {<br>                (StatusCode::INTERNAL_SERVER_ERROR, 5001)<br>            }<br><br>            Error::RunSyncTask(_) =&gt; (StatusCode::INTERNAL_SERVER_ERROR, 5005),<br>            Error::HashPassword(_) =&gt; (StatusCode::INTERNAL_SERVER_ERROR, 5006),<br>        }<br>    }<br><br><br>    pub fn bad_request(message : String) -&gt; Self {<br>        Error::BadRequest(BadRequest {message})<br>    }<br><br>    pub fn not_found() -&gt; Self {<br>        Error::NotFound(NotFound {})<br>    }<br>}<br><br><br></pre><pre>#[derive(thiserror::Error, Debug)]<br>#[error(&quot;...&quot;)]<br>pub enum AuthenticateError {<br>    #[error(&quot;Wrong authentication credentials&quot;)]<br>    WrongCredentials,<br>    #[error(&quot;Failed to create authentication token&quot;)]<br>    TokenCreation,<br>    #[error(&quot;Invalid authentication credentials&quot;)]<br>    InvalidToken,<br>    #[error(&quot;User is locked&quot;)]<br>    Locked,<br>}<br><br>#[derive(thiserror::Error, Debug)]<br>#[error(&quot;Bad Request {message}&quot;)]<br>pub struct BadRequest {<br>    message : String<br>}<br><br>#[derive(thiserror::Error, Debug)]<br>#[error(&quot;Not found&quot;)]<br>pub struct NotFound {}</pre><h3>Generating Error Responses</h3><p>Once we have defined our custom error types, we can implement methods to generate appropriate HTTP responses for different error scenarios.</p><pre>#[derive(Serialize,Deserialize)]<br>pub struct ErrorResponse {<br>    code : String,<br>    message : String<br>}<br><br>impl Error {<br>    pub fn error_response(&amp;self) -&gt; HttpResponse {<br>        let status_code = self.get_status_code();<br>        let message = self.to_string();<br>        let code = status_code.as_u16();<br>        let error_response = ErrorResponse { code, message };<br>        HttpResponse::build(status_code).json(error_response)<br>    }<br><br>    pub fn error_response_xml(&amp;self) -&gt; HttpResponse {<br>        let status_code = self.get_status_code();<br>        let message = self.to_string();<br>        let code = status_code.as_u16();<br>        let error_response = ErrorResponse { code, message };<br>        let error_resp_string = match to_string(&amp;error_response) {<br>            Ok(xml_string) =&gt; xml_string,<br>            Err(_) =&gt; &quot;&lt;error&gt;&quot;.to_string(), // Handle serialization error gracefully<br>        };<br>        HttpResponse::build(status_code).content_type(&quot;application/xml&quot;).body(error_resp_string)<br>    }<br>}</pre><h3>Usage Example</h3><p>Let’s see how we can use these error handling utilities in our Actix Web handlers:</p><pre>use std::io::Read;<br>use actix_web::{App, HttpRequest, HttpServer, Responder, ResponseError, web};<br>use actix_web::body::MessageBody;<br>use crate::errors::{AuthenticateError, Error};<br><br>mod errors;<br><br><br>pub async fn send_authenticate_error(_req: HttpRequest) -&gt; impl Responder {<br>    // This API will send auth error as response in JSON format<br>    Error::Authenticate(AuthenticateError::WrongCredentials).error_response()<br>}<br><br><br>pub async fn send_bad_request(_req: HttpRequest) -&gt; impl Responder {<br>    // This API will send  bad request error as response inXMl format<br>    Error::bad_request(&quot;Value is required&quot;.to_string()).error_response_xml()<br>}<br><br>#[actix_web::main]<br>async fn main() -&gt; std::io::Result&lt;()&gt; {<br>    HttpServer::new(|| {<br>        App::new()<br>            .route(&quot;/auth_error&quot;, web::get().to(send_authenticate_error))<br>            .route(&quot;/bad_request&quot;, web::get().to(send_bad_request))<br>    }).bind(&quot;0.0.0.0:8000&quot;)?<br>        .run()<br>        .await<br>}</pre><h3>Conclusion</h3><p>Custom error types and response formatting are powerful tools for handling errors effectively in Rust applications with Actix Web. By defining clear error semantics and providing informative error responses, we can enhance the reliability and usability of our web services.</p><h3>Contact Details</h3><p>For inquiries or collaborations, feel free to contact me:<br>- Email: <a href="/service/mailto:youremail@example.com">naveenkumarkarthigayan@gmail.com</a><br>- LinkedIn: <a href="https://www.linkedin.com/in/naveenkumar001/">https://www.linkedin.com/in/naveenkumar001/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=551602086fd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rust concurrency with me — Naveenkumar]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/rust-concurrency-with-me-naveenkumar-509e5a1db134?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/509e5a1db134</guid>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[multitasking]]></category>
            <category><![CDATA[threads]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Sun, 17 Mar 2024 10:09:10 GMT</pubDate>
            <atom:updated>2024-03-17T10:09:10.615Z</atom:updated>
            <content:encoded><![CDATA[<h3>Rust concurrency with me — Naveenkumar</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*I756-YDN-uu1mkRBV4aJNQ.png" /></figure><h3>Concurrency in Rust is based on ownership, types, and borrowing principles, which help manage memory safety without a garbage collector.</h3><h4>Rust achieves thread safety by ensuring that shared data is either immutable or only accessible from one thread at a time.</h4><h3>Key Concepts:</h3><ul><li><strong>Threads: </strong>Rust uses threads to run multiple parts of your code simultaneously. The std::thread module allows you to create new threads.</li><li><strong>Message Passing: </strong>Rust’s concurrency model follows the “Do not communicate by sharing memory; instead, share memory by communicating” philosophy. Channels in Rust, provided by the std::sync::mpsc module, facilitate this.</li><li><strong>Shared State Concurrency: </strong>Rust allows for shared-state concurrency using Mutex and Arc from the std::sync module to safely share and mutate data across threads.</li><li><strong>Sync and Send Traits:</strong> These marker traits ensure that only safe types are shared between threads. Send allows a type to be transferred across thread boundaries, while Sync allows a type to be safely shared through references.</li></ul><p>Let’s now put it into practice with exercises to help you grasp or refresh these key concepts! 🦀</p><h3>Exercises</h3><h3>Exercise 1: Spawning Threads</h3><p>Create a simple program that spawns multiple threads and prints “Hello from thread!” from each thread.</p><pre>use std::thread;<br>use std::time::Duration;<br><br>fn main() {<br>    for i in 0..5 {<br>        thread::spawn(move || {<br>            println!(&quot;Hello from thread {}&quot;, i);<br>            thread::sleep(Duration::from_millis(100));<br>        });<br>    }<br>    thread::sleep(Duration::from_secs(1)); // Wait for all threads to complete<br>}</pre><h3>Exercise 2: Message Passing</h3><p>Implement a program where you spawn two threads: one for sending a message using a channel, and another for receiving and printing that message.</p><pre>use std::sync::mpsc;<br>use std::thread;<br><br>fn main() {<br>    let (tx, rx) = mpsc::channel();<br>    thread::spawn(move || {<br>        let msg = String::from(&quot;Hello from the sender!&quot;);<br>        tx.send(msg).unwrap();<br>    });<br>    let received = rx.recv().unwrap();<br>    println!(&quot;Received: {}&quot;, received);<br>}</pre><h3>Exercise 3: Shared State with Mutex</h3><p>Create a program that uses a Mutex to safely increment a counter from multiple threads.</p><pre>use std::sync::{Arc, Mutex};<br>use std::thread;<br><br>fn main() {<br>    let counter = Arc::new(Mutex::new(0));<br>    let mut handles = vec![];<br>    for _ in 0..10 {<br>        let counter = Arc::clone(&amp;counter);<br>        let handle = thread::spawn(move || {<br>            let mut num = counter.lock().unwrap();<br>            *num += 1;<br>        });<br>        handles.push(handle);<br>    }<br>    for handle in handles {<br>        handle.join().unwrap();<br>    }<br>    println!(&quot;Result: {}&quot;, *counter.lock().unwrap());<br>}</pre><h3>Exercise 4: Building a Concurrent Web Server</h3><p>This is a more advanced exercise. Implement a basic concurrent web server using TcpListener and threads. Accept connections, read the request, and respond with a fixed message.</p><pre>use std::io::prelude::*;<br>use std::net::{TcpListener, TcpStream};<br>use std::thread;<br>use std::time::Duration;<br><br>fn handle_connection(mut stream: TcpStream) {<br>    let mut buffer = [0; 1024];<br>    stream.read(&amp;mut buffer).unwrap();<br>    let response = &quot;HTTP/1.1 200 OK\r\n\r\nHello from Rust server!&quot;;<br>    stream.write(response.as_bytes()).unwrap();<br>    stream.flush().unwrap();<br>}<br>fn main() {<br>    let listener = TcpListener::bind(&quot;127.0.0.1:7878&quot;).unwrap();<br>    for stream in listener.incoming() {<br>        let stream = stream.unwrap();<br>        thread::spawn(|| {<br>            handle_connection(stream);<br>        });<br>    }<br>}</pre><h3>Advanced Exercise: Implementing a Thread Pool</h3><p>An effective way to manage resources in a concurrent program is by using a thread pool. A thread pool creates a fixed number of threads at program start-up and reuses these threads to execute tasks. This exercise involves implementing a simple thread pool to handle tasks in a concurrent web server.</p><h4>Step 1: Define the ThreadPool Structure</h4><p>Start by defining the structure of your thread pool and the initial function to create a new thread pool with a specified number of threads.</p><pre>struct ThreadPool {<br>    workers: Vec&lt;Worker&gt;,<br>}<br><br>impl ThreadPool {<br>    fn new(size: usize) -&gt; ThreadPool {<br>        assert!(size &gt; 0);<br>        let mut workers = Vec::with_capacity(size);<br>        for _ in 0..size {<br>            // Create and store workers<br>        }<br>        ThreadPool { workers }<br>    }<br>}</pre><h4>Step 2: Implement the Worker</h4><p>Each worker will be responsible for executing tasks. A worker holds a thread and listens for tasks sent through a channel.</p><pre>struct Worker {<br>    id: usize,<br>    thread: Option&lt;thread::JoinHandle&lt;()&gt;&gt;,<br>}<br><br>impl Worker {<br>    fn new(id: usize) -&gt; Worker {<br>        let thread = thread::spawn(|| {<br>            // Placeholder for the task execution logic<br>        });<br>        Worker {<br>            id,<br>            thread: Some(thread),<br>        }<br>    }<br>}</pre><h4>Step 3: Sending Tasks to Threads</h4><p>Modify the ThreadPool to hold a sending end of a channel. Workers will listen on the receiving end for tasks to execute. Tasks can be represented as closures.</p><pre>use std::sync::mpsc;<br>use std::sync::Arc;<br>use std::sync::Mutex;<br><br>type Job = Box&lt;dyn FnOnce() + Send + &#39;static&gt;;<br>struct ThreadPool {<br>    workers: Vec&lt;Worker&gt;,<br>    sender: mpsc::Sender&lt;Job&gt;,<br>}<br>// Modify the ThreadPool::new function to include a channel</pre><h4>Step 4: Execute Tasks in the ThreadPool</h4><p>Implement a method on ThreadPool to send tasks to the worker threads for execution.</p><pre>impl ThreadPool {<br>    // ...<br><br>    fn execute&lt;F&gt;(&amp;self, f: F)<br>    where<br>      F: FnOnce() + Send + &#39;static,<br>      {<br>          let job = Box::new(f);<br>          self.sender.send(job).unwrap();<br>      }<br>}</pre><h4>Step 5: Graceful Shutdown</h4><p>Implement logic for a graceful shutdown of the thread pool. Ensure that all workers finish their current tasks before the program exits.</p><pre>impl Drop for ThreadPool {<br>    fn drop(&amp;mut self) {<br>        for worker in &amp;mut self.workers {<br>            println!(&quot;Shutting down worker {}&quot;, worker.id);<br><br>            if let Some(thread) = worker.thread.take() {<br>                thread.join().unwrap();<br>            }<br>        }<br>    }<br>}</pre><h3>Integrating the ThreadPool into the Web Server</h3><p>Replace the simple thread spawn logic in your web server with the ThreadPool. This will limit the number of concurrent threads and reuse them for incoming requests, which is more efficient than spawning a new thread per request.</p><pre>fn main() {<br>    let listener = TcpListener::bind(&quot;127.0.0.1:7878&quot;).unwrap();<br>    let pool = ThreadPool::new(4); // Adjust the number of threads in the pool as needed<br><br>    for stream in listener.incoming() {<br>        let stream = stream.unwrap();<br>        pool.execute(|| {<br>            handle_connection(stream);<br>        });<br>    }<br>    println!(&quot;Shutting down.&quot;);<br>}</pre><h3>Going Further: Async/Await in Rust</h3><p>As you become more comfortable with threads, message passing, and shared state concurrency, it’s important to explore Rust’s asynchronous programming capabilities. The async/await syntax in Rust provides a powerful model for writing concurrent code that is both efficient and easy to read.</p><h4>Exercise: Asynchronous File Reading</h4><p>In this exercise, you’ll write a program that reads the contents of a file asynchronously. This will demonstrate how to perform I/O-bound tasks without blocking the execution of your program.</p><p>First, include the necessary crates in your Cargo.toml:</p><pre>[dependencies]<br>tokio = { version = &quot;1&quot;, features = [&quot;full&quot;] }<br>futures = &quot;0.3&quot;</pre><p>Next, write the async function to read a file:</p><pre>use tokio::fs::File;<br>use tokio::io::AsyncReadExt;<br><br>async fn read_file_async(path: &amp;str) -&gt; Result&lt;String, std::io::Error&gt; {<br>    let mut file = File::open(path).await?;<br>    let mut contents = String::new();<br>    file.read_to_string(&amp;mut contents).await?;<br>    Ok(contents)<br>}</pre><p>To execute the async function, use the Tokio runtime:</p><pre>#[tokio::main]<br>async fn main() {<br>    let path = &quot;data.txt&quot;;<br>    match read_file_async(path).await {<br>        Ok(contents) =&gt; println!(&quot;File contents: {}&quot;, contents),<br>        Err(e) =&gt; println!(&quot;Error reading the file: {:?}&quot;, e),<br>    }<br>}<br></pre><h4>Exercise: Concurrent Web Requests</h4><p>In this exercise, you’ll make multiple web requests concurrently and wait for all of them to complete. This is a common scenario in web servers and clients that need to aggregate data from multiple sources.</p><p>First, add the reqwest and tokio crates to your Cargo.toml:</p><pre>[dependencies]<br>reqwest = { version = &quot;0.11&quot;, features = [&quot;json&quot;] }<br>tokio = { version = &quot;1&quot;, features = [&quot;full&quot;] }</pre><p>Then, write a function to perform a web request asynchronously:</p><pre>use reqwest::Error;<br><br>async fn fetch_url(/service/url: &amp;str) -&gt; Result&lt;String, Error&gt; {<br>    let response = reqwest::get(url).await?;<br>    let body = response.text().await?;<br>    Ok(body)<br>}</pre><p>Execute multiple requests concurrently and wait for all to complete:</p><pre>#[tokio::main]<br>async fn main() {<br>    let urls = [&quot;https://example.com&quot;, &quot;https://api.example.com&quot;, &quot;https://blog.example.com&quot;];<br><br>    let fetches = futures::future::join_all(urls.iter().map(|&amp;url| fetch_https://medium.com/url(url)));<br>    match fetches.await {<br>        Ok(responses) =&gt; {<br>            for response in responses {<br>                match response {<br>                    Ok(content) =&gt; println!(&quot;Fetched content: {}&quot;, content),<br>                    Err(e) =&gt; println!(&quot;Failed to fetch: {:?}&quot;, e),<br>                }<br>            }<br>        },<br>        Err(_) =&gt; println!(&quot;Error in executing requests&quot;),<br>    }<br>}</pre><p>Thank you for taking the time to read the blog!</p><h3>Contact Details</h3><p>For inquiries or collaborations, feel free to contact me:<br>- Email: <a href="/service/mailto:youremail@example.com">naveenkumarkarthigayan@gmail.com</a><br>- LinkedIn: <a href="https://www.linkedin.com/in/naveenkumar001/">https://www.linkedin.com/in/naveenkumar001/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=509e5a1db134" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficient File Sharing with Kafka and Rust: A Comprehensive Guide — Naveenkumar]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/efficient-file-sharing-with-kafka-and-rust-a-comprehensive-guide-naveenkumar-ef829a8554a1?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef829a8554a1</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[file-sharing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[filesystem]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Thu, 14 Mar 2024 17:45:29 GMT</pubDate>
            <atom:updated>2024-03-17T06:14:03.178Z</atom:updated>
            <content:encoded><![CDATA[<h3>Efficient File Sharing with Kafka and Rust: A Comprehensive Guide — Naveenkumar</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*5R-rNyhqFt4f_IZOvcc-cQ.png" /></figure><p>In today’s interconnected world, efficient file sharing is a fundamental requirement for many applications, especially those operating in distributed environments. Leveraging Apache Kafka, a distributed streaming platform, along with the Rust programming language, known for its performance and safety guarantees, offers a robust solution for implementing file sharing capabilities. In this guide, we’ll explore how to harness the power of Kafka and Rust to build an efficient file sharing system.</p><h3>Implementing File Sharing with Kafka and Rust</h3><h3>Setting Up Kafka with Docker Compose:</h3><p>To simplify the setup process, we can use Docker Compose to run Kafka and Zookeeper locally. Below is a Docker Compose configuration for Kafka:</p><pre>version: &#39;2&#39;<br>services:<br>  zookeeper:<br>    image: confluentinc/cp-zookeeper:latest<br>    container_name: zookeeper<br>    environment:<br>      ZOOKEEPER_CLIENT_PORT: 2181<br>      ZOOKEEPER_TICK_TIME: 2000<br><br>  kafka:<br>    image: confluentinc/cp-kafka:latest<br>    container_name: kafka<br>    depends_on:<br>      - zookeeper<br>    ports:<br>      - 9092:9092<br>      - 9093:9093<br>    environment:<br>      KAFKA_BROKER_ID: 1<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_HOST://kafka:9093<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT<br>      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1</pre><h3>Understanding rdkafka in Rust</h3><p>Before diving into implementation details, let’s briefly discuss rdkafka, the Rust binding for Kafka’s C API. <strong>rdkafka</strong> provides a high-level, idiomatic Rust interface for interacting with Kafka clusters, making it an excellent choice for Rust developers looking to integrate Kafka into their applications.</p><p>To get started, ensure you have Kafka installed and running locally or on a remote server. Next, add rdkafka as a dependency in your Rust project’s Cargo.toml file:</p><pre>[dependencies]<br>lazy_static = &quot;1.4.0&quot;<br>rdkafka = &quot;0.36.2&quot;<br>tokio =  { version = &quot;1.36.0&quot;, features = [&quot;full&quot;] }</pre><h3>Producer Code (Sending Files to Kafka):</h3><pre><br>use rdkafka::message::{Header, OwnedHeaders};<br>use rdkafka::producer::{FutureProducer, FutureRecord};<br>use rdkafka::util::Timeout::Never;<br>use tokio::fs::File;<br>use tokio::io::{AsyncReadExt, BufReader};<br>use tokio::time::Instant;<br><br><br>async fn produce_file_to_kafka(file_path: &amp;str, topic: &amp;str, brokers: &amp;str) -&gt; Result&lt;(), Box&lt;dyn std::error::Error&gt;&gt; {<br><br>    // Configure the Kafka producer<br>    let producer: FutureProducer = rdkafka::config::ClientConfig::new()<br>        .set(&quot;bootstrap.servers&quot;, brokers)<br>        .set(&quot;produce.offset.report&quot;, &quot;true&quot;)<br>        .set(&quot;message.max.bytes&quot;,(1 * 1024 * 1024).to_string())<br>        .create()?;<br><br><br>    // Read the   file in chunks and produce each chunk as a Kafka message with headers<br>    let mut file = File::open(file_path).await?;<br>    let mut reader = BufReader::new(&amp;mut file);<br>    let mut buffer = vec![0u8; 1000  * 1024]; // 1000KB buffer size<br><br>    let mut total_bytes_read = 0;<br>    let mut total_chunks_sent = 0;<br><br>    let filename_with_ext = match file_path.rsplit_once(&#39;/&#39;) {<br>        Some((_, name)) =&gt; name,<br>        None =&gt; file_path,<br>    };<br><br>    let now = Instant::now(); /* Noted the starting Time for time taken*/<br><br>    // Started Reading a File<br><br>    loop {<br>        // Read from the file<br>        let n = reader.read(&amp;mut buffer).await.unwrap_or(0);<br>        if n == 0 {<br>            break;<br>        }<br><br>        total_bytes_read += n;<br>        total_chunks_sent += 1;<br><br><br>        // Produce the file chunk as a Kafka message with headers<br>        let record = FutureRecord::to(topic)<br>            .payload(&amp;buffer[0..n]) // Send only the actual data read<br>            .key(filename_with_ext);<br><br>        match producer.send(record, Never).await {<br>            Ok(_) =&gt; println!(&quot;File chunk {} sent successfully.&quot;, total_chunks_sent),<br>            Err((kafka_error, _)) =&gt;{<br>                eprintln!(&quot;Error sending remaining data: {:?}&quot;, kafka_error);<br>                return Err(Box::new(kafka_error))<br>            },<br>        }<br><br>        // Reset buffer for the next iteration<br>        buffer = vec![0u8; 1000 * 1024];<br>    }<br>    // Process remaining data (if any)<br>    if total_bytes_read % (1000 * 1024) != 0 {<br>        let remaining_data = &amp;buffer[0..(total_bytes_read % (1000 * 1024))];<br><br><br>        // Produce the remaining data as a Kafka message with headers<br>        let record = FutureRecord::to(topic)<br>            .payload(remaining_data)<br>            .key(filename_with_ext)<br>            .headers(OwnedHeaders::new()<br>                .insert(Header{key : &quot;file_name&quot;,value : Some(filename_with_ext)}));<br><br>        match producer.send(record, Never).await {<br>            Ok(_) =&gt; println!(&quot;Remaining data sent successfully.&quot;),<br>            Err((kafka_error, _)) =&gt;{<br>                eprintln!(&quot;Error sending remaining data: {:?}&quot;, kafka_error);<br>                return Err(Box::new(kafka_error))<br>            },<br>        }<br>    }<br><br>    let elapsed_time = now.elapsed();<br>    println!(&quot;Running  took {} seconds.&quot;, elapsed_time.as_secs());<br>    println!(&quot;Total bytes read: {} bytes&quot;, total_bytes_read);<br><br>    // Close the file and producer<br>    drop(file);<br>    drop(producer);<br><br>    Ok(())<br>}<br><br>#[tokio::main]<br>async fn main() -&gt; Result&lt;(), Box&lt;dyn std::error::Error&gt;&gt; {<br>    let file_path = &quot;static/RustRover-233.10527.39.tar.gz&quot;; /* Your File Path Here */<br>    let topic = &quot;file-sharing-topic&quot;;<br>    let brokers = &quot;localhost:9092&quot;;<br><br>    produce_file_to_kafka(file_path, topic, brokers).await?;<br>    <br>    println!(&quot;File successfully sent to Kafka!&quot;);<br><br>    Ok(())<br>}</pre><h3>Explanation:</h3><ol><li><strong>Chunking the File</strong>: We read the file in chunks of a fixed size (1000 * 1024 bytes in this example) using a buffer. This helps in handling large files efficiently and avoids memory issues by processing the file in manageable pieces.</li><li><strong>Creating Kafka Records</strong>: For each chunk read from the file, we create a Kafka record using FutureRecord::to(topic). We set the payload of the record to the chunk of data read from the file.</li><li><strong>Sending Kafka Records</strong>: We send each Kafka record asynchronously using producer.send(record, Instant::now()). This sends the record to Kafka for processing. We also handle any errors that may occur during the sending process.</li><li><strong>Handling Remaining Data</strong>: After reading all chunks from the file, we check if there is any remaining data left unread. If so, we create a Kafka record with the remaining data and send it to Kafka similar to the chunked data.</li><li><strong>Cleanup and Statistics</strong>: Finally, we calculate and print the running time of the process using Instant::now() and elapsed(). We also ensure proper cleanup by dropping the file and producer objects.</li></ol><p>This approach ensures efficient and reliable transfer of file data to Kafka, handling both chunked data and any remaining data that may not fit perfectly into chunks. It’s a robust solution for file sharing applications using Kafka and Rust.</p><h3>Consumer Code (Receiving Files from Kafka):</h3><pre><br>use rdkafka::config::ClientConfig;<br>use rdkafka::consumer::{Consumer, StreamConsumer};<br>use rdkafka::message::Message;<br>use tokio::fs::OpenOptions;<br>use tokio::io::AsyncWriteExt;<br><br>async fn consume_files_from_kafka(<br>    topic: &amp;str,<br>    brokers: &amp;str,<br>) -&gt; Result&lt;(), Box&lt;dyn std::error::Error&gt;&gt; {<br>    // Kafka consumer configuration<br>    let consumer: StreamConsumer = ClientConfig::new()<br>        .set(&quot;group.id&quot;, &quot;file-consumer-group&quot;)<br>        .set(&quot;bootstrap.servers&quot;, brokers)<br>        .set(&quot;auto.offset.reset&quot;, &quot;earliest&quot;)<br>        .create()?;<br><br>    // Subscribe to the Kafka topic<br>    consumer.subscribe(&amp;[topic])?;<br><br>    // Directory to store files<br>    let output_directory = &quot;output/&quot;;<br><br>    // Start consuming messages<br>    loop {<br>        match consumer.recv().await {<br>            Ok(msg) =&gt; {<br>                let key = msg.key().map(|k| String::from_utf8_lossy(k));<br>                let payload = msg.payload().unwrap();<br><br>                //Assuming the key is the filename<br>                if let Some(filename) = key {<br>                    // Create file path<br>                    let file_path = format!(&quot;{}{}&quot;, output_directory, filename);<br><br>                    // Write payload to file<br>                    let mut file = OpenOptions::new()<br>                        .create(true)<br>                        .append(true)<br>                        .open(&amp;file_path)<br>                        .await?;<br><br>                    file.write_all(payload).await?;<br>                    println!(&quot;File saved: {}&quot;, file_path);<br>                } else {<br>                    println!(&quot;Message without a filename key&quot;);<br>                    return Err(Box::new(std::io::Error::new(std::io::ErrorKind::Other, &quot;Message without a filename key&quot;)));<br>                }<br>            }<br>            Err(e) =&gt; {<br>                eprintln!(&quot;Error consuming message: {:?}&quot;, e);<br>                return Err(Box::new(e))<br>            }<br>        }<br>    }<br>}<br><br>#[tokio::main]<br>async fn main() -&gt; Result&lt;(), Box&lt;dyn std::error::Error&gt;&gt; {<br>    let topic = &quot;file-sharing-topic&quot;;<br>    let brokers = &quot;localhost:9092&quot;;<br><br>    consume_files_from_kafka(topic, brokers).await?;<br><br>    Ok(())<br>}<br></pre><h3>Explanation:</h3><ol><li><strong>Consumer Configuration</strong>: We configure the Kafka consumer with necessary parameters such as group ID, bootstrap servers, and auto offset reset. This ensures that the consumer can connect to Kafka and start consuming messages from the specified topic.</li><li><strong>Subscribing to Topic</strong>: We subscribe the consumer to the Kafka topic &quot;file-sharing-topic&quot; using the subscribe() method. This instructs the consumer to start receiving messages from this topic.</li><li><strong>File Output Directory</strong>: We specify the directory where received files will be stored using the output_directory variable. This directory should be created beforehand if it doesn&#39;t exist.</li><li><strong>Consuming Messages</strong>: We enter a loop to continuously consume messages from Kafka. For each received message, we extract the key (filename) and payload (file content).</li><li><strong>Writing to File</strong>: We create a file path using the output directory and filename. Then, we open the file in append mode and write the payload (file content) to the file using write_all() method. This saves the received file to the local file system.</li><li><strong>Error Handling</strong>: We handle any errors that may occur during message consumption, file writing, or Kafka communication. Errors are printed to the console for debugging purposes.</li></ol><p>This Consumer code efficiently receives files from Kafka, saves them to the local file system, and handles errors gracefully. It’s a reliable solution for building file sharing applications using Kafka and Rust.</p><p>In conclusion, leveraging Apache Kafka and Rust provides a robust solution for implementing efficient file sharing systems in distributed environments. By utilizing Kafka’s distributed messaging platform and Rust’s performance and safety features, developers can build scalable, reliable, and high-performance file sharing applications.</p><h3>Benefits:</h3><ol><li><strong>Scalability</strong>: Kafka’s distributed architecture allows seamless scaling of file sharing services to handle increasing data volumes and user loads.</li><li><strong>Reliability</strong>: Kafka ensures fault-tolerant and durable storage of files, preventing data loss even in the event of failures.</li><li><strong>Real-time Processing</strong>: Kafka’s support for real-time stream processing enables immediate consumption of shared files, facilitating rapid data transfer.</li><li><strong>Performance</strong>: Rust’s emphasis on zero-cost abstractions and low-level control results in highly performant applications, crucial for handling large file transfers efficiently.</li><li><strong>Safety</strong>: Rust’s ownership model and strict compile-time checks help prevent common programming errors such as memory leaks and data races, enhancing the reliability of file sharing services.</li></ol><h3>Challenges:</h3><ol><li><strong>Complexity</strong>: Setting up and configuring Kafka clusters may require expertise and introduce complexity, especially for beginners.</li><li><strong>Learning Curve</strong>: Rust’s syntax and ownership rules may pose a learning curve for developers unfamiliar with the language, potentially slowing down initial development efforts.</li><li><strong>Resource Intensive</strong>: Running Kafka clusters can be resource-intensive, requiring adequate hardware resources and infrastructure to ensure optimal performance.</li></ol><p>Despite these challenges, the benefits of using Kafka and Rust for file sharing outweigh the drawbacks. With careful planning and implementation, developers can build robust and efficient file sharing systems capable of handling large volumes of data with ease.</p><p>Overall, Kafka and Rust provide a powerful combination for building file sharing applications, enabling developers to deliver reliable and scalable solutions in distributed environments. Start building your file sharing system with Kafka and Rust today, and unlock the full potential of distributed file sharing!</p><h3>Contact Details</h3><p>For inquiries or collaborations, feel free to contact me:<br>- Email: <a href="/service/mailto:youremail@example.com">naveenkumarkarthigayan@gmail.com</a><br>- LinkedIn: <a href="https://www.linkedin.com/in/naveenkumar001/">https://www.linkedin.com/in/naveenkumar001/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef829a8554a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Preventing Out of Memory Errors in Go Concurrency”]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/preventing-out-of-memory-errors-in-go-concurrency-ff8809c3b19?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff8809c3b19</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[goroutines]]></category>
            <category><![CDATA[golang-tutorial]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Fri, 20 Jan 2023 18:08:04 GMT</pubDate>
            <atom:updated>2023-01-20T18:08:04.398Z</atom:updated>
            <content:encoded><![CDATA[<p>“Best practices and code examples for avoiding OOM errors in multithreaded Go programs”</p><p>Out-of-memory (OOM) errors can be a major issue when working with concurrency in Go. These errors occur when a program attempts to allocate more memory than is available on the system, causing the program to crash. In this blog post, we will discuss some best practices for avoiding OOM errors in Go concurrency.</p><ol><li>Use buffered channels: Buffered channels allow goroutines to send and receive data without blocking, which can help to prevent OOM errors caused by goroutines waiting for memory to become available. When using buffered channels, be sure to set the buffer size appropriately to avoid overloading the channel and causing OOM errors.</li><li>Limit the number of goroutines: Starting too many goroutines can lead to OOM errors, as each goroutine requires memory to store its stack and other data structures. Consider using a goroutine pool to limit the number of goroutines that can be running at any given time.</li><li>Use context.WithCancel: Using context.WithCancel, you can stop the execution of a goroutine that is no longer needed, which can help to prevent OOM errors caused by goroutines that continue to run and consume memory.</li><li>Use the garbage collector: Go’s garbage collector can help to reclaim memory that is no longer being used by the program. By default, the garbage collector runs automatically, but you can also manually trigger a garbage collection by calling the runtime.GC() function.</li><li>Monitor memory usage: Monitoring memory usage can help you to identify potential OOM errors before they occur. You can use tools like pprof and memstats to monitor memory usage and detect any patterns that may indicate an OOM error is imminent.</li><li>Use memory-efficient data structures: Some data structures consume more memory than others, so it’s important to choose data structures that are memory-efficient when working with concurrency in Go.</li></ol><p>By following these best practices, you can help to avoid OOM errors in Go concurrency and ensure that your program runs smoothly and efficiently.</p><p>Here is an example of how to avoid out-of-memory errors in Go by using buffered channels and a goroutine pool:</p><pre>package main<br><br>import (<br>    &quot;fmt&quot;<br>    &quot;sync&quot;<br>)<br><br>const (<br>    maxGoroutines = 10  // maximum number of goroutines<br>    bufferSize    = 100 // buffer size for channel<br>)<br><br>func worker(id int, jobs &lt;-chan int, results chan&lt;- int, wg *sync.WaitGroup) {<br>    defer wg.Done()<br>    for j := range jobs {<br>        fmt.Println(&quot;worker&quot;, id, &quot;processing job&quot;, j)<br>        results &lt;- j * 2<br>    }<br>}<br><br>func main() {<br>    jobs := make(chan int, bufferSize)<br>    results := make(chan int, bufferSize)<br><br>    var wg sync.WaitGroup<br>    for w := 1; w &lt;= maxGoroutines; w++ {<br>        wg.Add(1)<br>        go worker(w, jobs, results, &amp;wg)<br>    }<br><br>    for j := 1; j &lt;= 100; j++ {<br>        jobs &lt;- j<br>    }<br>    close(jobs)<br><br>    wg.Wait()<br>    close(results)<br><br>    for r := range results {<br>        fmt.Println(&quot;result:&quot;, r)<br>    }<br>}</pre><p>In this example, we use a buffered channel to send jobs to a pool of goroutines. The number of goroutines is limited by the maxGoroutines constant, and the buffer size of the channel is set to bufferSize. This helps to prevent OOM errors caused by goroutines waiting for memory to become available. Additionally, we are using WaitGroup to wait for the goroutines to complete their work.</p><p>Also, we are closing the channels once the work is done, and the results channel once all the goroutines have completed their work, this will help the garbage collector to reclaim the memory.</p><p>Please note that this is just one example, and you can use different approaches to avoid OOM errors based on your use case.</p><p>If you still don’t understand the concept I give you an even more simple example than the above one,</p><pre>package main<br><br>import &quot;fmt&quot;<br>   <br><br><br>const maxGoroutinesLimitSize = 10  // maximum number of goroutines run cuncurrently <br><br><br>func main() {<br><br>    limiter := make(chan int, maxGoroutinesLimitSize)<br>   <br>    // Let consider this event manger stream minumum 300 to 500 events at a time<br>    for event:= range eventManget.Stream(){<br>        limiter &lt;- 1<br>        go func(e events.Event){<br>           go eventProceser.ProcessEvent(e)  // let consider there is function like this to process events<br>           &lt;- limiter<br>        }(event)<br>    }<br><br><br>}</pre><p>In this example, we use a buffered channel to limit the number of goroutines that can run concurrently. As we all know go channel has “blocking behavior”.Every time a goroutine is added before we add value to the channel, here inside the goroutine function we pull out the value from the channel which creates a space in the channel for the next goroutine, Goroutine needs to wait until the channel has space.</p><p>We may face some lag in catching up on the events but we don’t face Out Of Memory Problems anymore.</p><p>Gradually increase themaxGoroutinesLimitSize until you get a perfect fit for you</p><p>In conclusion, out-of-memory errors can be a major issue when working with concurrency in Go. By following best practices such as using buffered channels, limiting the number of goroutines, using context with cancel function, monitoring memory usage, using memory-efficient data structures, and using garbage collector, you can help to avoid OOM errors and ensure that your program runs smoothly and efficiently.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff8809c3b19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Thread-Safe Maps in Go: Implementing a Concurrent Map Data Structure”]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/thread-safe-maps-in-go-implementing-a-concurrent-map-data-structure-72ab161ac882?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/72ab161ac882</guid>
            <category><![CDATA[golang-development]]></category>
            <category><![CDATA[go-channels]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[goroutines]]></category>
            <category><![CDATA[concurrency]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Fri, 20 Jan 2023 17:05:03 GMT</pubDate>
            <atom:updated>2023-01-20T17:07:14.282Z</atom:updated>
            <content:encoded><![CDATA[<p>In Go, a map is not inherently thread-safe, which means that if multiple goroutines access and modify a map simultaneously, there is a risk of data corruption or race conditions. This is known as an “unsafe” map.</p><p>When using an unsafe map, it is the responsibility of the developer to ensure that proper synchronization is used to protect the map from race conditions. This can be done by using a sync.Mutex or a sync.RWMutex to synchronize access to the map.</p><p>However, it is important to note that using an unsafe map can lead to performance issues, as the use of locks can slow down the program and introduce contention among goroutines. Additionally, if synchronization is not implemented correctly, it can lead to bugs and data corruption.</p><p>Therefore, it is generally recommended to use a concurrent safe map in Go, as it eliminates the need to manually synchronize access to the map, and it can improve the performance and reliability of your program.</p><h3>Unsafe</h3><p>I created a simple goroutine to update a map, See what happens when I write to run with a race flag</p><pre>package main<br><br>func main() {<br> unsafe_map := make(map[int]int)<br><br> for i := 0; i &lt; 10; i++ {<br>  go func(i int) {<br>   unsafe_map[i] = i<br>  }(i)<br> }<br>}<br> </pre><pre>WARNING: DATA RACE<br>Write at 0x00c00007c390 by goroutine 9:</pre><h3>Safe map data structure</h3><p>Creating a concurrent safe map in Go is relatively easy, as the language provides built-in support for concurrency and synchronization.</p><p>One way to create a concurrent safe map in Go is to use the sync.RWMutex type. This type provides a read-write lock, which allows multiple goroutines to read from the map simultaneously, but only one goroutine to write to the map at a time.</p><p>Here is an example of how to create a concurrent safe map using sync.RWMutex:</p><pre>package main<br><br>import (<br>    &quot;sync&quot;<br>)<br><br>type ConcurrentMap struct {<br>    sync.RWMutex<br>    data map[string]interface{}<br>}<br><br>func (c *ConcurrentMap) Set(key string, value interface{}) {<br>    c.Lock()<br>    defer c.Unlock()<br>    c.data[key] = value<br>}<br><br>func (c *ConcurrentMap) Get(key string) (interface{}, bool) {<br>    c.RLock()<br>    defer c.RUnlock()<br>    value, ok := c.data[key]<br>    return value, ok<br>}<br><br>func main() {<br>    c := &amp;ConcurrentMap{data: make(map[string]interface{})}<br>    c.Set(&quot;key1&quot;, &quot;value1&quot;)<br>    value, _ := c.Get(&quot;key1&quot;)<br>    println(value)<br>}</pre><p>In this example, we define a ConcurrentMap struct that contains a sync.RWMutex and a data map. The Set and Get methods of the ConcurrentMap struct are used to set and get values in the map, respectively. The Set method acquires a write lock before modifying the map, and the Get method acquires a read lock before reading from the map. This ensures that only one goroutine can write to the map at a time, while multiple goroutines can read from the map simultaneously.</p><p>Please note that the above example is a basic one to understand the concept and it can be improved by adding more functionality like Delete, Iteration and etc.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72ab161ac882" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[create a pod or deployment with specific name and image you can simply run the kubectl run command.]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/create-a-pod-or-deployment-with-specific-name-and-image-you-can-simply-run-the-kubectl-run-command-7db0451ceb66?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7db0451ceb66</guid>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Sat, 24 Sep 2022 20:30:24 GMT</pubDate>
            <atom:updated>2022-09-24T20:30:24.018Z</atom:updated>
            <content:encoded><![CDATA[<h3>create a pod or deployment with a specific name and image you can simply run the kubectl run command.</h3><p><strong>Create an NGINX Pod</strong></p><p>kubectl run nginx --image=nginx</p><p><strong>Generate POD Manifest YAML file (-o yaml). Don’t create it(–dry-run)</strong></p><p>kubectl run nginx --image=nginx --dry-run=client -o yaml</p><p><strong>Create a deployment</strong></p><p>kubectl create deployment --image=nginx nginx</p><p><strong>Generate Deployment YAML file (-o yaml). Don’t create it(–dry-run)</strong></p><p>kubectl create deployment --image=nginx nginx --dry-run=client -o yaml</p><p><strong>Generate Deployment YAML file (-o yaml). Don’t create it(–dry-run) with 4 Replicas (–replicas=4)</strong></p><p>kubectl create deployment --image=nginx nginx --dry-run=client -o yaml &gt; nginx-deployment.yaml</p><p><strong>Save it to a file, make necessary changes to the file (for example, adding more replicas) and then create the deployment.</strong></p><p>kubectl create -f nginx-deployment.yaml</p><p><strong>OR</strong></p><p><strong>In k8s version 1.19+, we can specify the –replicas option to create a deployment with 4 replicas.</strong></p><p>kubectl create deployment --image=nginx nginx --replicas=4 --dry-run=client -o yaml &gt; nginx-deployment.yaml</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7db0451ceb66" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ETCD — Commands (Optional)]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/etcd-commands-optional-76b09a928cb3?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/76b09a928cb3</guid>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[kuberenetes]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Sat, 24 Sep 2022 20:28:08 GMT</pubDate>
            <atom:updated>2022-09-24T20:28:08.024Z</atom:updated>
            <content:encoded><![CDATA[<h3>ETCD — Commands (Optional)</h3><p>(Optional) Additional information about ETCDCTL UtilityETCDCTL is the CLI tool used to interact with ETCD.ETCDCTL can interact with ETCD Server using 2 API versions — Version 2 and Version 3. By default it’s set to use Version 2. Each version has different sets of commands.</p><p>For example, ETCDCTL version 2 supports the following commands:</p><p>etcdctl backup<br>etcdctl cluster-health<br>etcdctl mk<br>etcdctl mkdir<br>etcdctl set</p><p>Whereas the commands are different in version 3</p><p>etcdctl snapshot save<br>etcdctl endpoint health<br>etcdctl get<br>etcdctl put</p><p>To set the right version of API set the environment variable ETCDCTL_API command</p><p>export ETCDCTL_API=3</p><p>When the API version is not set, it is assumed to be set to version 2. And version 3 commands listed above don’t work. When API version is set to version 3, version 2 commands listed above don’t work.</p><p>Apart from that, you must also specify the path to certificate files so that ETCDCTL can authenticate to the ETCD API Server. The certificate files are available in the etcd-master at the following path. We discuss more about certificates in the security section of this course. So don’t worry if this looks complex:</p><p>--cacert /etc/kubernetes/pki/etcd/ca.crt<br>--cert /etc/kubernetes/pki/etcd/server.crt<br>--key /etc/kubernetes/pki/etcd/server.key</p><p>So for the commands, I showed in the previous video to work you must specify the ETCDCTL API version and path to certificate files. Below is the final form:</p><p>kubectl exec etcd-controlplane -n kube-system -- sh -c &quot;ETCDCTL_API=3 etcdctl get / --prefix --keys-only --limit=10 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key&quot;</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76b09a928cb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Imperative Commands with Kubectl]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/imperative-commands-with-kubectl-75422fb1167b?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/75422fb1167b</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[imperative]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Sat, 24 Sep 2022 20:26:59 GMT</pubDate>
            <atom:updated>2022-09-24T20:26:59.133Z</atom:updated>
            <content:encoded><![CDATA[<p>While you would be working mostly the declarative way — using definition files, imperative commands can help in getting one time tasks done quickly, as well as generate a definition template easily. This would help save considerable amount of time during your exams.</p><p>Before we begin, familiarize with the two options that can come in handy while working with the below commands:</p><p>--dry-run: By default as soon as the command is run, the resource will be created. If you simply want to test your command , use the --dry-run=client option. This will not create the resource, instead, tell you whether the resource can be created and if your command is right.</p><p>-o yaml: This will output the resource definition in YAML format on screen.</p><p>Use the above two in combination to generate a resource definition file quickly, that you can then modify and create resources as required, instead of creating the files from scratch.</p><h3>POD</h3><p><strong>Create an NGINX Pod</strong></p><p>kubectl run nginx --image=nginx</p><p><strong>Generate POD Manifest YAML file (-o yaml). Don’t create it(–dry-run)</strong></p><p>kubectl run nginx --image=nginx --dry-run=client -o yaml</p><h3>Deployment</h3><p><strong>Create a deployment</strong></p><p>kubectl create deployment --image=nginx nginx</p><p><strong>Generate Deployment YAML file (-o yaml). Don’t create it(–dry-run)</strong></p><p>kubectl create deployment --image=nginx nginx --dry-run=client -o yaml</p><p><strong>Generate Deployment with 4 Replicas</strong></p><p>kubectl create deployment nginx --image=nginx --replicas=4</p><p>You can also scale a deployment using the kubectl scale command.</p><p>kubectl scale deployment nginx --replicas=4</p><p><strong>Another way to do this is to save the YAML definition to a file and modify</strong></p><p>kubectl create deployment nginx --image=nginx --dry-run=client -o yaml &gt; nginx-deployment.yaml</p><p>You can then update the YAML file with the replicas or any other field before creating the deployment.</p><h3>Service</h3><p><strong>Create a Service named redis-service of type ClusterIP to expose pod redis on port 6379</strong></p><p>kubectl expose pod redis --port=6379 --name redis-service --dry-run=client -o yaml</p><p>(This will automatically use the pod’s labels as selectors)</p><p>Or</p><p>kubectl create service clusterip redis --tcp=6379:6379 --dry-run=client -o yaml (This will not use the pods labels as selectors, instead it will assume selectors as <strong>app=redis. </strong><a href="https://github.com/kubernetes/kubernetes/issues/46191">You cannot pass in selectors as an option.</a> So it does not work very well if your pod has a different label set. So generate the file and modify the selectors before creating the service)</p><p><strong>Create a Service named nginx of type NodePort to expose pod nginx’s port 80 on port 30080 on the nodes:</strong></p><p>kubectl expose pod nginx --type=NodePort --port=80 --name=nginx-service --dry-run=client -o yaml</p><p>(This will automatically use the pod’s labels as selectors, <a href="https://github.com/kubernetes/kubernetes/issues/25478">but you cannot specify the node port</a>. You have to generate a definition file and then add the node port in manually before creating the service with the pod.)</p><p>Or</p><p>kubectl create service nodeport nginx --tcp=80:80 --node-port=30080 --dry-run=client -o yaml</p><p>(This will not use the pods labels as selectors)</p><p>Both the above commands have their own challenges. While one of it cannot accept a selector the other cannot accept a node port. I would recommend going with the `kubectl expose` command. If you need to specify a node port, generate a definition file using the same command and manually input the nodeport before creating the service.</p><h3>Reference:</h3><p><a href="https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands">https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands</a></p><p><a href="https://kubernetes.io/docs/reference/kubectl/conventions/">https://kubernetes.io/docs/reference/kubectl/conventions/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75422fb1167b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang Application Development for Beginners (part 2)]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/golang-application-development-for-beginners-part-1-1d8af0634ec9?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d8af0634ec9</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Mon, 12 Sep 2022 10:09:56 GMT</pubDate>
            <atom:updated>2022-09-12T17:45:21.620Z</atom:updated>
            <content:encoded><![CDATA[<p>I hope everyone completed the project setup from my previous blog</p><h4>Database configuration</h4><p>Create a new directory called <em>config, </em>inside that create a new file database-config.go</p><pre>package config<br><br>import (<br>   &quot;fmt&quot;<br>   &quot;github.com/joho/godotenv&quot;<br>   &quot;gorm.io/driver/mysql&quot;<br>   &quot;gorm.io/gorm&quot;<br>   &quot;os&quot;<br>)<br><br>func SetupDatabaseConnection() *gorm.DB {<br><br>   if errEnv := godotenv.Load(); errEnv != nil {<br>      panic(&quot;Failed to load an .env file&quot;)<br>   }<br><br>   dbUser := os.Getenv(&quot;DB_USER&quot;)<br>   dbPass := os.Getenv(&quot;DB_PASS&quot;)<br>   dbHost := os.Getenv(&quot;DB_HOST&quot;)<br>   dbName := os.Getenv(&quot;DB_NAME&quot;)<br><br>   dsn := fmt.Sprintf(&quot;%s:%s@tcp(%s:3306)/%s?charset=utf8mb4&amp;parseTime=True&amp;loc=Local&quot;, dbUser, dbPass, dbHost, dbName)<br>   db, err := gorm.Open(mysql.Open(dsn), &amp;gorm.Config{})<br>   if err != nil {<br>      panic(&quot;Failed to create a connection to the database&quot;)<br>   }<br>   fmt.Println(&quot;hello&quot;)<br>   return db<br>}</pre><p><strong>close existing database connection</strong></p><pre>func CloseDatabaseConnection(db *gorm.DB) {<br>   dbSQl, err := db.DB()<br>   if err != nil {<br>      panic(&quot;Failed to close database connection&quot;)<br>   }<br>   dbSQl.Close()<br>}</pre><h3><strong>Let&#39;s start writing Entity</strong></h3><p>create entity package</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6RrXGqQmCogvs4M6w1kHYQ.png" /></figure><p><strong>user entity</strong></p><p>create user.go in entity package</p><pre>package entity<br><br>type User struct {<br>   ID       uint64 `gorm:&quot;primaryKey:auto_increment&quot; json:&quot;id&quot;`<br>   Name     string `gorm:&quot;type:varchar(255)&quot; json:&quot;name&quot;`<br>   Email    string `gorm:&quot;uniqueIndex;type:varchar(255)&quot; json:&quot;email&quot;`<br>   password string `gorm:&quot;-&gt;;&lt;-;not null&quot; json:&quot;-`<br>   Token    string `gorm:&quot;-&quot; json:&quot;-,omitempty&quot;`<br>}</pre><p><strong>book entity</strong></p><p>create book.go in entity package</p><pre>package entity<br><br>type Book struct {<br>   ID          uint64 `gorm:&quot;primaryKey:auto_increment&quot; json:&quot;id&quot;`<br>   Title       string `gorm:&quot;type:varchar(255)&quot; json:&quot;title&quot;`<br>   Description string `gorm:&quot;type:text&quot; json:&quot;description&quot;`<br>   UserID      uint64 `gorm:&quot;not null&quot; json:&quot;-&quot;`<br>   User        User   `gorm:&quot;foreignKey:UserID;constraint:onUpdate:CASCADE,onDelete:CASCADE&quot; json:&quot;user&quot;`<br>}</pre><p>GORM Fields Tags: <a href="https://gorm.io/docs/models.html#Fields-Tags">read here</a></p><p>GOLang Fields Tags: <a href="https://www.geeksforgeeks.org/how-to-use-field-tags-in-the-definition-of-struct-type-in-golang/">read here</a></p><h3><strong>Common API Response</strong></h3><h4>create response helper</h4><p>create a helper package then inside that create a response.go</p><pre>// Response - Common API Response Structure<br>type Response struct {<br>   Success bool        `json:&quot;success&quot;`<br>   Message string      `json:&quot;message&quot;`<br>   Error   interface{} `json:&quot;errors,omitempty&quot;`<br>   Data    interface{} `json:&quot;data,omitempty&quot;`<br>}</pre><pre>// EmptyObj - If you don&#39;t wish to send data as nil then send this Empty Object<br>type EmptyObj struct{}</pre><p>Build Success Response</p><pre>// BuildResponse method generate dynamic success response<br>func BuildResponse(mesaage string, data interface{}) Response {<br>   return Response{<br>      Success: true,<br>      Message: mesaage,<br>      Data:    data,<br>   }<br>}</pre><p>Build Error Response</p><pre>// BuildErrorResponse method generate dynamic error Response<br>func BuildErrorResponse(mesaage string, error string) Response {<br>   errors := strings.Split(error, &quot;\n&quot;)<br>   return Response{<br>      Success: false,<br>      Message: mesaage,<br>      Error:   errors,<br>   }<br>}</pre><p>We will meet in next part</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d8af0634ec9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linux For Newbies (part -3)]]></title>
            <link>https://medium.com/@naveenkumarkarthigayan/linux-for-newbies-part-3-d539c29f0969?source=rss-9f62ce466dd3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d539c29f0969</guid>
            <category><![CDATA[linux-boot-process]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[systemd]]></category>
            <category><![CDATA[booting]]></category>
            <dc:creator><![CDATA[Naveenkumar Karthigayan]]></dc:creator>
            <pubDate>Fri, 09 Sep 2022 09:56:11 GMT</pubDate>
            <atom:updated>2022-09-09T09:56:11.530Z</atom:updated>
            <content:encoded><![CDATA[<h3>Linux Booting Process -(modern Linux)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*zicreNXPj_MwBcfCDUyRrw.jpeg" /></figure><p>Every time you power on your Linux PC, it goes through a series of stages before finally displaying a login screen that prompts for your username or password. There are 4 distinct stages that every Linux distribution goes through in a typical boot-up process.</p><p>In this guide, we will highlight the various steps the Linux OS takes from when it is powered on to when you log in. Kindly note that this guide only considers the <strong>GRUB2</strong> bootloader and <strong>systemd</strong> init as they are currently in use by a vast majority of modern Linux distributions.</p><p>The booting process takes the following 4 steps that we will discuss in greater detail:</p><ol><li>BIOS Integrity check (<strong>POST</strong>)</li><li>Loading of the Boot loader (<strong>GRUB2</strong>)</li><li>Kernel initialization</li><li>Starting <strong>systemd</strong>, the parent of all processes</li></ol><h3>1. The BIOS Integrity Check (POST)</h3><p>The boot process is usually initialized when a user presses the <strong>power-on</strong> button — if the PC was already shut down — or reboots the system using either the GUI or the command line.</p><p>When the Linux system powers up, the <strong>BIOS</strong> (<strong>Basic Input Output System</strong>) kicks in and performs a <strong>Power On Self Test</strong> (<strong>POST</strong>). This is an integrity check that performs a plethora of diagnostic checks.</p><p>The <strong>POST</strong> probes the hardware operability of components such as the <strong>HDD</strong> or <strong>SSD</strong>, <strong>Keyboard</strong>, <strong>RAM</strong>, <strong>USB</strong> ports, and any other piece of hardware. If some hardware device is not detected, or if there’s a malfunction in any of the devices such as a corrupt HDD or SSD, an error message is splashed on the screen prompting your intervention.</p><p>In some cases, a beeping sound will go off, especially in the event of a missing RAM module. However, if the expected hardware is present and functioning as expected, the booting process proceeds to the next stage.</p><h3>2. The Bootloader (GRUB2)</h3><p>Once the <strong>POST</strong> is complete and the coast is clear, the <strong>BIOS</strong> probes the <strong>MBR</strong> (<strong>Master Boot Record</strong>) for the bootloader and disk partitioning information.</p><p>The <strong>MBR</strong> is a 512-byte code that is located on the first sector of the hard drive which is usually /dev/sda or /dev/hda depending on your hard drive architecture. Note, however, that sometimes the <strong>MBR</strong> can be located on a <strong>Live USB</strong> or <strong>DVD</strong> installation of Linux.</p><p>There are 3 main types of bootloaders in Linux: <strong>LILO</strong>, <strong>GRUB,</strong> and <strong>GRUB2</strong>. The <strong>GRUB2</strong> bootloader is the latest and primary bootloader in modern Linux distributions and informs our decision to leave out the other two which have become antiquated with the passage of time.</p><p><strong>GRUB2</strong> stands for <strong>GRand Unified Bootloader</strong> version 2. Once the <strong>BIOS</strong> locates the grub2 bootloader, it executes and loads it onto the main memory (<strong>RAM</strong>).</p><p>The <strong>grub2</strong> menu allows you to do a couple of things. It allows you to select the Linux kernel version that you’d want to use. If you have been upgrading your system a couple of times, you might see different kernel versions listed. Additionally, it gives you the ability to edit some kernel parameters by pressing a combination of keyboard keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*wbxLt_6FkPwN0c2bfyLwoA.png" /></figure><p>Also, in a dual-boot setup where you have <a href="https://www.tecmint.com/install-multiple-centos-servers-using-ftp/">multiple OS installations</a>, the grub menu allows you to select which OS to boot into. The grub2 configuration file is the <strong>/boot/grub2/grub2.cfg</strong> file. GRUB’s main objective is to load the Linux kernel onto the main memory.</p><h3>3. Kernel Initialization</h3><p>The kernel is the core of any Linux system. It interfaces the PC’s hardware with the underlying processes. The kernel controls all the processes on your Linux system. Once the selected Linux kernel is loaded by the bootloader, it must self-extract from its compressed version before undertaking any task. Upon self-extracting, the selected kernel mounts the root file system and initializes the <strong>/sbin/init</strong> program commonly referred to as <strong>init</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*6hztpyVJ1uEeBXs4CNlshQ.png" /></figure><p><strong>Init</strong> is always the first program to be executed and is assigned the process ID or PID of 1. It’s the init process that spawns various daemons &amp; mounts all partitions that are specified in the <strong>/etc/fstab</strong> file.</p><p>The kernel then mounts the initial RAM disk (<strong>initrd</strong>) which is a temporary root filesystem until the real root filesystem is mounted. All kernels are located in the /boot directory together with the initial RAM disk image.</p><h3>4. Starting Systemd</h3><p>The kernel finally loads <strong>Systemd</strong>, which is the replacement of the old <strong>SysV</strong> init. <strong>Systemd</strong> is the mother of all Linux processes and manages among other things mounting of file systems, and starting and stopping services to mention just a few.</p><p>Systemd uses the <strong>/etc/systemd/system/default.target</strong> file to determine the state or target that the Linux system should boot into.</p><ul><li>For a desktop workstation (with a GUI) the default target value is 5 which is the equivalent of run level 5 for the old SystemV init.</li><li>For a server, the default target is <strong>multi-user.target</strong> which corresponds to run level 3 in SysV init.</li></ul><p>Here’s a breakdown of the systemd targets:</p><ul><li><strong>poweroff.target (runlevel 0)</strong>: Poweroff or Shutdown the system.</li><li><strong>rescue.target (runlevel 1)</strong>: launches a rescue shell session.</li><li><strong>multi-user.target (runlevel 2,3,4)</strong>: Configures the system to a non-graphical (console) multi-user system.</li><li><strong>graphical.target (runlevel 5)</strong>: Set the system to use a graphical multi-user interface with network services.</li><li><strong>reboot.target (runlevel 6)</strong>: reboots the system.</li></ul><p>To check the current target on your system, run the command:</p><pre>$ systemctl get-default</pre><p>You can switch from one target to another by running the following command on the terminal:</p><pre>$ init runlevel-value</pre><p>For example, init 3 configures the system to a non-graphical state.</p><p>The <strong>init 6</strong> command reboots your system and <strong>init 0</strong> powers off the system. Be sure to invoke <strong>sudo command</strong> when you want to switch to these two targets.</p><p>The booting process ends once <strong>systemd</strong> loads all the daemons and sets the target or run level value. It’s at this point you are prompted for your username and password upon which you gain entry to your Linux system.</p><p><strong>We will see in Part 4: coming soon…</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d539c29f0969" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>