<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Erdem Çulcu on Medium]]></title>
        <description><![CDATA[Stories by Erdem Çulcu on Medium]]></description>
        <link>https://medium.com/@erdemculcu?source=rss-dafb5e4539e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8aGSehYoBe1HyjIzIcJSuQ.jpeg</url>
            <title>Stories by Erdem Çulcu on Medium</title>
            <link>https://medium.com/@erdemculcu?source=rss-dafb5e4539e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:37:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@erdemculcu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Move Your Company From “Good” to “Great”]]></title>
            <link>https://medium.com/@erdemculcu/how-to-move-your-company-from-good-to-great-95636058208f?source=rss-dafb5e4539e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/95636058208f</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[bookreading]]></category>
            <dc:creator><![CDATA[Erdem Çulcu]]></dc:creator>
            <pubDate>Thu, 26 Feb 2026 10:05:34 GMT</pubDate>
            <atom:updated>2026-02-26T10:05:34.242Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xMMl1mNTap44SzydP_B-JQ.jpeg" /></figure><p>Every day, we grapple with complex issues like growth targets, market competition, and talent management. Recently, upon the recommendation of Aykut Karaalioglu, I picked up Jim Collins’ legendary book, <em>Good to Great</em>. The notes I took while reading provided a brand-new perspective on our company’s current state. I’ve tried to distill these insights into writing. We often focus on what we should do to become “better”; however, great companies look not only at what they should do, but also at what they should <strong>stop</strong> doing.</p><p>Here are the vital takeaways filtered from my notes and the highlighted sections of the book:</p><h3>1. “Who” Comes Before Vision: Getting the Right People on the Bus</h3><p>In meetings, our first topic is usually “Where are we going?” However, the book confronted me with a jolting reality: Leaders must first get the right people on the bus, get everyone in the right seats, and <em>then</em> decide where the bus is going. Our most valuable asset is not “people,” it is <strong>“the right people.”</strong></p><p>If we feel love and respect for the people we spend most of our lives with, and if we enjoy traveling on the same bus with them, then no matter where the bus goes, that journey turns into a great life. The book suggests two questions when building teams:</p><ol><li><strong>Do I feel the need to constantly manage this employee?</strong> If the answer is yes, I made a mistake hiring them in the first place, because the best people don’t need to be managed.</li><li><strong>Would I hire this person again today?</strong> Or, if they told me they wanted to leave, would I be secretly relieved?</li></ol><p>If doubt persists, do not hire that person and keep looking for the “right” one. We must place our best talent in front of our greatest opportunities, not our greatest problems.</p><h3>2. Level 5 Leadership: Company Love Over Ego</h3><p>Contrary to the image of the high-profile “savior” CEO, the “Level 5 Leaders” who take companies to greatness are understated, quiet, and even shy. They resemble Lincoln and Socrates more than Patton or Caesar.</p><p>The “Mirror and the Window” metaphor moved me most: True leaders look out the <strong>window</strong> to find someone else to credit when things go well; they look in the <strong>mirror</strong> to take responsibility when things go poorly. We must be humble, yet display an unwavering resolve when doing what is necessary for the company.</p><h3>3. A Culture of Discipline: Growing Without the Bureaucracy Trap</h3><p>As companies grow, things get complicated. That initial visionary and creative entrepreneurial spirit can eventually be replaced by “growth pains” and the bureaucracy introduced by professional managers. Rules, forms, and “us-versus-them” mentalities kill the company’s soul, and mediocrity begins to eat the organization from within.</p><p>How do we avoid bureaucracy? The answer: The Creative Discipline Matrix. We must create a culture of discipline — not to be confused with a dictatorship — and build a framework centered around freedom and responsibility. When we combine a culture of discipline with an entrepreneurial spirit, we stop being a “Bureaucratic Organization” and transform into a “Great Organization” with superior performance. As managers, we should start creating a <strong>“stop-doing list”</strong> today.</p><h3>4. The Stockdale Paradox and Confronting Reality</h3><p>In tough competitive conditions, morale can waver. Citing the example of Kimberly-Clark facing the giant P&amp;G, the book notes that CEO Darwin Smith saw this competition not as a burden, but as an opportunity. Instead of feeling weaker than their rivals, they faced the brutal facts and said, “We will never surrender. We will find a way to prevail in the end.” This echoed Aykut’s words in my mind: <em>Competing with the biggest is an opportunity for those like us.</em></p><p>The philosophy a manager must adopt is the <strong>Stockdale Paradox</strong>: Confront the most brutal facts of your current reality, whatever they may be, but never lose faith that you will prevail in the end.</p><h3>5. Focusing with the Hedgehog Concept</h3><p>Instead of being like a fox — chasing many complex things at once — we should be like a hedgehog, viewing the world through a single, simple concept. We must seat our company strategy at the intersection of these three circles, our “sacred ground”:</p><ul><li>What can we be the best in the world at?</li><li>What drives our economic engine?</li><li>What are we deeply passionate about?</li></ul><h3>Final Word</h3><p>The road from “good to great” is not a miraculous leap that happens overnight; it is a <strong>Flywheel</strong> composed of leadership, the right people, brutal honesty, and disciplined steps. If we, as leaders, focus on the work without caring who gets the credit, we can carry our organizations toward lasting excellence.</p><p>As the author says: “If you’re looking for an answer to the question ‘Why should we be great?’, you’re probably in the wrong business.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95636058208f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pyspark Output Destinations]]></title>
            <link>https://medium.com/mobileaction-tech/pyspark-output-destinations-dfe151f1cd5f?source=rss-dafb5e4539e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfe151f1cd5f</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[parquet]]></category>
            <dc:creator><![CDATA[Erdem Çulcu]]></dc:creator>
            <pubDate>Mon, 20 Mar 2023 07:49:22 GMT</pubDate>
            <atom:updated>2023-03-20T07:49:22.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*od8CVSZu83bcqG0Trw8N9Q.jpeg" /><figcaption>Pyspark Output Destinations</figcaption></figure><p><a href="https://spark.apache.org/docs/latest/api/python/">Pyspark</a> is a powerful tool for processing large datasets and performing complex computations. When working with data in Pyspark, it is essential to understand the different output destinations available for your results. In this blog post, we will explore the various output destinations in Pyspark and how to use them effectively.</p><h4><strong>Console Output</strong></h4><p>Console output is the simplest and most commonly used output destination in Pyspark. When you print a data frame or any other object in Pyspark, the output is printed to the console by default. This output destination is useful for quickly inspecting data and debugging code. You can use the show() method to print the first few rows of a data frame, and print() to print any other object.</p><pre># Print first few rows of a dataframe<br>df.show()<br><br># Print an object<br>print(&quot;Hello World!&quot;)</pre><h4><strong>File Output</strong></h4><p>File output is used when you want to save the results of your computations to a file. Pyspark supports writing to various file formats, including CSV, JSON, <a href="https://parquet.apache.org/">Parquet</a>, and more. You can use the write method to write data to a file. The following example shows how to save a data frame to a CSV file.</p><pre># Write dataframe to CSV file<br>df.write.csv(&quot;output.csv&quot;)</pre><p><a href="https://aws.amazon.com/s3/?nc1=h_ls">Amazon S3</a> is a cloud storage service provided by Amazon Web Services (AWS) that allows storing and retrieving data from anywhere on the web. With Pyspark, we can write data directly to S3 using the s3a protocol.</p><pre># Writing a dataframe to S3 in json format<br>df.write.format(&#39;json&#39;).mode(&#39;overwrite&#39;).option(&#39;compression&#39;, &#39;gzip&#39;).save(&#39;s3a://bucket-name/path/to/destination&#39;)</pre><h4><strong>Database Output</strong></h4><p>Database output is used when you want to write data to a database. Pyspark supports writing to various databases, including MySQL, Postgres, and more. You can use the jdbc method to write data to a database. The following example shows how to save a data frame to a MySQL database</p><pre># Write dataframe to MySQL database<br>df.write.format(&quot;jdbc&quot;) \<br>    .option(&quot;url&quot;, &quot;jdbc:mysql://localhost/mydatabase&quot;) \<br>    .option(&quot;dbtable&quot;, &quot;mytable&quot;) \<br>    .option(&quot;user&quot;, &quot;myuser&quot;) \<br>    .option(&quot;password&quot;, &quot;mypassword&quot;) \<br>    .save()</pre><h4><strong>Streaming Output</strong></h4><p>Streaming output is used when you want to process data in real time. Pyspark supports streaming data from various sources, including <a href="https://kafka.apache.org/documentation/streams/">Kafka</a>, <a href="https://flume.apache.org/">Flume</a>, and more. You can use the writeStream method to write streaming data to a destination. The following example shows how to write streaming data to a Kafka topic.</p><pre># Write streaming data to Kafka topic<br>df.writeStream \<br>    .format(&quot;kafka&quot;) \<br>    .option(&quot;kafka.bootstrap.servers&quot;, &quot;localhost:9092&quot;) \<br>    .option(&quot;topic&quot;, &quot;mytopic&quot;) \<br>    .start()</pre><p>In conclusion, Pyspark provides several output destinations to suit various use cases. The console output is useful for a quick inspection, while file output is used for saving data to files. Database output is used for writing data to databases, and streaming output is used for processing data in real time. Understanding these output destinations is essential for effectively using Pyspark and ensuring that your computations are written to the appropriate destination.</p><h4><strong>References</strong></h4><ol><li><a href="https://spark.apache.org/docs/latest/api/python/">https://spark.apache.org/docs/latest/api/python/</a></li><li><a href="https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python.html">https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfe151f1cd5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mobileaction-tech/pyspark-output-destinations-dfe151f1cd5f">Pyspark Output Destinations</a> was originally published in <a href="https://medium.com/mobileaction-tech">MobileAction Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Partitioning Challenge on AWS with S3, Glue, and Athena]]></title>
            <link>https://medium.com/mobileaction-tech/partitioning-challenge-on-aws-with-s3-glue-and-athena-f6bdb14448e9?source=rss-dafb5e4539e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6bdb14448e9</guid>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[partitioning]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Erdem Çulcu]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 05:22:30 GMT</pubDate>
            <atom:updated>2022-12-01T05:43:40.439Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qb79oujpeqtrSnu9eANthQ.jpeg" /></figure><p>We’ll continue our challenge on the <a href="https://medium.com/p/e565b12cc793">previous article</a> with a different perspective, we will talk about the partitioning problem that causes the system to change shape.</p><h3>What is partitioning?</h3><p>In S3, we can use folders as <strong>Partitions</strong> to group our data into physical entities. However, in Glue Data Catalog or Hive Metastore, we use partitions as logical entities with metadata properties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*cYQS4fehmVtZRzBZ.png" /><figcaption>S3 Folder Structure</figcaption></figure><h3>Why is partitioning important?</h3><p>Optimizing query capabilities is important when using data warehouses and data lakes like S3 instead of RDBMS.</p><p>In our environment on AWS, we are using Athena to query our data on S3. By partitioning our data, we can restrict the amount of data scanned by each query, thus improving performance and reducing the cost.</p><p>In S3, you can partition your data by any key. A common practice is to partition the data based on time, often leading to a multi-level partitioning scheme. For example, a customer who has data coming in every hour might decide to partition by year, month, date, and hour.</p><h3>Athena And Partitions</h3><p>Athena can use Apache Hive style partitions, whose data paths contain key-value pairs connected by equal signs (for example, country=us/... or year=2021/month=01/day=26/...). Thus, the paths include both the names of the partition keys and the values that each path represents.</p><blockquote>To load new Hive partitions into a partitioned table, you can use the MSCK REPAIR TABLE command, which works only with Hive-style partitions.</blockquote><p>Athena can also use non-Hive style partitioning schemes. For example, CloudTrail logs and Kinesis Data Firehose delivery streams use separate path components for date parts such as data/2021/01/26/us/6fc7845e.json.</p><blockquote>For such non-Hive style partitions, you can use ALTER TABLE ADD PARTITION to add the partitions manually.</blockquote><p>Glue Data Crawler can be used to add new partitions metadata when Data Catalogs are used. It seeks all partitions on S3 and adds new partitions&#39; metadata into the catalog.</p><h3>Partition Limits</h3><p>Amazon Web Services support partitioning on your data but, almost all of the tools have limits for partitions.</p><p><strong>Kinesis Data Firehose:</strong> Default 500 active partitions in a buffer. It can be up to 5000 from AWS Support. We talked about this <a href="https://medium.com/p/e565b12cc793">previous article</a>.</p><p><strong>Glue Data Catalog:</strong> Max partitions per table for each supported region is 10,000,000.</p><h3>Our Usage and Challenges with Partitioning</h3><p>In the first iteration, we set up three Kinesis Data Firehoses for different partition keys and different S3 locations. The limits weren’t enough for a bucket that was partitioned daily by high range date. This was the first challenge with partitioning. The partition keys were very important for our queries. Hence, we could not give up on it. We had to find a different solution to solve this problem.</p><h4>Partitioning with Glue Spark</h4><p>The solution was a Glue Spark Job for partitioning the data within a second iteration according to our keys and copying it to another S3 bucket. We set partition keys as the date and customer id and execute scheduled spark jobs to prepare data for our needs.</p><h4>Updating the Glue Data Catalog</h4><p>After completing the spark jobs, a second problem appeared. The new partitions from Spark Job were not recognized by the Data Catalog and Athena queries. We’ve already discussed some solutions to this problem, but:</p><ul><li>MSCK REPAIR TABLE is very slow for huge tables.</li><li>Data Crawler is an extra cost and needs more time for huge tables.</li></ul><h4>Partition Projection</h4><p>In partition projection, partition values and locations are calculated from configuration rather than reading from a repository like the AWS Glue Data Catalog. Since in-memory operations are often faster than remote operations, partition projection can reduce the runtime of queries against highly partitioned tables. Depending on the specific characteristics of the query and underlying data, partition projection can significantly reduce query runtime for queries that are constrained on partition metadata retrieval.</p><p>For example a partition projection :</p><pre>PARTITIONED BY (<br> datehour STRING<br>)<br>LOCATION &quot;s3://DOC-EXAMPLE-BUCKET/prefix/&quot;<br>TBLPROPERTIES (<br> &quot;projection.enabled&quot; = &quot;true&quot;,<br> &quot;projection.datehour.type&quot; = &quot;date&quot;,<br> &quot;projection.datehour.format&quot; = &quot;yyyy/MM/dd/HH&quot;,<br> &quot;projection.datehour.range&quot; = &quot;2021/01/01/00,NOW&quot;,<br> &quot;projection.datehour.interval&quot; = &quot;1&quot;,<br> &quot;projection.datehour.interval.unit&quot; = &quot;HOURS&quot;,<br> &quot;storage.location.template&quot; = &quot;s3://DOC-EXAMPLE-BUCKET/prefix/${datehour}/&quot;<br>)</pre><p>Tataaa!! We had a new problem :) Our customer id range is 1– 300.000. If we set the customer id and date, data catalog partition limits are exceeded. Since these fields are the main WHERE clauses in our queries, are directly affecting our query performance.</p><blockquote>Of course, we are very happy that we have 300 thousand registered users and this number continues to grow.</blockquote><h4>Partition Injection</h4><p>Luckily partition injection comes to the rescue. The problematic field has the following characteristics:</p><ul><li>An extremely high number (potentially billions) of values.</li><li>Since its values are random strings, it is not projectable using other projection methods.</li><li>The extremely large number of partitions cannot be stored in commonly used meta stores.</li></ul><p>However, if all of your queries include a WHERE clause that filters for only a single id, you can use the following approach in your CREATE TABLE statement.</p><pre>PARTITIONED BY<br> (<br>     device_id STRING<br> )<br> LOCATION &quot;s3://bucket/prefix/&quot;<br> TBLPROPERTIES<br> (<br>     &quot;projection.enabled&quot; = &quot;true&quot;,<br>     &quot;projection.device_id.type&quot; = &quot;injected&quot;,<br>     &quot;storage.location.template&quot; = &quot;s3://bucket/prefix/${device_id}&quot;<br> )</pre><p>A SELECT query on this table would look like the following:</p><pre>SELECT<br>  col1,<br>  col2,...,<br>  device_id<br>FROM<br>  table<br>WHERE<br>  device_id = &quot;b6319dc2-48c1-4cd5-a0a3-a1969f7b48f7&quot;<br>  AND (<br>    col1 &gt; 0<br>    or <br>    col2 &lt; 10<br>  )</pre><h3>End of the day</h3><p>After these experiences, we partitioned buckets with the required partition keys. Our queries are working with high performance. Our cloud adventure stories will be continued. If you wonder more, please keep following us.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6bdb14448e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mobileaction-tech/partitioning-challenge-on-aws-with-s3-glue-and-athena-f6bdb14448e9">Partitioning Challenge on AWS with S3, Glue, and Athena</a> was originally published in <a href="https://medium.com/mobileaction-tech">MobileAction Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developing a High-Traffic Data Service With AWS Kinesis]]></title>
            <link>https://medium.com/mobileaction-tech/developing-a-high-traffic-data-service-with-aws-kinesis-1-e565b12cc793?source=rss-dafb5e4539e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e565b12cc793</guid>
            <category><![CDATA[kinesis]]></category>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[high-traffic]]></category>
            <dc:creator><![CDATA[Erdem Çulcu]]></dc:creator>
            <pubDate>Mon, 07 Nov 2022 16:07:22 GMT</pubDate>
            <atom:updated>2022-11-30T10:31:06.162Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi everyone,</p><p>In this series of articles, we will try to give information about our method of storing and processing incoming high-traffic data that we have used on one of our projects.</p><p>The topics that we will briefly touch on in this series of articles will be as follows:</p><ul><li>Service Needs</li><li>Cloud solutions for problems: Kinesis Data Stream, Data Firehose, S3 Buckets,</li><li>Data ETL Tools (Extract, Transform, and Load): Glue Spark, Glue Data Catalog</li><li>Query Optimization: Athena, Partitioning</li><li>Conclusion</li></ul><h3>Service Needs</h3><p>Our service collects data from external data sources with an average of 1000 webhook requests per second. We expect this density to reach 10 thousand according to the new customer potential.</p><p>We need to store and query this data both in granular and aggregated forms. We use an RDBMS database to join the aggregated results with other tables.</p><h3>Solutions For Problems</h3><p>In this section, let’s briefly talk about the solutions used for the related problems and give the details in the next sections.</p><ul><li>Handle high load requests → Data Streaming method: Ability to convert incoming data to stream and handle high requests. Related technology: AWS Kinesis Data Stream</li><li>Keeping the data in a warehouse instead of an RDBMS → Data warehouse: Querying the data directly from the stream by writing to AWS S3 in a partitioned form: AWS Kinesis Firehose</li><li>Repartition data according to query needs → Data Partitioning and Transform: Spark Glue Jobs</li><li>Ability to query with SQL, which is simple, understandable, easy to maintain and manage, to use data effectively: AWS Glue Data Catalog</li><li>Ability to run high-performance queries in the relevant data warehouse without moving the data elsewhere: Amazon Athena</li></ul><p>After summarizing the current situation, let’s start to explain the part of meeting high-traffic data with Kinesis, which is the main topic of this article. We’ll have follow-up articles about other topics in the future.</p><h3>AWS Kinesis</h3><p>A suite of products developed by Amazon to easily collect, process and analyze data streams in real-time.</p><blockquote><a href="https://aws.amazon.com/kinesis/">Amazon Kinesis</a> makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and responds instantly instead of having to wait until all your data is collected before the processing can begin.</blockquote><h3>Advantages</h3><ul><li>Real-time: Amazon Kinesis enables you to ingest, buffer, and process streaming data in real-time, so you can derive insights in seconds or minutes instead of hours or days.</li><li>Fully managed: Amazon Kinesis is fully managed and runs your streaming applications without requiring you to manage any infrastructure.</li><li>Scalable: Amazon Kinesis can handle any amount of streaming data and process data from hundreds of thousands of sources with very low latencies.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrbMmf2a5woxNXUu0XioaQ.png" /><figcaption>AWS Kinesis Tools</figcaption></figure><h3>Kinesis and High-Load Data</h3><p>The main point we changed in the current system was to use the advantages of the scalable structure by converting the incoming data to the format we want and transferring it to the data stream, instead of queuing and processing. In this way, we would be able to reach high RPS easily and remove our bottleneck parts such as RabbitMQ and MySQL.</p><p>It is possible to do the related transformation with Lambda on the AWS side, but it brings high costs in high RPS systems. Since lambda costs are calculated as ‘per request’, the cost increases exponentially as the request increases.</p><p>By routing the incoming stream data with Kinesis Firehose to S3, we achieved a data warehouse solution instead of the traditional relational database. Let’s continue with how we use these tools, their limitations, and their challenges.</p><h3>AWS Kinesis Data Streams</h3><p>Amazon Kinesis Data Streams is a serverless streaming service that makes it easy to capture, process, and store data streams at any scale. The most important factor in selecting the Kinesis Data Streams is; it does not have a server to manage and it can dynamically provide the required capacity with an on-demand mode.</p><h3>How does it work?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Gvd27PKXoVw6qEmCeh6jA.png" /><figcaption>AWS Kinesis Data Stream</figcaption></figure><p>The diagram below shows the architecture of the Kinesis Data Stream. It can process and analyze data in real-time.</p><p>The output of the Kinesis Data Stream can be an input of another system. For a stream, multiple applications can also use the data independently and in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPpudvXycUHPXg5SAhQnJA.png" /></figure><h3>Components</h3><p>The main component of the Kinesis Data Stream structures is called <strong>shard</strong>s.</p><ul><li><strong>Shard</strong>: The shard, which is defined as a uniquely defined set of data records, creates a data stream by combining one or more. Each shard offers a fixed capacity. 5 transactions/sec per shard provides a total of 2 MB/s of read, while in writing it provides 1,000 records per second, a total capacity of 1 MB/sec. The capacity of the stream varies depending on how many shards it contains.</li><li><strong>Data Record:</strong> It is the type of data record written into the shard. It basically consists of three components. These are called partition key, sequence number, and data blob.</li><li><strong>Partition Key:</strong> It is the key used to group data among shards. The relevant key hash output is modded with the number of shards, and it is decided which shard the data will be placed on. The maximum length can be 256 characters.</li><li><strong>Sequence Number:</strong> It is the sequence number of each data record in the relevant shard.</li><li><strong>Payload:</strong> It is the part where the data is written. It is used by converting JSON data type to byte array.</li></ul><h3>Quotas and Limits</h3><p>There is no upper limit for creating data streams in your AWS account. However, we can create a maximum of 50 data streams in on-demand mode. This limit can be increased per account by AWS Support (500 is the hard limit). With a simple calculation, it can grow up to 500,000 RPS with a 500x1000 write record.</p><p>On the other hand, Kinesis Firehose has a maximum 2000 RPS limit. But if we feed the Firehose with DataStreams, the limit is not applied.</p><p>The maximum payload size on Kinesis is 1 MB before BASE-64 encoding.</p><p>Another life-saving feature is the Data Retention Period, which can be adjusted between 24 hours and 365 days, higher numbers bring more cost.</p><p>In cases where we cannot record the data properly, knowing that there is raw data in Data Stream gives confidence while building the system.</p><h3>API Samples,</h3><pre>ObjectMapper mapper = new ObjectMapper();             PutRecordRequest putRecordRequest = PutRecordRequest.builder().streamName(kinesisDeliveryStreamName)                     .data(SdkBytes.fromByteArray(mapper.writeValueAsBytes(event)))                     .partitionKey(Timestamp.from(Instant.now()).toString()).build();             kinesisClient.putRecord(putRecordRequest);</pre><p>This example demonstrates how we send an incoming event object to the Data Stream with Java.</p><h3>AWS Kinesis Data Firehose</h3><p>Data Firehose reliably loads real-time streams into data lakes, warehouses, and analytics services. In our new system, Data Firehose has been positioned as an intermediary that carries the data from the stream to S3. As with all Kinesis tools, Firehose can also scale automatically without the need for any manual operation. It converts incoming data to formats such as Apache Parquet and dynamically partitions stream data without creating any processing lines.</p><p>We will talk about dynamic partitioning and its importance in more detail in the next stages; but to summarize, much more performance results can be achieved with partitioning. It represents an index-like structure in traditional databases.</p><p>Firehose can connect to more than 30 fully integrated AWS services and streaming targets such as Amazon Simple Storage Service (S3) and Amazon Redshift. We selected S3 as our target to minimize the costs and benefit from Athena.</p><h3>How does it work?</h3><p><a href="https://aws.amazon.com/kinesis/data-firehose/">Amazon Kinesis Data Firehose</a> is an extract, transform, and load (ETL) service that reliably captures, transforms and delivers streaming data to data lakes, data stores, and analytics services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sa2g-R5BrpiysaGQc9d6SQ.png" /><figcaption>AWS Data Firehose</figcaption></figure><p>Data received from Kinesis Data Stream or an SDK can be converted with Lambda, but it should not be forgotten that Lambda cost is also present. Firehose itself can convert to Apache Parquet or Apache ORC formats without the need for any Lambda. The Parquet format will be the most suitable option if the data need to be queried. Recent studies propose that Parquet has obvious superiority over JSON.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*AtmIWYIop4NfnFl2FbprGg.png" /><figcaption><a href="https://weber-stephen.medium.com/csv-vs-parquet-vs-json-for-data-science-cf3733175176">https://weber-stephen.medium.com/csv-vs-parquet-vs-json-for-data-science-cf3733175176</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*4-7G38oG5e7tYPIFVq2EPA.png" /><figcaption>Firehose Settings From AWS Console</figcaption></figure><p>If you have activated the Format Conversion process, Firehose will ask you to select a Glue schema and will convert the incoming data to the appropriate format using the schema you created before.</p><h4>Dynamic Partitioning</h4><p>Another structure that should be used carefully and one of the most important issues about Data Firehose is dynamic partitioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*NW6Vnzo2r6cbe1X0665hzQ.png" /><figcaption>Dynamic Partitioning with JQ Expressions</figcaption></figure><p>The data from a stream is written to the buffer memory in the Firehose and when this memory is full or expires, it is transferred to the final destination. Buffer size and interval can be set and their ranges are as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*evnykl_GwtNL0pb_4RN6NA.png" /><figcaption>Buffer Size and Interval</figcaption></figure><p>There cannot be more than 500 active partitions in this buffer. This value can be increased up to 5000 after contacting AWS Support. When this limit is exceeded, the operations with the data are stopped and the raw state S3 is written directly in the error destination.</p><p>For example, if the account_id is a partition key and there are more than 500 customer events in 60 seconds or 128MB of data, the system will not work properly. You have to pick your keys carefully.</p><p><em>NOTE: Starting with a prefix such as ‘data’ while writing data to S3 eliminates many problems to be experienced in the future. It allows separating the error content since Firehose need to store the content with error in the same S3 Bucket</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*Sn63Je9E3lkiATX6KeHkbg.png" /></figure><p>Another feature of Firehose is that it can store incoming data in a backup S3 Bucket before transforming or format conversion. Having a backup of your data in case something goes wrong can always save lives.</p><h3>Basic Architecture With Kinesis</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAoS1eXssQHWojHyAOsPmQ.jpeg" /></figure><p>You can briefly see the structure that forms the backbone of our new system above. Our main motivation here was to write the incoming data to an S3 Bucket in a secure and desired format without being stuck with limits. With Cloud and stream technology, we will have no loss from high-load incoming data. The system will be able to scale automatically, will not require any maintenance, and data will be stored in secure environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*6Mf8_PtUUBcBXRK6fPasqw.png" /><figcaption>S3 Partition Samples</figcaption></figure><p>In the following posts, we will explain how we repartition, query, and store this data and how the expired data is deleted. Follow us…</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e565b12cc793" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mobileaction-tech/developing-a-high-traffic-data-service-with-aws-kinesis-1-e565b12cc793">Developing a High-Traffic Data Service With AWS Kinesis</a> was originally published in <a href="https://medium.com/mobileaction-tech">MobileAction Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>