<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kristian Agung Nugroho on Medium]]></title>
        <description><![CDATA[Stories by Kristian Agung Nugroho on Medium]]></description>
        <link>https://medium.com/@xtian.agung.2020?source=rss-a355d39c6c5f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XSN8i1wCS_6xB8r62281vw.gif</url>
            <title>Stories by Kristian Agung Nugroho on Medium</title>
            <link>https://medium.com/@xtian.agung.2020?source=rss-a355d39c6c5f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:15:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@xtian.agung.2020/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analisis Protokol dan Alur Layer OSI pada Panggilan Suara (Voice Channel) di Discord]]></title>
            <link>https://medium.com/@xtian.agung.2020/analisis-protokol-dan-alur-layer-osi-pada-panggilan-suara-voice-channel-di-discord-f4ff7759b991?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4ff7759b991</guid>
            <category><![CDATA[discord]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Mon, 16 Nov 2026 06:20:26 GMT</pubDate>
            <atom:updated>2026-11-16T06:20:26.125Z</atom:updated>
            <content:encoded><![CDATA[<p>Oleh: Kristian Agung (2505060044) &amp; Euaggelion (2505060049)</p><p><em>Mata Kuliah Jaringan Komputer</em></p><h3>1. Studi Kasus</h3><p>Aplikasi yang dianalisis adalah Discord (platform komunikasi suara, video, dan teks), dengan fokus pada satu aktivitas spesifik yaitu proses komunikasi suara <em>real-time</em> saat pengguna bergabung dan berbicara di dalam <em>Voice Channel</em>. Aktivitas ini menarik dianalisis karena berbeda dengan lalu lintas web biasa, pengiriman data suara menuntut latensi serendah mungkin (<em>low latency</em>) sehingga mengorbankan pengiriman ulang paket (<em>retransmission</em>) demi kelancaran percakapan langsung.</p><h3>2. Service Name dan Transport Protocol Port Number</h3><p>Berdasarkan standar resmi <strong>IANA (Internet Assigned Numbers Authority)</strong> dan arsitektur jaringan Discord, komunikasi suara pada <em>Voice Channel</em> melibatkan spesifikasi berikut:</p><p><strong>Service Name (Layer 7)</strong></p><ul><li>rtp (<em>Real-time Transport Protocol</em>) yang dienkapsulasi di atas protokol suara khusus Discord (<em>Discord Voice Protocol</em>) untuk mengalirkan paket audio secara langsung.</li><li>https (<em>Hypertext Transfer Protocol Secure</em>) sebagai layanan pendukung untuk <em>signaling</em> awal (autentikasi masuk ke kanal suara dan pertukaran kunci enkripsi).</li></ul><p><strong>Transport Protocol (Layer 4)</strong></p><ul><li><strong>UDP (User Datagram Protocol)</strong> untuk aliran suara utama. Discord memilih UDP karena bersifat <em>connectionless</em> dan tanpa mekanisme pengiriman ulang (<em>no retransmission</em>). Dalam komunikasi suara langsung, keterlambatan akibat menunggu paket hilang dikirim ulang justru merusak kualitas percakapan dibandingkan mengabaikan paket yang hilang (<em>packet loss</em>).</li></ul><p><strong>Official Port Number (IANA)</strong></p><ul><li><strong>Rentang Port Dinamis / Terdaftar UDP (umumnya Port 50000–65535 UDP)</strong> untuk aliran suara <em>real-time</em> (dan <strong>Port 5004 UDP</strong> sebagai standar resmi IANA untuk layanan rtp).</li><li><strong>Port 443 (TCP)</strong> untuk layanan https (<em>control/signaling</em> awal).</li></ul><h3>3. Alur Kerja Tiap Layer OSI (Top-Down: Layer 7 ke Layer 1)</h3><p>Saat suara masuk ke mikrofon pengguna hingga diterima oleh server suara Discord dan didengar anggota kanal lain, terjadi alur enkapsulasi data dari Layer 7 ke Layer 1:</p><p>1) <strong>Layer 7 — Application Layer:</strong></p><p>Aplikasi Discord menangkap gelombang suara analog dari mikrofon, mengubahnya menjadi sinyal digital, dan membungkusnya ke dalam paket protokol <strong>RTP</strong> dengan <em>header</em> berisi <em>timestamp</em> serta nomor urut agar pemutaran suara di sisi penerima tetap sinkron.</p><p><strong>2) Layer 6 — Presentation Layer:</strong></p><p>Data suara dikompresi secara efisien menggunakan <em>codec</em> audio berlatensi rendah <strong>Opus</strong> (dengan bitrate adaptif, misalnya 64 kbps). Setelah dikompresi, <em>payload</em> suara dienkripsi menggunakan algoritma <strong>AEAD_XChaCha20_Poly1305</strong> (atau <strong>XSalsa20</strong>) agar percakapan tidak dapat disadap.</p><p><strong>3) Layer 5 — Session Layer:</strong></p><p>Sesi koneksi logis antara klien Discord dan <em>Discord Voice Server</em> dikelola secara aktif melalui paket <em>heartbeat</em> berkala (<em>keep-alive</em>) untuk memastikan pengguna masih terhubung di dalam <em>Voice Channel</em> serta mengatur negosiasi IP melalui protokol <strong>STUN/ICE</strong>.</p><p><strong>4) Layer 4 — Transport Layer:</strong></p><p>Paket suara terenkripsi dimasukkan ke dalam datagram <strong>UDP</strong>. Sistem operasi menambahkan <em>UDP Header</em> yang hanya berisi 4 komponen ringkas: <em>Source Port</em> lokal (misalnya 54821), <em>Destination Port</em> server suara Discord (misalnya 50012), <em>Length</em>, dan <em>Checksum</em>, tanpa melakukan proses <em>handshake</em>.</p><p><strong>5) Layer 3 — Network Layer:</strong></p><p>Datagram UDP dibungkus menjadi paket IP dengan menambahkan <em>IP Header</em> berisi alamat IP lokal perangkat pengirim dan alamat IP <em>Voice Server</em> Discord (misalnya region Singapura). Router meneruskan paket melalui jalur tercepat menuju server Discord.</p><p><strong>6) Layer 2 — Data Link Layer:</strong></p><p>Paket IP dienkapsulasi ke dalam <em>frame</em> Ethernet atau Wi-Fi (IEEE 802.11) dengan menyisipkan alamat fisik <strong>MAC Address</strong> antarmuka jaringan perangkat sebagai sumber dan MAC Address router lokal sebagai tujuan, dilengkapi <em>Frame Check Sequence</em> (FCS).</p><p><strong>7) Layer 1 — Physical Layer:</strong></p><p><em>Frame</em> biner (deretan bit 1 dan 0) dikonversi menjadi sinyal fisik berupa gelombang radio frekuensi (Wi-Fi / seluler) atau pulsa listrik/cahaya (kabel jaringan) yang ditransmisikan melalui media fisik menuju infrastruktur jaringan internet.</p><h3>4. Bukti Observasi Aplikasi dan Pengecekan Standar</h3><p>Validasi dilakukan dengan memverifikasi standar registri IANA dan memeriksa soket koneksi UDP aktif saat bergabung ke <em>Voice Channel</em> Discord:</p><p><strong>Pengecekan Registri IANA (<em>Service Name and Transport Protocol Port Number Registry</em>):</strong></p><ul><li>Pada laman resmi iana.org/assignments/service-names-port-numbers, tercatat bahwa <em>Service Name</em> <strong>rtp</strong> (<em>Real-time Transport Protocol</em>) dialokasikan pada <strong>Port Number 5004</strong> dengan transport protocol <strong>UDP</strong>, sementara komunikasi suara VoIP modern memanfaatkan port UDP dinamis setelah proses <em>signaling</em> pada port <strong>443 (TCP)</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*tvdT3cI-93yGaE9-.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*Sgp1s9YVuzkIPZca.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*VPLGoes5ZK2BD41O.png" /></figure><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*O-B13M5UoxqAidEs.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/0*dOLg4GXjkS-s6E8q.png" /></figure><p>Observasi Koneksi Aktif (netstat &amp; tasklist): Saat pengguna masuk ke dalam Voice Channel di Discord, pemeriksaan soket jaringan melalui Command Prompt (netstat -ano) yang dicocokkan dengan daftar proses (tasklist) menunjukkan dua jenis koneksi aktif pada proses DiscordPTB.exe (PID 13468 dan 5600):</p><ol><li>Koneksi UDP (Aliran Suara Real-Time): Pada perintah netstat -ano -p UDP, proses DiscordPTB.exe (PID 13468 dan 5600) secara aktif membuka soket protokol UDP pada port dinamis lokal (58485, 61197, dan 52408) untuk mengirimkan dan menerima aliran paket suara secara real-time.</li><li>Koneksi TCP Port 443 (Signaling &amp; Gateway): Pada perintah netstat -ano | findstr 443, proses Discord.exe dengan PID yang sama (13468) tercatat menjalin koneksi aktif berstatus ESTABLISHED ke server tujuan pada port 443 (seperti 162.159.134.234:443 dan 162.159.136.232:443) untuk pertukaran data signaling dan gateway.</li></ol><h3>Kesimpulan</h3><p>Berdasarkan hasil observasi tersebut, dapat disimpulkan bahwa aktivitas panggilan suara (Voice Channel) pada Discord bekerja dengan mengombinasikan dua protokol transportasi sekaligus agar komunikasi berlangsung efisien dan tetap andal. Layanan https di atas protokol TCP port 443 dimanfaatkan untuk proses signaling awal, autentikasi sesi, serta koneksi gateway, sementara protokol transportasi UDP pada port dinamis lokal digunakan untuk mentransmisikan aliran paket data suara berbasis RTP yang telah dikompresi dengan codec Opus. Pemisahan peran protokol ini terbukti selaras dengan arsitektur tujuh lapisan model referensi OSI dan standar registri IANA dalam menjamin komunikasi suara real-time yang minim latensi.</p><h3>Daftar Pustaka</h3><p>Internet Assigned Numbers Authority (IANA). (2026). Service Name and Transport Protocol Port Number Registry.</p><p>Discord Inc. (2026). Discord Voice Connections &amp; Network Port Requirements. Discord Developer &amp; Support Documentation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4ff7759b991" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigData(10): Real-Time Data Processing using PySpark Structured Streaming]]></title>
            <link>https://medium.com/@xtian.agung.2020/bigdata-10-real-time-data-processing-using-pyspark-structured-streaming-c64b4ffb89f5?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c64b4ffb89f5</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 13:41:04 GMT</pubDate>
            <atom:updated>2026-11-11T13:42:07.241Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><h4>What Is Batch Processing?</h4><p>Batch processing means we process data that is already complete and stored somewhere, like a CSV file. We usually run it on a schedule.</p><p>For example, a monthly sales report. We wait until the month ends, then we calculate everything at once.</p><h4>What Is Stream Processing?</h4><p>Stream processing means we process data while it is still coming in. We do not wait for the data to be complete. We handle it a little at a time, as soon as it arrives.</p><p>For example, a food delivery app that shows “orders per minute”. New orders arrive all the time, so the number must update all the time too.</p><h4>What Is Spark Structured Streaming?</h4><p>Spark Structured Streaming is the streaming tool inside Spark. We write the code with the same DataFrame API we already know, and Spark runs it as many tiny batches (called micro-batches) in the background.</p><p>Real streaming sources like Kafka need extra setup. So in this practicum, we <strong>simulate</strong> the stream with a folder. Every time a new CSV file appears in that folder, Spark reads it, just like new data flowing in.</p><h3>2. Hands-On</h3><h4>2.1. Preparing the SparkSession and the Stream Folder</h4><p>Run all of these in Jupyter Notebook. First, we import the libraries and create the Spark session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Af4MR-CNFVKKlyIfnQeJ8A.png" /></figure><ul><li>.master(&quot;local[*]&quot;) runs Spark on our own computer using all CPU cores.</li><li>setLogLevel(&quot;ERROR&quot;) hides unimportant log messages.</li><li>spark.sql.shuffle.partitions = 4 lowers the default 200 partitions. Our data is tiny, so 200 is too many and slow.</li></ul><p>Next, we create an empty folder. This folder acts as our “streaming source”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*rQP7hBuVZ7UrhxMqn-tXBA.png" /></figure><ul><li>shutil.rmtree(...) deletes old folders, so we always start clean.</li><li>os.makedirs(...) creates the new empty folder.</li></ul><h4>2.2. Defining the Schema</h4><p>In streaming, we must define the schema ourselves. Spark cannot use inferSchema=True, because the data has not fully arrived yet, so there is nothing complete to guess from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2Tii-rf1ZGhzaYi-B_BOw.png" /></figure><ul><li>StructType is the whole table structure.</li><li>StructField(name, type, nullable) describes one column.</li></ul><h4>2.3. Defining the Input Source (readStream)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*tCnBOArxaHhMT39wMjj7Ew.png" /></figure><ul><li>readStream replaces read. It keeps watching the folder for new files.</li><li>isStreaming is True, so this is a streaming DataFrame.</li><li>Nothing runs yet. Spark only makes a plan (lazy evaluation).</li></ul><h4>2.4. Defining the Transformation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*oj2FOkIjdR0qeCoZpz0Okw.png" /></figure><ul><li>The syntax is exactly the same as a normal batch groupBy and agg.</li><li>This gives the total revenue and the number of transactions per city.</li></ul><h4>2.5. Starting the Streaming Query (writeStream)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g0TZWk44TBRFWs19eYwnHA.png" /></figure><ul><li>outputMode(&quot;complete&quot;) shows the <strong>whole</strong> latest result every time new data arrives.</li><li>format(&quot;memory&quot;) saves the result as an in-memory table. It is good for learning and debugging.</li><li>queryName(&quot;hasil_agregasi&quot;) is the table name we use later in spark.sql().</li><li>.start() begins the query. It keeps running until we stop it.</li></ul><h4>2.6. Sending New Data</h4><p>Now we simulate data that “flows in” by writing new CSV files into the folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*3Ep7N_IN1QV0Sk7TqC0v_Q.png" /></figure><ul><li>.to_csv(...) writes a new file into the watched folder. This is our “new data”.</li><li>time.sleep(6) gives Spark time to process. If the table is empty, wait a few more seconds and run the spark.sql line again. It is not an error.</li></ul><p>Now the second batch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*8K3sDtv2LzZVv0AwP_EbIQ.png" /></figure><p>Look at <strong>Magelang</strong>. The total grew from 225,000 to 525,000 (225,000 + 300,000), and the count grew from 2 to 3. We did not change any code. Spark keeps the <strong>state</strong> (the running total) and adds the new data to it. This is the main power of Structured Streaming.</p><h4>2.7. Stopping the Query</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKPEVSXKC-3KJzbzMAa3Dg.png" /></figure><ul><li>A streaming query never stops by itself, so we must call stop().</li></ul><h4>2.8. Window-Based Aggregation</h4><p>Besides grouping by a normal column, we can group by a <strong>time window</strong>. For example: “how much is the total every 1 minute?”. This is different from the window function (rank, row_number) in Meeting 5. Here, the window means a time range. We need a timestamp column, so Spark knows when each record happened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*KFxwjuqhxxRf9opAWrxVpg.png" /></figure><ul><li>TimestampType() is the data type for date and time.</li><li>window(col(&quot;waktu_transaksi&quot;), &quot;1 minute&quot;) puts every record into a 1-minute time range.</li></ul><p>Now we send data with the current time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*aa8Rug74AM8IjHK5MFtJnw.png" /></figure><p>Both transactions happened in the same minute, so they are grouped together: 120,000 + 180,000 = 300,000. The time values will be different on your computer. This idea is behind real-time dashboards like “orders per minute”.</p><h4>2.9. Stopping Session</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*OdVh-cYKLaSJOXkHBtoUCw.png" /></figure><h3>3. Practice</h3><p>First, we create a new session and a new folder for the exercises.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*TWJ4xTwkRnOMcrCiNGNquA.png" /></figure><h4>3.1. Question 1: Average Revenue per Category</h4><p>Create a readStream from latihan10_stream, group by kategori, and calculate the average pendapatan using avg. Name the query latihan_agg.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lub8iw0KoAs51IW3WgUNiQ.png" /></figure><ul><li>It is the same as the hands-on, but we use avg instead of sum.</li></ul><h4>3.2. Question 2: Send the First Batch</h4><p>Send a CSV file with at least 3 rows, wait a few seconds, then show the table latihan_agg.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sqCsfZDWFFV_aRYcXsoyqw.png" /></figure><ul><li>Elektronik is (100,000 + 300,000) / 2 = 200,000, so the result is correct.</li></ul><h4>3.3. Question 3: Send the Second Batch</h4><p>Send a second batch with a category that already exists, then show the table again. Does the average change correctly?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Asdu8XwEJYYtkerfgkOkw.png" /></figure><p>Yes, it changes correctly, and it uses <strong>both batches together</strong>:</p><ul><li>Elektronik: (100,000 + 300,000 + 500,000) / 3 = <strong>300,000</strong></li><li>Makanan: (50,000 + 70,000) / 2 = <strong>60,000</strong></li><li>Fashion has no new data, so it stays at 250,000.</li></ul><h4>3.4. Question 4: Reflection</h4><p>Why does Structured Streaming require an explicit schema, while batch processing can use inferSchema=True?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ckz79F_zhgXJs6aou36mTQ.png" /></figure><h3>4. Assignment</h3><p><strong>Scenario:</strong> “Kopi Nusantara” wants a <strong>real-time order monitoring dashboard</strong> to see how each branch performs at any time. We build a prototype streaming pipeline for it.</p><h4>4.1. Preparing the SparkSession</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6pw_F827z5O58H2F0Z3GA.png" /></figure><ul><li>We create a new session for the assignment and set the shuffle partitions to 4.</li></ul><h4>4.2. A: Prepare the Streaming Source</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*frDlLqoaY64K3ZblKwt-DQ.png" /></figure><ul><li>We define the schema with all 6 columns from the task.</li><li>tugas10_stream is the data source folder. We also delete old output and checkpoint folders to start clean.</li></ul><h4>4.3. B: Aggregation per Branch</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*C0r3LpJ6q23wovMlnffsHQ.png" /></figure><ul><li>withColumn(&quot;total_harga&quot;, ...) adds a new column: quantity times unit price.</li><li>We group by branch, then calculate the total price and the number of orders.</li></ul><h4>4.4. C: Aggregation with a 2-Minute Time Window</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*OAIxDObQca8oYVjAqRySyw.png" /></figure><ul><li>This is a <strong>second</strong> query that reads from the same folder. One data source can be read by more than one query.</li><li>window(col(&quot;waktu_pesan&quot;), &quot;2 minutes&quot;) groups orders into 2-minute time ranges.</li></ul><h4>4.5. D: Test with Simulated Data</h4><p>We send 3 batches with different branch combinations. The helper function kirim (“send”) keeps the code short.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*8yov0M9-yRvm3T0oEWTigw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*6b3qmkUo_NzZ-02OriBOCQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*j1xAOzyelyOSrsz87DrD_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*g3QXH6NGFDyZ4uSRjuAKqw.png" /></figure><p>Time window result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*RhMQY1lg_o7leIR0933eJA.png" /></figure><ul><li>Proof that the totals accumulate. For Magelang: batch 1 is (2 × 18,000) = 36,000 with 1 order. Batch 2 adds (2 × 22,000) + (1 × 20,000), so 100,000 with 3 orders. Batch 3 adds (2 × 25,000), so 150,000 with 4 orders.</li><li>Time window. All transactions across the 3 batches sum up to 36,000 + 20,000 + 75,000 + 44,000 + 72,000 + 20,000 + 50,000 + 90,000 + 22,000 + 50,000 = 479,000. Divided across the 2-minute time windows, the total is 212,000 + 136,000 + 131,000 = 479,000. It matches.</li></ul><h4>4.6. E: File-Based Sink</h4><p>Now we write the <strong>raw data</strong> (no aggregation) to Parquet files and read the Parquet files back as a normal batch DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*UNaAB980gW1KGQo41D-5Iw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*p2tU75y3DJdcG1l-2rNs6A.png" /></figure><ul><li>format(&quot;parquet&quot;) saves the data as Parquet files.</li><li>path is the output folder. checkpointLocation is where Spark stores its progress.</li><li>outputMode(&quot;append&quot;) only adds new rows. This is the right mode when there is no aggregation.</li><li>The Parquet files are saved and can be read back. The result has <strong>all 9 orders</strong>, not only batch 4. This is because the new query starts from the beginning and reads every file already in the folder.</li></ul><p>Finally, we stop everything.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*9hqtgs95-7MzlfMhiQLSvg.png" /></figure><h3>5. Conclusion</h3><p>In this practicum, we learned that Spark Structured Streaming lets us process data that keeps arriving, using the same DataFrame API we already knew from batch processing. We simulated a data stream with a folder, built a streaming query with readStream and writeStream, and saw that Spark keeps the running state, so totals and averages update correctly when new batches arrive. The main challenge was that streaming needs an explicit schema, because Spark cannot guess the column types from data that has not arrived yet. We also used a time window to group orders by 2-minute ranges, and wrote raw data to Parquet files with a file sink. Finally, we learned that a streaming query never stops by itself, so we must always call stop().</p><h3>6. Reference</h3><p>Modul Praktikum Big Data. (2026). <em>Pertemuan 10: Pemrosesan Data Streaming dengan Spark Structured Streaming</em>. Program Studi Teknologi Informasi, Universitas Tidar.</p><h3>7. GitHub Link</h3><ul><li><a href="https://github.com/KristianAgungN/Big_Data/blob/main/Pertemuan10/Modul_Praktikum_BigData_Pertemuan10%20(1).ipynb">Big_Data/Pertemuan10/Modul_Praktikum_BigData_Pertemuan10 (1).ipynb at main · KristianAgungN/Big_Data</a></li><li><a href="https://github.com/KristianAgungN/Big_Data/blob/main/Pertemuan10/Tugas10_2505060044_Krisitian_Agung_Nugroho.ipynb">Big_Data/Pertemuan10/Tugas10_2505060044_Krisitian_Agung_Nugroho.ipynb at main · KristianAgungN/Big_Data</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c64b4ffb89f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jaringan Komputer: Pengenalan MikroTik]]></title>
            <link>https://medium.com/@xtian.agung.2020/jaringan-komputer-pengenalan-mikrotik-e0f62dcd3965?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0f62dcd3965</guid>
            <category><![CDATA[computer-network]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 13:01:00 GMT</pubDate>
            <atom:updated>2026-11-09T13:01:00.137Z</atom:updated>
            <content:encoded><![CDATA[<h3>Latar Belakang</h3><p>Dalam perkembangan teknologi informasi, router merupakan perangkat utama yang bertugas mengelola lalu lintas data dan menghubungkan berbagai segmen jaringan. MikroTik RouterOS yang dijalankan pada perangkat keras RouterBoard menjadi salah satu solusi efisien dalam manajemen jaringan lokal (LAN) maupun penyedia akses internet.</p><p>Untuk mempermudah proses konfigurasi tanpa harus bergantung pada antarmuka teks (CLI), MikroTik menyediakan aplikasi berbasis Graphical User Interface (GUI) yaitu Winbox. Memahami konfigurasi dasar MikroTik mulai dari koneksi awal, pemberian identitas, manajemen hak akses pengguna, hingga pengalamatan IP merupakan langkah awal yang krusial sebelum membangun infrastruktur jaringan yang lebih kompleks.</p><h3>Tujuan Praktikum</h3><ol><li>Memahami fungsi dasar router MikroTik RB951 dan penggunaan aplikasi Winbox.</li><li>Menghubungkan perangkat PC/laptop ke MikroTik dan melakukan konfigurasi identitas serta akun pengguna.</li><li>Mengatur IP Address pada interface MikroTik dan PC/laptop dalam satu subnet.</li><li>Melakukan uji koneksi jaringan antara PC/laptop dan MikroTik menggunakan perintah</li></ol><h3>Alat dan Bahan</h3><ol><li><strong>Router MikroTik RB951</strong> beserta <strong>Power Adapter</strong>.</li><li><strong>PC </strong>dengan port Ethernet (atau adapter LAN to USB).</li><li><strong>Kabel UTP</strong> jenis <em>Straight-Through</em>.</li><li><strong>Aplikasi Winbox</strong>.</li></ol><h3>Pengertian MikroTik</h3><p>MikroTik adalah sistem operasi berbasis Linux yang dikembangkan oleh MikroTikls SIA di Latvia untuk keperluan manajemen router jaringan. Produk MikroTik terbagi menjadi dua komponen utama:</p><ul><li><strong>RouterOS</strong>: Sistem operasi yang memiliki fitur routing, firewall, bandwidth management, DHCP, VPN, dan lainnya.</li><li><strong>RouterBoard (RB)</strong>: Perangkat keras yang dirancang khusus untuk menjalankan sistem operasi RouterOS.</li></ul><h3>Perangkat MikroTik RB951</h3><p>MikroTik RB951 merupakan salah satu seri RouterBoard yang umum digunakan untuk skala jaringan kecil hingga menengah atau laboratorium praktikum. Perangkat ini dilengkapi dengan 5 port Ethernet (Ether1–Ether5), interface wireless 2.4 GHz, serta CPU dan RAM internal.</p><h3>Aplikasi Winbox</h3><p>Winbox adalah aplikasi berbasis Windows GUI yang digunakan untuk mengonfigurasi dan memantau perangkat MikroTik melalui jaringan lokal. Winbox memungkinkan administrator terhubung ke router menggunakan <strong>MAC Address</strong> (jika router belum memiliki IP) maupun <strong>IP Address</strong>.</p><h3>Konsep IP Address dan Uji Koneksi (Ping)</h3><ul><li><strong>IP Address</strong>: Alamat numerik unik yang diidentifikasikan pada perangkat jaringan agar dapat saling berkomunikasi. Dalam satu segmen jaringan lokal (subnet), IP Address tiap perangkat harus berada dalam rentang prefix yang sama (misal /24).</li><li><strong>Ping (Packet Internet Groper)</strong>: Perintah dasar utilitas jaringan yang menggunakan protokol ICMP untuk menguji apakah suatu host/perangkat tujuan dapat dijangkau dan merespons paket data yang dikirimkan.</li></ul><h3>Langkah Praktikum</h3><h3>Langkah 1: Persiapan Fisik &amp; Koneksi Perangkat</h3><ol><li>Hubungkan adaptor daya ke router Mikrotik RB951 hingga lampu indikator LED menyala.</li></ol><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*J3pRFIkbgbal0oD2.jpeg" /></figure><p>2. Ambil kabel UTP Straight-Through, hubungkan salah satu ujungnya ke port <strong>Ether1</strong> (atau port LAN lain) pada Mikrotik, dan ujung lainnya ke <strong>Port LAN PC</strong>.</p><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*bJ_HGxX-Zgw5l3XK.jpeg" /></figure><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*gLa3WIBCLiwa0QeV.jpeg" /></figure><p>Langkah ini bertujuan untuk membangun jalur koneksi fisik Layer 1 (Physical Layer) antara komputer pengelola (admin) dan router Mikrotik agar sinyal data dapat mentransmisikan paket untuk proses konfigurasi. Kabel <em>Straight-Through</em> digunakan karena kita menghubungkan dua perangkat jaringan yang berbeda kategori.</p><h3>Langkah 2: Menjalankan Winbox &amp; Koneksi Awal (MAC-Address Login)</h3><ol><li>Buka aplikasi <strong>Winbox.exe</strong> pada laptop/PC.</li><li>Masuk ke tab <strong>Neighbors</strong> dan klik <strong>Refresh</strong>. Winbox akan mendeteksi Mikrotik yang terhubung secara otomatis.</li><li>Klik pada baris <strong>MAC Address</strong> router yang muncul di daftar.</li><li>Isi kolom Login: dengan <strong>admin</strong> dan kosongkan kolom Password:</li><li>Klik tombol <strong>Connect</strong>.</li></ol><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*pYNnZ5jU2NqK9f-q.jpeg" /></figure><p>Koneksi melalui MAC Address dilakukan karena router Mikrotik baru/reset belum tentu memiliki alokasi IP Address yang se-subnet dengan komputer. Winbox menggunakan protokol MNDP (MikroTik Neighbor Discovery Protocol) untuk menemukan perangkat di jaringan lokal.</p><h3>Langkah 3: Eksplorasi Antarmuka Winbox</h3><ul><li>Amati bagian <strong>Identitas Router</strong> pada bilah judul di bagian atas.</li><li>Buka menu <strong>Interfaces</strong> di bilah menu sebelah kiri untuk melihat daftar port Ethernet (ether1–ether5) dan antarmuka nirkabel (wlan1).</li></ul><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*m2cot334aonfUKs9.jpeg" /></figure><p>Jendela <em>Interface List</em> menampilkan status seluruh port fisik yang ada pada MikroTik RB951 (ether1 sampai ether5 dan wlan1). Huruf R (<em>Running</em>) di samping nama port menandakan bahwa port tersebut terhubung fisik dan aktif mengirim/menerima data.</p><ul><li>Klik menu <strong>Quick Set</strong> untuk melihat gambaran umum konfigurasi mode router.</li></ul><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*RAqoxfDflZLrwRG6.jpeg" /></figure><p>Di bagian background ditampilkan menu <em>Quick Set</em> yang menyajikan ringkasan konfigurasi dasar jaringan (seperti mode router/bridge, IP address, DHCP, dan status wireless).</p><p>Langkah ini untuk memastikan bahwa semua interface fisik maupun virtual pada RB951 terdeteksi dalam kondisi aktif (R = Running) dan siap dikonfigurasi.</p><h3>Langkah 4: Pengaturan Identitas Router (System Identity)</h3><ul><li>Pilih menu <strong>System</strong> -&gt; <strong>Identity</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/0*Cb-j_m9wMtKOk5sd.jpeg" /></figure><ul><li>Di kotak teks <em>Identity</em>, ubah nama default (MikroTik) menjadi nama identitas baru (contoh: Router_KelompokSigmaBoy).</li><li>Klik <strong>Apply</strong> lalu <strong>OK</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/0*W6AsAY2J7B22kXx4.jpeg" /></figure><p>Mengubah <em>System Identity</em> bertujuan memberikan penamaan/penanda unik pada router. Hal ini memudahkan pembedaan perangkat dalam jaringan logis yang memiliki banyak router.</p><h3>Langkah 5: Pengaturan Manajemen User Login (System Users)</h3><ol><li>Buka menu <strong>System</strong> -&gt; <strong>Users</strong>.</li><li>Pada tab <em>Users</em>, klik tombol <strong>+ (Add)</strong>.</li><li>Isikan parameter berikut:</li></ol><ul><li><strong>Name:</strong> admin123</li><li><strong>Group:</strong> full (memberikan hak akses penuh)</li><li><strong>Password:</strong> praktikum123</li><li><strong>Confirm Password:</strong> password123</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*lmiNK8q_Etvid9KR.jpeg" /></figure><p>4. Klik <strong>Apply</strong> -&gt; <strong>OK</strong>.</p><p>5. Logout dari Winbox (<strong>Session</strong> -&gt; <strong>Exit</strong>), lalu Login kembali menggunakan username admin123 dan password password123.</p><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*Jr5bGIDk1Wcr5fyy.jpeg" /></figure><p>Penambahan user administrator baru dengan sandi yang kuat meningkatkan keamanan router dari akses tanpa izin. Menghapus/menggantikan akun admin default adalah praktik <em>best practice</em> keamanan jaringan.</p><h3>Langkah 6: Konfigurasi IP Address Manual pada Interface Mikrotik</h3><ol><li>Pilih menu <strong>IP</strong> -&gt; <strong>Addresses</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/0*CDwCvMyzBzovKu2F.jpeg" /></figure><p>2. Klik tombol <strong>+ (Add)</strong>.</p><p>3. Pada jendela <em>New Address</em>, isi konfigurasi sebagai berikut:</p><ul><li><strong>Address:</strong> 192.168.10.1/24</li><li><strong>Interface:</strong> ether1</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/0*YOmgiMgWmc83ad1E.jpeg" /></figure><p>4. Klik <strong>Apply</strong> (kolom <em>Network</em> akan terisi otomatis 192.168.10.0) lalu klik <strong>OK</strong>.</p><p>Menambahkan IP Address pada interface ether1 menetapkan alamat Layer 3 (Network Layer) bagi router. Notasi /24 (Subnet Mask 255.255.255.0) menentukan bahwa rentang IP yang se-subnet adalah 192.168.10.1 hingga 192.168.10.254.</p><h3>Langkah 7: Pengaturan IP Address pada PC/Laptop Client</h3><ol><li>Di komputer (Windows), buka <strong>Settings</strong> / <strong>Control Panel</strong> -&gt; <strong>Network Connections</strong>.</li><li>Pilih adapter Ethernet yang terhubung ke Mikrotik, lalu atur IP v4 secara manual:</li></ol><ul><li><strong>IP Address:</strong> 192.168.10.2</li><li><strong>Subnet Mask:</strong> 255.255.255.0</li><li><strong>Default Gateway:</strong> 192.168.10.1</li></ul><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*5MRFVNZHhvkMHZMV.jpeg" /></figure><p>3. Klik <strong>OK</strong>.</p><p>Agar PC dan Router dapat saling berkomunikasi via protokol IP, keduanya wajib berada dalam satu <em>network ID</em> yang sama (192.168.10.0/24).</p><h3>Langkah 8: Pengujian Konektivitas (Ping Test)</h3><ol><li>Buka <strong>Command Prompt (CMD)</strong> pada Windows/PC.</li><li>Jalankan perintah tes ping ke alamat IP Mikrotik:</li></ol><pre>ping 192.168.10.1</pre><p>Press enter or click to view image in full size</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*8DvytEGom4ggEMT2.jpeg" /></figure><p>Amati hasilnya: Jika keluar respon Reply from 192.168.10.1: bytes=32 time&lt;1ms TTL=64, maka koneksi antara PC dan Mikrotik telah berhasil dan stabil.</p><h3>Kesimpulan</h3><p>Praktikum ini memperlihatkan bahwa konfigurasi dan pengoperasian perangkat Router Mikrotik RB951 dapat dilakukan secara efisien melalui antarmuka grafis Winbox. Proses koneksi awal menggunakan MAC Address mempermudah akses ke router sebelum pengalamatan IP dikonfigurasi. Setelah pengalokasian IP Address pada antarmuka ether1 (192.168.10.1/24) diselaraskan dengan IP PC client (192.168.10.2), komunikasi logis Layer 3 berhasil dibangun dengan baik dan teruji melalui perintah ping. Selain itu, penetapan identitas router (<em>System Identity</em>) serta pembuatan akun manajemen baru (<em>System Users</em>) menjadi langkah krusial untuk menjaga keamanan dan kerapian administrasi jaringan. Secara keseluruhan, pemahaman konfigurasi dasar ini menjadi fondasi utama untuk menerapkan fungsi jaringan yang lebih kompleks pada modul-modul berikutnya.</p><h3>Referensi</h3><p>Hidayat, M. M. (2025). <em>Modul Praktikum Jaringan Komputer (Based on MikroTik MTCNA)</em>. Magelang: Program Studi Teknologi Informasi, Fakultas Teknik, Universitas Tidar.</p><h3>Anggota Kelompok</h3><p>Kristian Agung (2505060044)</p><p>Euaggelion Purnomo (2505060049)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0f62dcd3965" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PBO(10): Learning Interface with Java]]></title>
            <link>https://medium.com/@xtian.agung.2020/pbo-10-learning-interface-with-java-ee2fa5304511?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee2fa5304511</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[java-programming]]></category>
            <category><![CDATA[netbeans]]></category>
            <category><![CDATA[objectorientedprogramming]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 15:06:50 GMT</pubDate>
            <atom:updated>2026-11-04T15:06:50.333Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><h4>What Is an Interface?</h4><p>An interface is a concept in OOP that works like a contract. It tells a class what it must be able to do, but not how to do it. This makes the code more consistent and easier to extend.</p><p>For example, every TV remote has a power button and a volume button. The buttons are the same, but the inside of each remote is different for each brand. The buttons are the contract. The inside is the implementation.</p><h4>What Does an Interface Contain?</h4><p>An interface only contains method declarations (without a body) and constants.</p><ul><li>All methods in an interface are public and abstract by default.</li><li>All variables in an interface are public, static, and final by default.</li><li>An interface cannot be used to create an object directly.</li></ul><h4>How Do We Use an Interface?</h4><p>A class uses the keyword implements to follow an interface. The class must write the code for all methods from the interface. If not, the class must be declared abstract.</p><pre>public class Student extends Person implements Learnable, Payable {<br>    // ...<br>}</pre><ul><li>A class can implement more than one interface, separated by a comma.</li><li>If a class uses both, write extends first, then implements.</li><li>An interface can inherit from another interface using extends.</li></ul><p>Interface vs Abstract Class</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*JLVSmRJjRT1Gk3bChxQJgw.png" /></figure><h3>2. Hands-On</h3><h4>2.1. Creating the OperasiHitung Interface</h4><p>First, we create an interface called OperasiHitung. This interface contains one method named hitung().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*PQ_tU8afEauoMlKJBIuhBw.png" /></figure><ul><li>public interface OperasiHitung defines an interface named OperasiHitung.</li><li>int hitung(int a, int b); declares a method without a body. Classes that implement this interface must write the code for it.</li><li>int is the return type, and a and b are the two numbers we want to calculate.</li></ul><p>The OperasiHitung interface acts as a contract for all calculation classes.</p><h4>2.2. Creating the Penjumlahan Class</h4><p>Next, we create a class called Penjumlahan. This class implements the OperasiHitung interface and does an addition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*7RotZKgjVjlKELaWAghCFg.png" /></figure><ul><li>implements OperasiHitung means this class follows the contract from the interface.</li><li>@Override tells Java that this method comes from the interface.</li><li>return a + b; returns the sum of a and b.</li></ul><p>The Penjumlahan class fills the contract with addition logic.</p><h4>2.3. Creating the Pengurangan Class</h4><p>Then, we create a class called Pengurangan. It also implements OperasiHitung, but it does a subtraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*-oE42idg29cfj-HvLZc_iQ.png" /></figure><ul><li>implements OperasiHitung means this class follows the same contract.</li><li>public int hitung(int a, int b) has the same method name and parameters as the interface.</li><li>return a - b; returns the result of a minus b.</li></ul><p>The Pengurangan class uses the same method name as Penjumlahan, but the logic is different.</p><h4>2.4. Creating the Main Class</h4><p>Finally, we create the Main class to test both classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*KM64LqOhOhHo8OO9Jgczbg.png" /></figure><ul><li>OperasiHitung penjumlahan = new Penjumlahan(); creates a Penjumlahan object and stores it in an interface-type variable.</li><li>penjumlahan.hitung(10, 5) calls the addition logic, so the result is 15.</li><li>OperasiHitung pengurangan = new Pengurangan(); creates a Pengurangan object in the same way.</li><li>pengurangan.hitung(10, 5) calls the subtraction logic, so the result is 5.</li><li>System.out.println() displays the result in the output.</li></ul><h3>3. Program Output</h3><p>After running Main.java, the program displays the following output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*O-mrz9RxX-WRNKZN3QpUgA.png" /></figure><p>The output shows that Penjumlahan and Pengurangan have different implementations of hitung(). However, both classes are used in the same way through the OperasiHitung interface.</p><h3>4. Assignment</h3><p>Task: Build a payment program for a shop that sells Electronics and Food. Electronics have 10% tax, and Food has 5% tax.</p><p>All classes are in the same package as the hands-on, praktikum10. Because Main already exists from section 2, the main class for this assignment is named MainBayar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*hLacjrZ6tF0NjeRzOe0sLg.png" /></figure><h4>4.1. Creating the Pembayaran Interface</h4><p>First, we create an interface called Pembayaran. It contains a method named hitungPajak().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*FeVOS5qRmo_0NbXB8dqnsA.png" /></figure><ul><li>public interface Pembayaran defines an interface named Pembayaran.</li><li>double hitungPajak(double harga); declares a method without a body. It receives a price and returns the tax.</li><li>double is used because prices and taxes can have decimal values.</li></ul><p>The Pembayaran interface acts as a basic contract for all product classes.</p><h4>4.2. Creating the Elektronik Class</h4><p>Next, we create the Elektronik class. It implements Pembayaran and calculates a 10% tax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*lgPnngDPMndqs6bzTEdCFg.png" /></figure><ul><li>implements Pembayaran means this class must write the hitungPajak() method.</li><li>@Override marks the method that comes from the interface.</li><li>harga * 0.10 calculates 10% of the price.</li></ul><p>The Elektronik class defines the tax rule for electronic products.</p><h4>4.3. Creating the Makanan Class</h4><p>Then, we create the Makanan class. It implements Pembayaran and calculates a 5% tax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*Gr_wxZQdowNB1CZQZNGNjQ.png" /></figure><ul><li>implements Pembayaran means this class follows the same contract as Elektronik.</li><li>harga * 0.05 calculates 5% of the price.</li></ul><p>The Makanan class defines the tax rule for food products.</p><h4>4.4. Creating the MainBayar Class</h4><p>Finally, we create the MainBayar class to test both product classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*z7TwR7IJZBNpaLdQK0l1UA.png" /></figure><ul><li>public class MainBayar is the main class for the assignment. It has a different name from Main so both can live in the same package.</li><li>Pembayaran elektronik = new Elektronik(); creates an Elektronik object using the interface type.</li><li>elektronik.hitungPajak(100000) calculates the tax for a price of 100000. 10% of it is 10000.0.</li><li>Pembayaran makanan = new Makanan(); creates a Makanan object using the same interface type.</li><li>makanan.hitungPajak(50000) calculates the tax for a price of 50000. 5% of it is 2500.0.</li><li>System.out.println() displays each result in the output.</li></ul><p>The MainBayar class shows that two different products can be used through the same Pembayaran interface.</p><h4>4.5. Assignment Output</h4><p>After running MainBayar.java, the program displays the following output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*qZRUIhLLMJYANZbYxk4w6A.png" /></figure><p>The output shows that Elektronik and Makanan follow the same contract but calculate tax in different ways: 10% for electronics and 5% for food.</p><h3>5. Insights</h3><ul><li>An interface separates “what” from “how.” Pembayaran only says that a product must be able to calculate tax. Each class decides the rate by itself.</li><li>An interface type can hold any class that implements it. That is why we can write Pembayaran makanan = new Makanan();. The same code can work with many different classes.</li><li>It is easy to add new features. If the shop sells clothes later, we only create a new class that implements Pembayaran. The existing classes do not need to change.</li><li>The compiler protects the contract. If a class says implements but forgets one method, Java gives an error. This helps us avoid missing features.</li><li>Interface or abstract class? Use an interface when different classes need the same ability. Use an abstract class when the classes are closely related and share some code or data.</li></ul><h3><strong>6. Conclusion</strong></h3><p>In this module, we learned that an interface is a contract that tells a class what to do, while the class decides how to do it. With implements, different classes can share the same method name but have different logic, like Penjumlahan and Pengurangan, or Elektronik and Makanan. A class can also implement more than one interface, which normal class inheritance cannot do. This makes our programs more consistent, flexible, and easy to extend.</p><h3>7. Reference</h3><p>Modul 10: Interface. Magelang: Program Studi Teknologi Informasi, Universitas Tidar.</p><h3>8. GitHub Link</h3><p><a href="https://github.com/KristianAgungN/Pemrograman-Berorientasi-Objek/tree/main/Pertemuan10/PraktikumPBO_10">Pemrograman-Berorientasi-Objek/Pertemuan10/PraktikumPBO_10 at main · KristianAgungN/Pemrograman-Berorientasi-Objek</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee2fa5304511" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BigData(9): Learning NoSQL using MongoDB and PySpark]]></title>
            <link>https://medium.com/@xtian.agung.2020/bigdata-9-learning-nosql-using-mongodb-and-pyspark-a3aaf943a836?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3aaf943a836</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 07:48:53 GMT</pubDate>
            <atom:updated>2026-11-04T07:48:53.668Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><h4>What Is NoSQL?</h4><p>NoSQL is a type of database that doesn’t force data into fixed tables. Each record can have its own fields, so it works well for data that looks different from one record to another.</p><p>For example, a shirt has a “size” and a laptop has a “voltage”. In a normal SQL table, we would need columns for both, and most of them would stay empty.</p><h4>What Is MongoDB?</h4><p>MongoDB is a popular NoSQL database. It stores data as documents (like JSON objects) inside collections (like tables).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*zqP8WAAIeTPSgsc-E9ukSw.png" /></figure><p>MongoDB doesn’t replace SQL or Spark. In real projects, they are used together.</p><h4>What Is PyMongo?</h4><p>PyMongo is the official Python driver for MongoDB. It lets us talk to MongoDB from Python code, so we can combine it with pandas and PySpark.</p><h3>2. Hands-On</h3><h4>2.1. Installing MongoDB</h4><p>Run all of these in the Ubuntu terminal. First, we import MongoDB’s GPG key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*mAZB7Li5nW98xAy4_SHzMg.png" /></figure><ul><li>curl -fsSL downloads the key quietly and follows redirects.</li><li>gpg --dearmor converts the key into a format that apt understands.</li><li>We need this key because apt only trusts signed packages.</li></ul><p>If the command say error “curl does’nt exist” you can install curl using this command below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*kKvMSNMXg1pdL4HJ8Jbj0w.png" /></figure><p>Next, we add the MongoDB repository. Choose <strong>only one</strong> based on your Ubuntu version.</p><p>Ubuntu 22.04 (Jammy):</p><pre>echo &quot;deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/8.0 multiverse&quot; | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list</pre><p>Ubuntu 24.04 (Noble):</p><pre>echo &quot;deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-8.0.gpg ] https://repo.mongodb.org/apt/ubuntu noble/mongodb-org/8.0 multiverse&quot; | sudo tee /etc/apt/sources.list.d/mongodb-org-8.0.list</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*ODgtehFrm4MSfm-Ttpn8ug.png" /></figure><ul><li>signed-by=... links this repository to the key we just imported.</li><li>sudo tee writes the line into /etc/apt/sources.list.d/. We use tee because a plain &gt; can’t write there with sudo.</li><li>MongoDB is not in Ubuntu’s default repository, so we add the official one.</li></ul><p>Then we update the package list and install MongoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*eiZBQw4r0dPRvnfjaja3Nw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*zPiClSrGh7yRmBc_zXUU7w.png" /></figure><ul><li>apt update refreshes the package list so it can see the new repository.</li><li>apt install -y mongodb-org installs MongoDB. -y answers “yes” automatically.</li></ul><p>Finally, we start MongoDB and check that it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*vLCpeXyQIx4VRorqXpbr-Q.png" /><figcaption>systemctl start mongod runs MongoDB right now.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*QEIN4VAW-5ZvN408rhPRtQ.png" /><figcaption>systemctl enable mongod makes MongoDB start automatically on every boot.</figcaption></figure><p>systemctl status mongod shows whether it’s running. Look for Active: active (running), then press q to exit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*3Lf2FAYdcIshvE4_GKTMdw.png" /><figcaption>failed…</figcaption></figure><p>2.1.1. Troubleshooting: mongod Failed to Start</p><p>When we check the status, we may see Active: failed (Result: exit-code) in red instead of active (running). That means MongoDB started but crashed right away. If this happens, try the steps below.</p><p>First, we read the log to find out why it crashed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*_PjFNCDmPmy88Eak4mgGxQ.png" /></figure><ul><li>journalctl -u mongod shows the logs of the mongod service.</li><li>--no-pager prints everything directly, so we don’t have to scroll or press q.</li><li>-n 20 shows only the last 20 lines.</li></ul><p>In my case, the log said:</p><p>“MongoDB cannot start: Linux kernel versions 6.19 and newer has a known incompatibility with this version of MongoDB. See <a href="https://jira.mongodb.org/browse/SERVER-121912">https://jira.mongodb.org/browse/SERVER-121912</a> for more information.”</p><p>So the problem is not the installation. It’s a known incompatibility between this MongoDB version and the newer Linux kernel.</p><p>To fix it, we create a systemd override file. This lets us change how the service starts without editing the original service file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*N5WyHKjDVLcrdhrEsAEEZg.png" /></figure><ul><li>mkdir -p creates the override folder. -p means no error if it already exists.</li><li>nano .../override.conf opens a new file in the text editor. systemd reads every .conf file in this folder and applies it on top of the original service.</li></ul><p>Inside the file, write:</p><pre>[Service]<br>Environment=&quot;GLIBC_TUNABLES=glibc.pthread.rseq=1&quot;<br>ExecStart=<br>ExecStart=/usr/bin/mongod --config /etc/mongod.conf</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*qSVt3KxM8gl04LBSwsGrCQ.png" /></figure><ul><li>[Service] is the section we want to override.</li><li>Environment=&quot;GLIBC_TUNABLES=glibc.pthread.rseq=1&quot; sets an environment variable that changes a glibc setting when MongoDB starts. This is the part that works around the kernel incompatibility.</li><li>The empty ExecStart= clears the original start command. This is needed because systemd doesn’t allow two ExecStart lines for this kind of service. Without it, the override would fail.</li><li>ExecStart=/usr/bin/mongod --config /etc/mongod.conf sets the start command again, using the default MongoDB path and config file.</li></ul><p>Save with Ctrl + O, press Enter, then exit with Ctrl + X. Then reload systemd and restart MongoDB.</p><pre>sudo systemctl daemon-reload<br>sudo systemctl restart mongod<br>sudo systemctl status mongod</pre><ul><li>daemon-reload makes systemd read our new override file. Without it, the change is ignored.</li><li>restart mongod stops and starts MongoDB again.</li><li>status mongod checks the result. We should now see Active: active (running) in green.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*L12eFhhVWQeDKmwA6vPGOQ.png" /></figure><p>After restarting, systemctl status mongod shows Active: active (running) in green, so MongoDB is now running and we can continue to the next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*hCfnafQqUwPHmEN0g1oA5g.png" /><figcaption>mongod --version shows the installed version.</figcaption></figure><h4>2.2. Using mongosh</h4><p>mongosh is MongoDB’s command-line shell. We use it to try basic commands directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*0HiufbSglOrRgPkW4OZRRw.png" /></figure><p>Inside the shell:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/353/1*0GFUViTX9szQFCWiiy58wQ.png" /><figcaption>use praktikum_bigdata switches to the database. It’s created when we first save data in it.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*tdNb4drIFiypBRPTD2tMqA.png" /><figcaption>db.produk.insertOne({...}) inserts one document. The produk collection is created automatically.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*IfCHLRky78K3YSLW9MV6jg.png" /><figcaption>db.produk.find() shows all documents in the collection.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*sGR5J7LIgPYDaHXl5PUxUg.png" /><figcaption>.pretty() formats the output so it’s easier to read.</figcaption></figure><p>If want to leave from mongod type exit and enter.</p><p>We never defined a table structure. This is the <strong>flexible schema</strong> in action.</p><h4>2.3. Connecting to MongoDB from Python</h4><p>From here, we work in Jupyter Notebook. First, install PyMongo.</p><pre>pip install pymongo</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*XUtIQSN1qjWW0a9ql3_XNg.png" /><figcaption>pip install pymongo installs the driver. We don’t need to lock the version, because PyMongo is backward compatible.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*Is3suH0ujLCEEMyp1oK5Dw.png" /><figcaption>python -c &quot;...&quot; checks that the install worked.</figcaption></figure><p>Then go to Jupyter Notebook.</p><pre>cd ~/praktikum-bigdata<br>jupyter notebook</pre><p>Now we connect to MongoDB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*MmHyprToqyFffe9OKi8oYg.png" /></figure><ul><li>MongoClient(&quot;mongodb://localhost:27017/&quot;) connects to MongoDB on our own machine, using the default port 27017.</li><li>client[&quot;praktikum_bigdata&quot;] selects the database (created automatically if it doesn’t exist).</li><li>db[&quot;produk&quot;] selects the collection (also created automatically).</li><li>client.list_database_names() lists all databases, which proves the connection works.</li></ul><p>If we see the database list without errors, PyMongo is connected.</p><h4>2.4. Create: Inserting Documents</h4><p>CRUD stands for Create, Read, Update, Delete. We start with Create.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*HdMAPpHqJkQbf6sYbAbG2g.png" /></figure><ul><li>delete_many({}) deletes all documents. We do this so the cell is safe to re-run without duplicates.</li><li>insert_one({...}) inserts one document.</li><li>insert_many([...]) inserts many documents at once from a list.</li><li>count_documents({}) counts how many documents are in the collection.</li></ul><p>Look at tags: some products have 1 tag and some have 2. In SQL this needs an extra table, but in MongoDB it’s just a list.</p><h4>2.5. Read: Finding Documents</h4><p>Now we read the data back with filters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6CZ4nMMzxR2UmeLt6fvZw.png" /></figure><ul><li>find_one({...}) returns only the first matching document.</li><li>find({...}) returns all matching documents as a cursor, so we loop over it with for.</li><li>{&quot;harga&quot;: {&quot;$gt&quot;: 15000}} means “price greater than 15000”. $gt is a comparison operator.</li><li>{p[&#39;harga&#39;]:,} formats the number with commas, like 18,000.</li></ul><p>MongoDB also adds a unique _id to every document automatically.</p><h4>2.6. Update: Changing Documents</h4><p>Next, we change existing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*sB0CyWk6l2YjkEp8XllJsw.png" /></figure><ul><li>update_one(filter, change) updates only the first matching document.</li><li>update_many(filter, change) updates all matching documents.</li><li>$set replaces a field’s value. Here, the Es Teh price becomes 9000.</li><li>$inc changes a number relative to its current value. -5 means “minus 5”, like selling 5 drinks.</li><li>{&quot;_id&quot;: 0, &quot;nama_produk&quot;: 1, ...} is a projection. 1 means show the field, 0 means hide it.</li></ul><p>Only the drinks lost 5 stock. The food items stayed the same.</p><h4>2.7. Delete: Removing Documents</h4><p>Last in CRUD, we delete a document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*44BoG7eJgESYDZ_rxSfHWw.png" /></figure><ul><li>delete_one({...}) removes the first matching document. delete_many removes all matches.</li><li>The list comprehension [p[&quot;nama_produk&quot;] for p in ...] collects only the product names into a list.</li></ul><h4>2.8. MongoDB to Spark DataFrame</h4><p>Now the Big Data part. We take data from MongoDB, put it in pandas, then convert it to a Spark DataFrame.</p><p>2.8.1. Converting the Data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*WYZCx7NIpDGADcISm1TZxA.png" /></figure><ul><li>SparkSession.builder ... .getOrCreate() starts Spark. .master(&quot;local[*]&quot;) runs it on our machine using all CPU cores.</li><li>setLogLevel(&quot;ERROR&quot;) hides noisy logs.</li><li>list(koleksi_produk.find({})) pulls every document from MongoDB into a Python list.</li><li>pd.DataFrame(dokumen_mongo) turns that list into a pandas DataFrame.</li><li>df_pandas[&quot;_id&quot;].astype(str) converts ObjectId to a string, because Spark doesn’t know that type.</li><li>df_pandas[&quot;tags&quot;].apply(lambda ...) joins each list into one string like &quot;dingin, murah&quot;, because Spark can’t read Python lists from pandas this way.</li><li>spark.createDataFrame(df_pandas) converts the pandas DataFrame into a Spark DataFrame.</li><li>printSchema() shows the column names and types, and show() shows the rows.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xyLdc5RAFXpr74ZU6B9I1Q.png" /><figcaption>output</figcaption></figure><p>The MongoDB data is now a normal Spark DataFrame.</p><p>2.8.2. Analyzing with Spark</p><p>Now we can use Spark functions on it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*I2ClitNAbfiRx1q_jb-xMA.png" /></figure><ul><li>groupBy(&quot;kategori&quot;) groups the rows by category.</li><li>avg(&quot;harga&quot;) calculates the average price in each group.</li><li>count(&quot;nama_produk&quot;) counts the products in each group.</li><li>.alias(...) gives the result columns clearer names.</li><li>spark.stop() closes Spark to free memory.</li></ul><p>This “PyMongo -&gt; pandas -&gt; Spark” way works well for thousands to millions of documents. For billions of documents, we use the official <strong>MongoDB Spark Connector</strong> instead.</p><h3>3. Practice</h3><p>Before starting, make sure client, db, and koleksi_produk from 2.3 are still active. Run that cell again if the kernel was restarted.</p><h4>3.1. Question 1: Insert Five New Documents</h4><p>We add five snack products to the produk collection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beWq9ensdygClGJegqTOQw.png" /></figure><ul><li>insert_many([...]) adds all five documents in one go.</li><li>&quot;kategori&quot;: &quot;Snack&quot; is a brand new category. MongoDB doesn’t mind, because there is no fixed schema.</li><li>count_documents({}) confirms the total: 4 old + 5 new = 9.</li></ul><h4>3.2. Question 2: Products Cheaper Than 20000</h4><p>We use $lt to filter products by price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*hT0w2JESqt6XJhSarQqAxQ.png" /></figure><ul><li>{&quot;harga&quot;: {&quot;$lt&quot;: 20000}} means “price less than 20000”. $lt works like $gt, just in the other direction.</li><li>The for loop prints each matching product.</li></ul><p>Snickers (exactly 20000) is not included, because $lt means strictly less than.</p><h4>3.3. Question 3: Add a diskon Field to Every Document</h4><p>We add a new field to all documents at once.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*la54UM0BSnPh8f3o-0jf7Q.png" /></figure><ul><li>{} as the filter matches every document.</li><li>{&quot;$set&quot;: {&quot;diskon&quot;: 0}} creates the diskon field with value 0. If a field doesn’t exist yet, $set creates it.</li></ul><p>No ALTER TABLE is needed, which is the flexible schema again.</p><h4>3.4. Question 4: Reflection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AvwaHQsk9-XWkYRKS54kg.png" /></figure><h3>4. Assignment</h3><p>The scenario is a coffee shop platform called <strong>Kopi Nusantara</strong>. Each branch has seasonal menus with different attributes. Only coffee has tingkat_kepahitan (bitterness), and only food has pedas_level (spiciness). Our job is to build and analyze the menu catalog in MongoDB.</p><h4>4.1. Connecting to the Database</h4><p>First, we connect to a new database called kopi_nusantara.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*fwGRx408Hu9SWJ565MrDoQ.png" /></figure><ul><li>client[&quot;kopi_nusantara&quot;] selects the new database.</li><li>db[&quot;menu&quot;] selects the menu collection.</li><li>list_database_names() shows that kopi_nusantara is now in the list.</li></ul><h4>4.2. Part A: Build the Catalog (Create)</h4><p>We insert at least 12 menu items. Coffee and food have different fields.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*R09Op2lLqT7i830kXXCnYA.png" /></figure><ul><li>menu.delete_many({}) clears old data so the cell can be re-run safely.</li><li>Coffee documents have tingkat_kepahitan (scale 1–5) but no pedas_level.</li><li>Food documents have pedas_level (scale 0–3) but no tingkat_kepahitan.</li><li>stok is an extra field we added ourselves.</li><li>There are 7 coffee and 6 food items, which meets the minimum of 12.</li></ul><h4>4.3. Part B: Multi-Level Queries (Read)</h4><p>Now we run three different queries on the catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*2ixXJ2iJMY8Gai62pXLcCg.png" /></figure><ul><li>{&quot;kategori&quot;: &quot;Kopi&quot;, &quot;tingkat_kepahitan&quot;: {&quot;$gt&quot;: 3}} has two conditions, so both must be true (like AND).</li><li>{&quot;$gte&quot;: 15000, &quot;$lte&quot;: 30000} makes a range. $gte means “greater than or equal” and $lte means “less than or equal”.</li><li>{&quot;$exists&quot;: False} finds documents that don’t have the field at all.</li><li>$exists: False is different from a value of 0. Roti Bakar has pedas_level: 0, so it does have the field and is not included.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*e6WjJjSbJygPs_W_biJ31Q.png" /><figcaption>Output</figcaption></figure><p>Query 2 includes Roti Bakar (15000) because $gte is inclusive. Query 3 returns exactly the 7 coffee items.</p><h4>4.4. Part C: Updating Data (Update)</h4><p>We add a status field to every menu, then mark two items as sold out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*e51TzLUrPDApl1NJXG3IcQ.png" /></figure><ul><li>update_many({}, {&quot;$set&quot;: {&quot;status&quot;: &quot;Tersedia&quot;}}) gives every menu the status “Tersedia” (available).</li><li>update_one({&quot;nama_menu&quot;: &quot;Espresso&quot;}, ...) changes only Espresso to “Habis” (sold out). We did the same for Pisang Goreng.</li><li>Order matters. If update_many ran after the two update_one calls, it would overwrite “Habis” back to “Tersedia”.</li></ul><h4>4.5. Part D: Integration with Spark</h4><p>This is the tricky part. Coffee has no pedas_level and food has no tingkat_kepahitan, but Spark needs every row to have the same columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*Rs0GRp1H9_JoJKpVpHgQ2Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31lXkn-gco6U0-eEDHDhhg.png" /></figure><ul><li>pd.DataFrame(dokumen_mongo) builds one table from all documents. Fields that a document doesn’t have become NaN.</li><li>df_pandas[&quot;_id&quot;].astype(str) converts ObjectId to a string so Spark can read it.</li><li>astype(&quot;Int64&quot;) (with capital I) is pandas’ nullable integer. A normal integer column can’t hold NaN, so pandas would turn it into float (like 5.0). With Int64, the numbers stay as integers.</li><li>df_pandas.astype(object).where(df_pandas.notnull(), None) replaces every empty value with None, which Spark reads as null.</li><li>spark.createDataFrame(df_pandas) converts the cleaned table into a Spark DataFrame.</li><li>groupBy(&quot;kategori&quot;).agg(avg(...), count(...)) calculates the average price and menu count per category.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*Z0JOgBDO61e2u_3p9F1hkA.png" /><figcaption>Output</figcaption></figure><p>4.6. Part E: Assignment Conclusion</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ItrXuQz7USRrts1IV6MLag.png" /></figure><h3>5. Conclusion</h3><p>we learned that MongoDB is a good choice for semi-structured data, because each document can have its own fields without a fixed table schema. We installed it on Ubuntu, used PyMongo to do CRUD operations, and moved the data into Spark through PyMongo -&gt; pandas -&gt; Spark. The main challenge was making the data uniform for Spark: converting ObjectId to a string and turning missing fields into null. After that, we could run analysis like groupBy on data that started with different fields.</p><h3>6. Reference</h3><p><em>Modul Praktikum Big Data Pertemuan 9 — Basis Data NoSQL: Instalasi MongoDB &amp; Integrasi dengan PySpark</em>. Magelang: Program Studi Teknologi Informasi, Universitas Tidar.</p><h3>7. GitHub Link</h3><p><a href="https://github.com/KristianAgungN/Big_Data/tree/main/Pertemuan9">Big_Data/Pertemuan9 at main · KristianAgungN/Big_Data</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3aaf943a836" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jaringan Komputer(9): VLAN in Cisco Packet Tracer]]></title>
            <link>https://medium.com/@xtian.agung.2020/jaringan-komputer-9-vlan-5b13cf97087d?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b13cf97087d</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[vlan]]></category>
            <category><![CDATA[cisco-packet-tracer]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[computer-network]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 05:03:59 GMT</pubDate>
            <atom:updated>2026-11-01T06:50:23.548Z</atom:updated>
            <content:encoded><![CDATA[<h3>I. INTRODUCTION</h3><h3>1.1 Background</h3><p>VLAN (Virtual Local Area Network) is a technology that divides one physical network into several logical networks. It allows devices connected to the same switch to be separated into different groups.</p><p>VLANs can improve network management and help separate network traffic. Devices in the same VLAN can communicate with each other, while devices in different VLANs need a router or Layer 3 device to communicate.</p><p>In this practical work, I used Cisco Packet Tracer to configure VLANs on a Cisco 2960 switch. I created VLAN 10 and VLAN 20 based on the practical module, then added another PC to VLAN 20 and created VLAN 30 as an additional experiment.</p><h3>1.2 Objectives</h3><p>The objectives of this practical work are:</p><ul><li>To understand the basic concept of VLAN.</li><li>To configure VLANs on a Cisco switch.</li><li>To assign PCs to different VLANs.</li><li>To test network connectivity using Add Simple PDU in Cisco Packet Tracer.</li></ul><h3>II. METHODOLOGY</h3><h3>2.1 Network Devices and Components</h3><p>The simulation was built using <strong>Cisco Packet Tracer</strong> with the following virtual devices:</p><ul><li><strong>Switches:<br>- 1</strong>Switch (2960)</li><li><strong>End Devices:<br>- 9 </strong>Wired PCs (PC)</li></ul><h3>2.2 Network Topology</h3><p>I created a network topology using one switch and nine PCs. The PCs were divided into three VLANs.</p><ul><li>VLAN 10<br>- Device (PC0, PC1)<br>- Network (10.10.44.0/24)</li><li>VLAN 20<br>- Device (PC2, PC3, PC4)<br>- Network (20.20.44.0/24)</li><li>VLAN 30<br>- Device (PC5, PC6, PC7, PC8)<br>- Network (30.30.44.0/24)</li></ul><p>VLAN 10 and VLAN 20 were created based on the practical module. I added one more PC to VLAN 20 and created VLAN 30 with four PCs.</p><p>Each VLAN uses a different IP network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*Vkc5-vRI57zS0Q1hkrib9w.png" /><figcaption><em>Network topology with three VLANs.</em></figcaption></figure><h3>2.3 IP Address Configuration</h3><p>Next, I configured the IP addresses and subnet masks for all PCs.</p><ul><li>PC0-PC1<br>IP Address: 10.10.44.1–10.10.44.2<br>Subnet Mask: 255.255.255.0</li><li>PC2-PC4<br>IP Address: 20.20.44.3–20.20.44.5<br>Subnet Mask: 255.255.255.0</li><li>PC5-PC8<br>IP Address: 30.30.44.6–30.30.44.9<br>Subnet Mask: 255.255.255.0</li></ul><p>To configure an IP address, I opened a PC, selected <strong>Desktop</strong>, and clicked <strong>IP Configuration</strong>. Then, I entered the IP address and subnet mask.</p><p>I repeated these steps for all PCs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*fgmD3o5Z5BzChtwZ2M-AFA.png" /><figcaption><em>Configuring the IP address and subnet mask.</em></figcaption></figure><h3>2.4 VLAN Configuration</h3><p>After configuring the IP addresses, I opened the switch CLI and entered the following commands:</p><p>First, create VLAN 10 and VLAN 20.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*ke6NDcu5TFiM39ViRSqicQ.png" /></figure><p>Then, assigned the switch ports connected to each PC to the correct VLAN.</p><p>For example, if PC0 and PC1 were connected to FastEthernet0/1 and FastEthernet0/2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*I04N1FGomEbHypbKmFFBJA.png" /></figure><p>Next, I created VLAN 30 as an additional experiment and assigned the ports connected to PC5, PC6, PC7, and PC8 to VLAN 30.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*BScBOrL51SJB6iWADjjs2Q.png" /></figure><p>Finally, I checked the VLAN configuration using show vlan</p><p>This command displays the VLAN information on the switch, including VLAN IDs, VLAN names, status, and assigned ports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*mUmr8yn5x3joPc181KXftg.png" /><figcaption>Checking the VLAN configuration.</figcaption></figure><h3>2.5 Connectivity Testing</h3><p>After configuring the VLANs, I tested the network connections using <strong>Add Simple PDU</strong> in Cisco Packet Tracer.</p><p>I selected the envelope icon, clicked the source PC, and then clicked the destination PC. I repeated this process to test communication within the same VLAN and between different VLANs.</p><p>The tests included:</p><ul><li>PC0 to PC1 in VLAN 10.</li><li>PC2 to PC3 in VLAN 20.</li><li>PC5 to PC6 in VLAN 30.</li><li>PC0 to PC2 between VLAN 10 and VLAN 20.</li><li>PC0 to PC5 between VLAN 10 and VLAN 30.</li><li>PC2 to PC5 between VLAN 20 and VLAN 30.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*ELvPEe9CPE3Eg5apgl_fEA.png" /><figcaption>Testing network connectivity using Add Simple PDU.</figcaption></figure><h3>III. RESULTS AND ANALYSIS</h3><h3>3.1 VLAN Configuration Results</h3><p>The network was designed with three VLANs containing nine PCs. VLAN 10 had two PCs, VLAN 20 had three PCs, and VLAN 30 had four PCs.</p><p>The show vlancommand was used to check whether the VLANs existed and whether the correct switch ports were assigned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*mUmr8yn5x3joPc181KXftg.png" /></figure><h3>3.2 Connectivity Test Results</h3><p>The Add Simple PDU feature was used to test connections between PCs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*ELvPEe9CPE3Eg5apgl_fEA.png" /></figure><p>The connectivity tests help demonstrate how VLANs separate devices into different logical networks.</p><p>PCs in the same VLAN should be able to communicate because they belong to the same logical network. However, PCs in different VLANs cannot communicate directly through a Layer 2 switch.</p><p>This practical work also showed that one switch can manage several VLANs. Adding another PC to VLAN 20 and creating VLAN 30 allowed me to practice configuring a larger network with more than two VLANs.</p><h3>IV. CONCLUSION</h3><p>In this practical work, I learned how to configure VLANs using Cisco Packet Tracer. I created VLAN 10 and VLAN 20, added one PC to VLAN 20, and created VLAN 30 with four PCs.</p><p>I also learned how to assign switch ports to VLANs and test network connectivity using Add Simple PDU. This practical work helped me understand how VLANs separate network devices into different logical groups and make network management more organized.</p><h3>V. REFERENCE</h3><ul><li><em>Modul 9: Virtual Local Area Network (VLAN)</em>. Praktikum Jaringan Komputer. Magelang: Program Studi Teknologi Informasi, Universitas Tidar.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b13cf97087d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PBO(9): Learning Abstraction with Java]]></title>
            <link>https://medium.com/@xtian.agung.2020/pbo-9-learning-abstraction-with-java-8822e4d5a23f?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/8822e4d5a23f</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[objectorientedprogramming]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Thu, 29 Oct 2026 14:45:23 GMT</pubDate>
            <atom:updated>2026-10-29T14:45:23.790Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><h3>What Is Abstraction?</h3><p>Abstraction is a concept in OOP that hides unnecessary details and shows only the important parts of a program. This makes the code easier to understand and use.</p><p>For example, we can use a car without knowing how its engine works. We only need to know how to use it.</p><h3>What Is an Abstract Class?</h3><p>An abstract class is a class that cannot be used to create an object directly. It can contain abstract methods and regular methods.</p><p>An abstract method is a method without a body. Classes that inherit from the abstract class must provide their own implementation of that method.</p><h3>2. Hands-On</h3><h3>2.1. Creating the Kendaraan Class</h3><p>First, we create an abstract class called Kendaraan. This class contains an abstract method named berjalan() and a regular method named info().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*Vzu2IeJZ9Vk_fAaduIcqaw.png" /></figure><ul><li>abstract class Kendaraan defines an abstract class named Kendaraan.</li><li>abstract void berjalan(); declares a method without a body. Its subclasses must implement this method.</li><li>void info() defines a regular method that can be used by its subclasses.</li><li>System.out.println() displays a message in the output.</li></ul><p>The Kendaraan class acts as a basic structure for other vehicle classes.</p><h3>2.2. Creating the Mobil Class</h3><p>Next, we create a class called Mobil that inherits from Kendaraan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*2-13Ho0ykQlneE6cEETkRA.png" /></figure><ul><li>extends Kendaraan means that Mobil inherits from Kendaraan.</li><li>@Override indicates that this class provides its own implementation of an inherited method.</li><li>void berjalan() implements the abstract method from Kendaraan.</li><li>System.out.println() displays the message &quot;Mobil sedang berjalan dengan cepat.&quot;</li></ul><p>This means that the Mobil class defines its own behavior for the berjalan() method.</p><h3>2.3. Creating the Sepeda Class</h3><p>Now, we create another subclass called Sepeda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*XDpsXj3BSLexX6vpA416wQ.png" /></figure><ul><li>extends Kendaraan allows Sepeda to inherit from Kendaraan.</li><li>@Override indicates that the method is implemented in this class.</li><li>void berjalan() provides the behavior of a bicycle.</li><li>System.out.println() displays the message &quot;Sepeda sedang berjalan dengan pelan.&quot;</li></ul><p>Although Mobil and Sepeda inherit from the same class, they have different implementations of berjalan().</p><h3>2.4. Creating the Main Class</h3><p>After creating the other classes, we create Main.java to run the program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*fYKzmfLjuq3EHhWB8K7Egw.png" /></figure><ul><li>Kendaraan mobil = new Mobil(); creates a Mobil object using a Kendaraan reference.</li><li>Kendaraan sepeda = new Sepeda(); creates a Sepeda object using a Kendaraan reference.</li><li>mobil.berjalan(); calls the method implemented in Mobil.</li><li>sepeda.berjalan(); calls the method implemented in Sepeda.</li><li>mobil.info(); and sepeda.info(); call the regular method inherited from Kendaraan.</li></ul><p>The Main class lets us test how the abstract class works with its subclasses.</p><h3>3. Program Output</h3><p>After running Main.java, the program displays the following output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*99eFUjX0HnYyxIGsw42xNg.png" /></figure><p>The output shows that Mobil and Sepeda have different implementations of berjalan(). However, both classes can use the same info() method from Kendaraan.</p><h3>Can We Create an Object from Kendaraan?</h3><p>An abstract class cannot be instantiated directly, we need to create an object from a concrete subclass, such as Mobil or Sepeda. This happens because Kendaraan is an abstract class.</p><h3>4. Assignment</h3><p>In this assignment, we will create an abstract class called Hewan and two subclasses called Kucing and Anjing.</p><p>The goal is to make each animal produce a different sound using the same abstract method.</p><h3>4.1. Creating the Hewan Class</h3><p>The Hewan class contains an abstract method called suara().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*JXssKDpKOy--2RD01bPOLw.png" /></figure><ul><li>abstract class Hewan defines an abstract class named Hewan.</li><li>abstract void suara(); declares a method that must be implemented by its subclasses.</li></ul><p>This class provides a basic structure for the animal classes.</p><h3>4.2. Creating the Kucing Class</h3><p>The Kucing class inherits from Hewan and implements the suara() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*EAa07cooymtYfkDzqd9oLQ.png" /></figure><ul><li>extends Hewan allows Kucing to inherit from Hewan.</li><li>@Override indicates that the method is implemented in this subclass.</li><li>suara() displays the cat&#39;s sound using System.out.println().</li></ul><p>The output message is Kucing mengeluarkan suara: Meong.</p><h3>4.3. Creating the Anjing Class</h3><p>The Anjing class also inherits from Hewan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*ozoZLnRsbnUV8SA2huF0lQ.png" /></figure><ul><li>extends Hewan allows Anjing to inherit from Hewan.</li><li>@Override indicates that the method is implemented in this subclass.</li><li>suara() displays the dog&#39;s sound using System.out.println().</li></ul><p>The output message is Anjing mengeluarkan suara: Guk Guk.</p><h3>4.4. Running the Program</h3><p>Create a main class to call the methods from Kucing and Anjing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*rlCfKOKzOAxKWxDVQ03OzQ.png" /></figure><p>The output shows that each animal has its own implementation of the suara() method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*KsB8PffDLGyMhzyHMuiAeA.png" /></figure><h3>5. Insights</h3><p>From this practice, we can learn a few things:</p><ul><li>An abstract class cannot be instantiated directly.</li><li>An abstract method does not have a method body.</li><li>Subclasses must implement the abstract methods they inherit.</li><li>An abstract class can also contain regular methods.</li><li>Different subclasses can implement the same abstract method in different ways.</li></ul><p>These concepts help us organize our Java code and make it easier to understand.</p><h3>6. Conclusion</h3><p>In this practice, we learned how to use abstraction and abstract classes in Java.</p><p>We created vehicle and animal classes to see how subclasses implement abstract methods in their own ways. We also learned that abstract classes can contain regular methods that their subclasses can reuse.</p><p>Overall, abstraction helps us create a clear structure for our programs without needing to define every detail in the abstract class.</p><h3>7. Reference</h3><p>Modul 9: Abstraksi dan Kelas Abstrak. Magelang: Program Studi Teknologi Informasi, Universitas Tidar.</p><h3>8. GitHub Link</h3><p><a href="https://github.com/KristianAgungN/Pemrograman-Berorientasi-Objek/tree/main/Pertemuan9/PraktikumPBO_9/src/main/java/praktikum9">https://github.com/KristianAgungN/Pemrograman-Berorientasi-Objek/tree/main/Pertemuan9/PraktikumPBO_9/src/main/java/praktikum9</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8822e4d5a23f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jaringan Komputer(7): Creating my own Topology]]></title>
            <link>https://medium.com/@xtian.agung.2020/jaringan-komputer-7-creating-my-own-topology-84b75bcf6e8c?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/84b75bcf6e8c</guid>
            <category><![CDATA[routing]]></category>
            <category><![CDATA[subnetting]]></category>
            <category><![CDATA[network-topology]]></category>
            <category><![CDATA[cisco-packet-tracer]]></category>
            <category><![CDATA[wireless-lan]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 05:17:48 GMT</pubDate>
            <atom:updated>2026-10-19T09:14:37.729Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*BIISbInVHIz7AetSpp8ldw.png" /><figcaption>Here’s the design I made, based on a building at my campus.</figcaption></figure><blockquote>I. INTRODUCTION</blockquote><h3>1.1 Background</h3><p>In designing modern computer networks, an infrastructure must be able to bridge internal local communications with external or public networks securely and efficiently. A standard comprehensive network architecture consists of two main segments: the Inside (Local) Network, which integrates routers, switches, wireless access points, and wired/wireless end devices utilizing DHCP and WPA2-PSK security; and the Outside (Internet) Network, which connects to ISP routers, public servers, and implements Network Address Translation (NAT). This practical work builds a custom network topology in Cisco Packet Tracer based on specific structural requirements.</p><h3>1.2 Objective</h3><ul><li>Design and build a custom network topology containing both Inside (Local) and Outside (Internet) architectures in Cisco Packet Tracer.</li><li>Implement an Inside Network featuring a Main Router (Gateway), a Switch, a Wireless Router, and a mix of wired/wireless end devices with DHCP and WPA2-PSK security.</li><li>Implement an Outside Network featuring an ISP Router connected to the Main Router, a Public Server, and NAT configuration.</li></ul><blockquote>II. METHODOLOGY</blockquote><h3>2.1 Network Devices and Components</h3><p>The simulation was built using <strong>Cisco Packet Tracer</strong> with the following virtual devices:</p><ul><li><strong>Routers:<br>- </strong>Main Router(PT-Router)<br>- RouterISP(PT-Router)</li><li><strong>Switches:<br>- 2 </strong>Switch (2960)</li><li><strong>Wireless Devices:<br>- </strong>Wireless Router (WRT300N)</li><li><strong>Servers:<br>- </strong>Public Server (Server)</li><li><strong>End Devices:<br>- 3 </strong>Wired PCs (PC)<br>- 3 Wireless Laptops (Laptop)<br>- 2 Smartphones (Smartphone)</li></ul><h3>2.2 Procedure</h3><p>a. Outside (Internet) Architecture:</p><p>Place the Public Server and ISP Router in the workspace, connecting them using a crossover cable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*5cNwklL2LahVLuOAsDLwVw.png" /></figure><p>Assign public IP addresses</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*PY4N53hbJzTokiLD1d8JiQ.png" /><figcaption>Server to RouterISP</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*4soxTYgPo6Wk5E3AerjaQA.png" /><figcaption>RouterISP to Server</figcaption></figure><p>b. Inside (Local) Architecture:</p><ul><li>Connect a Switch to the Main Router using a straight-through cable</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*_QF6FBp6h_IPRYPNLXGsJQ.png" /></figure><ul><li>Configure DHCP on the main routers for automated IP distribution.</li></ul><p>Main Router to Switch:<br>Port status: On<br>IP Address: 192.168.44.1<br>Subnet Mask: 255.255.255.0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*pXlP42YP_cKlqCo2Dq6LKw.png" /><figcaption>Main Router to Switch</figcaption></figure><p>Main Router to RouterISP:<br>Port status: On<br>IP Address: 44.44.44.2<br>Subnet Mask: 255.255.255.0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*WlwUKNqpvkDCuC5xqYOPow.png" /><figcaption>Main Router to RouterISP</figcaption></figure><p>RouterISP to Main Router:<br>Port status: On<br>IP Address: 44.44.44.1<br>Subnet Mask: 255.255.255.0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*Ls0PQRWoV0lmGJy-DtPopg.png" /><figcaption>RouterISP to Main Router</figcaption></figure><ul><li>Configure DHCP on the wireless routers for automated IP distribution, and set up WPA2-PSK security on the wireless network.</li></ul><p>Wireless router Config:</p><p>Internet Connection type: Automatic Configuration — DHCP<br>IP Address: 192.168.0.1<br>Subnet Mask: 255.255.255.0<br>DHCP Server: Enabled<br>Start IP Address: 2<br>Maximum Users: 50</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D1v4iLPMTdVn01ZRrVHwuA.png" /></figure><p>Network Name: Wifi-kelas<br>SSID Broadcast: Enabled</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PuuUEnFwTIOcp8vZOPNHQ.png" /></figure><p>Security Mode: WPA2 Personal<br>Encryption: AES<br>Passphrase: kelas123</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HLaJtZ-YlTV8dSQOLfwq6A.png" /></figure><p>c. Configure the DHCP Server Pool on Main Router</p><ul><li>Set up an automatic IP address distribution pool for the local network (192.168.44.0/24)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*rkzy0QwpsJ7wBA8EQFxPsw.png" /><figcaption>CLI in Main Router</figcaption></figure><p>d. Configuring NAT</p><p>Set up NAT overload on the Main/ISP routers so that private local IP addresses can communicate with the external public server network.</p><ul><li>Configure the default static route on Main Router pointing to the ISP router&#39;s IP address (44.44.44.1) as the next hop (0.0.0.0 0.0.0.0 44.44.44.1).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*8-eFBINw2iNBNs54Ge-DiQ.png" /></figure><ul><li>Define the inside and outside interfaces on Main Router</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*jAZWE1pDiYSjBAqo61cH8g.png" /><figcaption>NAT Inside</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*VPdC1On43P61rBLOGXnG5A.png" /><figcaption>NAT Outside</figcaption></figure><ul><li>Create an Access Control List to permit the local network subnet (192.168.44.0/24) and enable NAT Overload on the public interface (fa0/0)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*KYfnksKG3QCAVXRCLjGFoQ.png" /><figcaption>CLI in Main Router</figcaption></figure><blockquote>III. RESULTS AND ANALYSIS</blockquote><h3>3.1 Implementation Results</h3><p>The network topology was created and tested using Cisco Packet Tracer. The network consists of an inside network and an outside network. The inside network connects wired and wireless devices, while the outside network connects the main router to the ISP router and public server.</p><p>To access wireless on laptop we need to change laptop module into WPC300N.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*D2RaPY2qbC-uhrwdqx4tPw.png" /><figcaption>Change laptop module into WPC300N</figcaption></figure><p>click laptop&gt;desktop&gt;pc wireless</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*rIc_ppI3hbbeK1Mx4v-h8w.png" /><figcaption>This will pop up</figcaption></figure><p>Choose Security WPA2-Personal then fill pre-shared Key with “kelas123” then connect</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*tKaqEJwqwATBJekfJJqlkg.png" /></figure><p>Or you can like this</p><p>Click laptop&gt;config&gt;Wireless0</p><p>Port Status: On<br>SSID: Wifi-kelas<br>Authentication: WPA2-PSK<br>PSK Pass Phrase: kelas123<br>Encryption Type: AES<br>IP Configuration: DHCP</p><ul><li>They received IP addresses automatically via DHCP and secured their connection using WPA2-PSK.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*a-BuF8bExP7uw3F2jchZxQ.png" /><figcaption>Laptop Wireless</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*dtbm3fnvFpXu0qCkRKEzSQ.png" /><figcaption>Smartphone Wireless</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*9uZ5xw5jiGRWXBOMxNH1Tg.png" /><figcaption>PC Wired</figcaption></figure><h3>3.1.1. Wireless Network Testing</h3><ul><li>The laptops were connected to the wireless network using the SSID Wifi-kelas. WPA2-PSK with AES encryption was used to protect the wireless connection.</li><li>Ping test from wireless clients<br>- laptop to server(1.1.1.1) successful<br>- laptop to wireless router(192.168.0.1) successful<br>- laptop to main router(192.168.44.1) successful</li><li>The ping tests were successful. This shows that the tested laptop could connect to the wireless router, the main router, and the public server.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/502/1*-YYolUUeIFhoyUhL3M5kiw.png" /><figcaption>prove</figcaption></figure><h3>3.1.2. Wired Network Testing</h3><ul><li>The PCs were connected to the network using a Cisco 2960 switch. The main router was used as the gateway for the local network.</li><li>Ping test from wired PCs<br>- PC to server(1.1.1.1) successful<br>- PC to PC(192.168.44.3) successful<br>- PC to main router(192.168.44.1) successful</li><li>The ping tests were successful. This shows that the tested PC could communicate with another PC, the main router, and the public server.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*FkXqeYZCb1QQOblkqlCqoA.png" /><figcaption>prove</figcaption></figure><ul><li>Implementation Summary</li></ul><p>Based on the test results, the network topology worked as expected for the tested connections. Both wired and wireless devices could communicate with their destinations.</p><h3>3.2 Analysis</h3><h3>3.2.1. Inside Network</h3><ul><li>The inside network connects PCs, laptops, and smartphones through switches, a wireless router, and the main router.</li><li>The switches allow wired devices to communicate with each other. The wireless router allows laptops and smartphones to connect to the network.</li><li>DHCP automatically assigns IP addresses to devices. This makes network configuration easier because users do not need to enter IP addresses manually.</li><li>WPA2-PSK with AES encryption helps protect the wireless network from unauthorized access.</li><li>The successful ping tests show that the tested wired and wireless devices could communicate with the main router and other destinations.</li></ul><h3>3.2.2. Outside Network</h3><ul><li>The outside network connects the main router to the ISP router and the public server.</li><li>The main router uses a default route to send traffic to the ISP router at 44.44.44.1. This allows traffic to be forwarded to external networks when no more specific route is available.</li><li>NAT allows devices using private IP addresses to communicate with external networks by translating their source IP addresses.</li><li>The successful ping tests to 1.1.1.1 show that the tested PCs and laptops could reach the public destination.</li></ul><blockquote>IV. Conclusion</blockquote><p>The Inside network effectively integrated the main gateway router, switch, wireless router, DHCP, and WPA2-PSK security, while the Outside network successfully connected the ISP router and public server using NAT.</p><blockquote>V. Reference</blockquote><ul><li><em>Modul Praktikum Jaringan Komputer — Pertemuan 3: Dasar Routing dan Switching</em>. Program Studi Teknologi Informasi, Universitas Tidar.</li><li><em>Modul Praktikum Jaringan Komputer: Pertemuan 4 — DHCP dan NAT</em>. Universitas Tidar.</li><li><em>Modul 5 Praktikum Jaringan Komputer: Wireless Network</em>. Program Studi Teknologi Informasi.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=84b75bcf6e8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data(7): End-to-End Big Data Analytics with HDFS, PySpark, and Parquet]]></title>
            <link>https://medium.com/@xtian.agung.2020/big-data-7-end-to-end-big-data-analytics-with-hdfs-pyspark-and-parquet-870023eb0920?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/870023eb0920</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 10:13:13 GMT</pubDate>
            <atom:updated>2026-10-16T10:13:13.414Z</atom:updated>
            <content:encoded><![CDATA[<p>Learning Objectives:<br>- Designing and executing end-to-end Big Data analysis workflows, ranging from multiple data sources to the final report.<br>- Integrating HDFS, the Spark DataFrame API, Spark SQL, and the Parquet format into a single, coherent pipeline.<br>- Generate business summaries or insights from large-scale data processing.</p><h3>Hands-On</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*srb1J2EUstG6mKZReQIN8A.png" /><figcaption>Case study</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*SzLI0E2Gvq7j_GM03kJ8Eg.png" /><figcaption>Import function, SparkSession and creating SparkSession</figcaption></figure><p>Random seed makes the random numbers the same every time, so my results match yours. The store file is saved as JSON Lines, meaning one JSON record per line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*UUrxGJLcmMhm-M9gCxzXVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*cwToDWoVXOhlta7dZP01dg.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*OoiwXPewyeIUX4gbn1Mvqg.png" /><figcaption>Upload the raw data to HDFS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*Q0teE8hytcpNaj0b6z3D4g.png" /><figcaption>Spark reads the three files from HDFS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*kxPsFIxx_JVLaZpUca6KZg.png" /><figcaption>Join the three tables</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*vPcNM4qy1BaTZA_M-4V6Hw.png" /><figcaption>Show top 3 products in each store using window function</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*6JynnF0IbrGm4tzT9a9iXg.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*tqzfAvxkKBnxMvHAS7t8YA.png" /><figcaption>Sales per manager</figcaption></figure><p>Save the final table back to HDFS as Parquet. overwrite replaces old results if the folder exists. partitionBy(&quot;kategori&quot;) puts each category in its own sub-folder, so later queries that need only one category read less data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*INdmyBg7dEC9zpacKbMqgw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTv9Iosa5ZbcGhxtOsf0pQ.png" /><figcaption>Read the saved Parquet back to prove it was stored correctly</figcaption></figure><h3>Practice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4re4m7ki3s7xkc9Ie1L4A.png" /><figcaption>Restart Spark and reload the data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*bWvMAO_m-334PLe0zaSIsQ.png" /></figure><ul><li>createOrReplaceTempView(&quot;gabungan&quot;): gives the DataFrame a temporary table name, gabungan, so SQL can use it.</li><li>spark.sql(&quot;&quot;&quot;...&quot;&quot;&quot;): runs an SQL query. The triple quotes let the query use many lines.</li><li>SUM(qty) AS total_qty: adds up the quantity. The question asks for units sold, not rupiah, so we use qty and not total_penjualan.</li><li>GROUP BY kategori: one result row per category.</li><li>ORDER BY total_qty DESC: the category with the most units comes first.</li><li>LIMIT 1: keeps only the first row, so only the winning category is shown.</li><li>.show(): prints the result.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSwCI5geFR6N7a9EwoDeQA.png" /></figure><ul><li>groupBy(&quot;kategori&quot;, &quot;nama_toko&quot;): puts rows with the same category and the same store into one group.</li><li>.agg(spark_sum(&quot;total_penjualan&quot;).alias(&quot;total_penjualan&quot;)): adds up the sales in each group. alias sets the name of the result column.</li><li>Window.partitionBy(&quot;kategori&quot;): makes a separate “window” for each category, so the ranking restarts in every category.</li><li>.orderBy(col(&quot;total_penjualan&quot;).desc()): inside each window, sorts from the highest sales to the lowest.</li><li>w: a short name for the window rule, so we can reuse it in the next step.</li><li>row_number().over(w): gives each store its position number (1, 2, 3, 4) inside its category.</li><li>.withColumn(&quot;peringkat&quot;, ...): saves that number in a new column named peringkat (rank).</li><li>.filter(col(&quot;peringkat&quot;) == 1): keeps only rank 1, the best store in each category.</li><li>.orderBy(&quot;kategori&quot;): sorts the result by category name.</li><li>.show(truncate=False): prints the table with full text, so nothing is cut off.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kPlBtDfUujbxg-1YOmQ7HA.png" /></figure><ul><li>groupBy(&quot;nama_toko&quot;, &quot;kota&quot;, &quot;manager&quot;): one group per store. City and manager are included so they appear in the table.</li><li>spark_sum(&quot;total_penjualan&quot;): total revenue of each store in the quarter.</li><li>count(&quot;trx_id&quot;): number of transactions of each store.</li><li>avg(&quot;total_penjualan&quot;): average money per transaction.</li><li>.orderBy(col(&quot;total_penjualan_triwulan&quot;).desc()): the store with the highest revenue comes first.</li><li>!hdfs dfs -mkdir -p /user/yan/latihan7: makes the HDFS folder first. ! runs a terminal command from the notebook, and -p means no error if the folder already exists.</li><li>.write.mode(&quot;overwrite&quot;): starts saving the DataFrame. If the folder already has data, it is replaced.</li><li>.parquet(&quot;hdfs://...&quot;): saves in Parquet format at this HDFS path. There is no partitionBy, so all data goes into one folder, with no sub-folders. The summary has only 4 rows, so splitting it up would not help.</li><li>!hdfs dfs -ls ...: lists the folder to confirm the file was saved.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AqRdrWsmYQNJpKJuPO4xVQ.png" /></figure><h3>Assignment</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EyVCqYc7sPeHZPsck3XkKA.png" /><figcaption>Scenario</figcaption></figure><p>Run this code below to get dataset</p><pre>import numpy as np<br>import pandas as pd<br>import json<br><br>np.random.seed(123)<br><br># Sumber 1: Data cabang (JSON)<br>cabang_list = [&quot;Kopi Nusantara Magelang&quot;, &quot;Kopi Nusantara Yogyakarta&quot;, &quot;Kopi Nusantara Semarang&quot;]<br>data_cabang = [<br>    {&quot;cabang_id&quot;: i + 1, &quot;nama_cabang&quot;: c, &quot;kota&quot;: c.split()[-1],<br>     &quot;kepala_barista&quot;: np.random.choice([&quot;Wulan&quot;, &quot;Reza&quot;, &quot;Nadia&quot;])}<br>    for i, c in enumerate(cabang_list)<br>]<br>with open(&quot;tugas7_cabang.json&quot;, &quot;w&quot;) as f:<br>    for c in data_cabang:<br>        f.write(json.dumps(c) + &quot;\n&quot;)<br><br># Sumber 2: Data menu (CSV)<br>menu_list = [<br>    {&quot;menu_id&quot;: i, &quot;nama_menu&quot;: f&quot;Menu-{i}&quot;,<br>     &quot;kategori_menu&quot;: np.random.choice([&quot;Kopi&quot;, &quot;Non-Kopi&quot;, &quot;Makanan Ringan&quot;]),<br>     &quot;harga&quot;: int(np.random.choice([18000, 22000, 25000, 30000, 35000]))}<br>    for i in range(1, 16)<br>]<br>pd.DataFrame(menu_list).to_csv(&quot;tugas7_menu.csv&quot;, index=False)<br><br># Sumber 3: Data transaksi (CSV) — data utama<br>n = 4000<br>data_transaksi_tugas7 = pd.DataFrame({<br>    &quot;trx_id&quot;: [f&quot;KN{i}&quot; for i in range(n)],<br>    &quot;cabang_id&quot;: np.random.randint(1, 4, size=n),<br>    &quot;menu_id&quot;: np.random.randint(1, 16, size=n),<br>    &quot;qty&quot;: np.random.randint(1, 5, size=n),<br>})<br>data_transaksi_tugas7.to_csv(&quot;tugas7_transaksi.csv&quot;, index=False)<br><br>!hdfs dfs -mkdir -p /user/yan/tugas7/raw<br>!hdfs dfs -put -f tugas7_transaksi.csv /user/yan/tugas7/raw/<br>!hdfs dfs -put -f tugas7_cabang.json /user/yan/tugas7/raw/<br>!hdfs dfs -put -f tugas7_menu.csv /user/yan/tugas7/raw/<br><br>print(&quot;Ketiga sumber data siap &amp; data transaksi sudah diunggah ke HDFS.&quot;)<br>print(f&quot;- Cabang: {len(data_cabang)} | Menu: {len(menu_list)} | Transaksi: {n}&quot;)</pre><ul><li>np.random.seed(123): fixes the random numbers, so the data is always the same.</li><li>data_cabang: 3 branches, each with an ID, the branch name, the city (the last word of the name), and a randomly chosen head barista. They are saved as JSON Lines, one record per line.</li><li>menu_list: 15 menu items with a name, a random category, and a random price. They are saved as CSV.</li><li>data_transaksi_tugas7: 4,000 transactions with a branch ID (1 to 3), a menu ID (1 to 15), and a quantity (1 to 4).</li><li>!hdfs dfs -mkdir -p and -put -f: create the HDFS folder and upload the three files.</li><li>The two print lines: confirm the data is ready and show the row counts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*rTMc7UJvTZzdvFAQRgTMiw.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89P7tZLhe5rt8wX7baW9UQ.png" /><figcaption>Creating SparkSession</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*I_Xv8EB8RIrMzxIjdTN1rw.png" /><figcaption>Question 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*laaXqsPa61sBSCgJUY_VpQ.png" /></figure><ul><li>base = &quot;hdfs://localhost:9000/user/yan/tugas7/raw&quot;: saves the shared HDFS folder path in one variable, so we don’t type it three times. The hdfs:// start means the data is read from HDFS, not from the local disk, as the task requires.</li><li>f&quot;{base}/tugas7_transaksi.csv&quot;: an f-string. Python puts the value of base in the braces and adds the file name.</li><li>spark.read.csv(...): reads a CSV file into a Spark DataFrame (a table).</li><li>header=True: the first row is the column names, not data.</li><li>inferSchema=True: Spark looks at the values and decides each column’s data type.</li><li>spark.read.json(...): reads the branch file (JSON Lines) into a DataFrame.</li><li>.count(): counts rows, so I can check that all data was loaded.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*5mOf289MGB8zBEkfhgujIg.png" /><figcaption>Question 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*7iQmBiVT-Gyt9hHZf4R-jA.png" /></figure><ul><li>df_transaksi.join(df_cabang, on=&quot;cabang_id&quot;, how=&quot;inner&quot;): adds branch details (name, city, head barista) to each transaction by matching cabang_id.</li><li>how=&quot;inner&quot;: keeps only rows that have a match in both tables.</li><li>.join(df_menu, on=&quot;menu_id&quot;, how=&quot;inner&quot;): then adds menu details (name, category, price) by matching menu_id.</li><li>\: lets one command continue on the next line.</li><li>.withColumn(&quot;total_penjualan&quot;, col(&quot;qty&quot;) * col(&quot;harga&quot;)): creates a new column that is quantity times price.</li><li>df_gabungan.show(5): prints the first 5 rows.</li><li>print(&quot;Jumlah baris gabungan:&quot;, df_gabungan.count()): checks that the join kept all 4,000 rows. If rows were lost, some IDs did not match.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*_qeTdOqTSmVya8NkI7V2Ow.png" /><figcaption>Question 3</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*gu4sfK0j5hhLeIA9lwglhw.png" /></figure><ul><li>groupBy(&quot;nama_cabang&quot;, &quot;nama_menu&quot;, &quot;kategori_menu&quot;): puts rows of the same branch and the same menu item into one group. kategori_menu is added so the category shows in the final table. It does not change the groups, because each menu item has only one category.</li><li>.agg(spark_sum(&quot;total_penjualan&quot;).alias(&quot;total_penjualan&quot;)): adds up the sales in each group. alias sets the name of the result column.</li><li>Window.partitionBy(&quot;nama_cabang&quot;): makes a separate “window” for each branch, so the ranking restarts in every branch.</li><li>.orderBy(col(&quot;total_penjualan&quot;).desc()): inside each window, sorts from the highest sales to the lowest.</li><li>row_number().over(window_cabang): gives each menu item its position number (1, 2, 3...) inside its branch.</li><li>.withColumn(&quot;peringkat&quot;, ...): saves that number in a new column named peringkat (rank).</li><li>.filter(col(&quot;peringkat&quot;) &lt;= 2): keeps only rank 1 and 2, the top-2 menu items.</li><li>.orderBy(&quot;nama_cabang&quot;, &quot;peringkat&quot;).show(truncate=False): sorts by branch and rank, then prints. truncate=False shows full text, so branch names are not cut off with “...”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*dxxWCjT0Nnxes5AvY4wh5g.png" /><figcaption>Question 4</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*H_6ro5FjGeG9gKztUhk6EA.png" /></figure><ul><li>createOrReplaceTempView(&quot;gabungan&quot;): gives the DataFrame a temporary table name, gabungan, so SQL can use it.</li><li>spark.sql(&quot;&quot;&quot;...&quot;&quot;&quot;): runs an SQL query and returns the result as a DataFrame. The triple quotes let the query use many lines. All the work is inside the query, which is what “pure Spark SQL” means.</li><li>SELECT kepala_barista, SUM(total_penjualan) AS total_penjualan: picks the head barista and adds up the sales.</li><li>FROM gabungan: the table we read from.</li><li>GROUP BY kepala_barista: one result row per head barista.</li><li>ORDER BY total_penjualan DESC: highest total first.</li><li>penjualan_barista.show(): prints the result.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDzHTLNXVvoutq35ECMnvQ.png" /><figcaption>Question 5</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*YKVozY626w2GDoZV_uxC7g.png" /></figure><ul><li>!hdfs dfs -mkdir -p .../processed: makes the HDFS folder for finished results, separate from the raw data. ! runs a terminal command from the notebook.</li><li>path_laporan = &quot;hdfs://...&quot;: saves the target path in a variable, so we can reuse it for writing and reading.</li><li>df_gabungan.write.mode(&quot;overwrite&quot;): starts saving the DataFrame. If the folder already has data, it is replaced.</li><li>.partitionBy(&quot;kategori_menu&quot;): creates one sub-folder per menu category (Kopi, Makanan Ringan, Non-Kopi). A later query for one category reads only that folder.</li><li>.parquet(path_laporan): saves in Parquet format, which stores data by column and compresses it.</li><li>!hdfs dfs -ls ...: lists the folder content to confirm the save worked.</li><li>df_verifikasi = spark.read.parquet(path_laporan): reads the saved data back from HDFS.</li><li>df_verifikasi.count(): counts the rows read back. It must be 4,000, the same as before saving. If it matches, no data was lost.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lu67OvVBkFOxWXnW79jUBw.png" /><figcaption>Question 6</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*jbTz7iskx6ukn6Cet5oyUQ.png" /></figure><ul><li>df_verifikasi.groupBy(&quot;nama_cabang&quot;): uses the data we read back from Parquet, so the summary comes from the saved report, not from memory. It makes one group per branch.</li><li>spark_sum(&quot;total_penjualan&quot;): total revenue of each branch.</li><li>count(&quot;trx_id&quot;): number of transactions of each branch.</li><li>avg(&quot;total_penjualan&quot;): average money per transaction.</li><li>.alias(...): gives each result column a clear name.</li><li>.orderBy(col(&quot;total_penjualan&quot;).desc()): the branch with the highest revenue comes first.</li><li>.show(truncate=False): prints the table with full text.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wb0uMe---hymPr9L1K9c0Q.png" /><figcaption>Markdown</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*s-XltM2IsfJM1qSFIqUCVg.png" /><figcaption>Closing SparkSession</figcaption></figure><h3>Conclusion</h3><p>In this practicum, we built a complete ETL (Extract, Transform, Load) pipeline: we read raw CSV and JSON data from HDFS, processed it with PySpark using join, row_number(), and Spark SQL, and saved the final report back to HDFS as Parquet. HDFS stored the data, Spark processed it in memory, and Parquet kept the results small and fast, with partitioning so queries read only the folders they need. In the retail case study, Toko Semarang had the highest sales, and in the Kopi Nusantara assignment, Semarang was the best branch and Yogyakarta the weakest, so we suggested promoting Menu-15 together with Menu-12 there. This shows how Big Data tools can turn raw data into clear business decisions.</p><h3>Reference</h3><p><em>Modul Praktikum Big Data: Pertemuan 7 — Studi Kasus Terintegrasi: Analisis Big Data End-to-End</em>. Program Studi Teknologi Informasi, Universitas Tidar.</p><h3>GitHub Link</h3><ul><li><a href="https://github.com/KristianAgungN/Big_Data/blob/main/Pertemuan7/Modul_Praktikum_BigData_Pertemuan7.ipynb">Big_Data/Pertemuan7/Modul_Praktikum_BigData_Pertemuan7.ipynb at main · KristianAgungN/Big_Data</a></li><li><a href="https://github.com/KristianAgungN/Big_Data/blob/main/Pertemuan7/Tugas7_2505060044_Kristian_Agung_Nugroho.ipynb">Big_Data/Pertemuan7/Tugas7_2505060044_Kristian_Agung_Nugroho.ipynb at main · KristianAgungN/Big_Data</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=870023eb0920" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data(6): In Big Data, Why Parquet and JSON better than CSV?]]></title>
            <link>https://medium.com/@xtian.agung.2020/big-data-6-in-big-data-why-parquet-and-json-better-than-csv-6491602d72a5?source=rss-a355d39c6c5f------2</link>
            <guid isPermaLink="false">https://medium.com/p/6491602d72a5</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Kristian Agung Nugroho]]></dc:creator>
            <pubDate>Wed, 30 Sep 2026 15:38:46 GMT</pubDate>
            <atom:updated>2026-09-30T15:38:46.748Z</atom:updated>
            <content:encoded><![CDATA[<p>CSV is easy to read for human, but it is slow and inefficient for Big Data. Since CSV is plain text, Spark has to guess data types every time, and because it stores data row-by-row, Spark is forced to read entire rows even if you only need a single column.</p><p>To fix this, we use <strong>Parquet</strong> and <strong>JSON Lines. </strong>Parquet stores data by column to automatically shrink file sizes and allow Spark to read only necessary columns, while JSON Lines easily handles nested data and supports fast parallel processing.</p><h3>Hands-On</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spq9b8xC1f17S3QQ3n0DHQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*rnizl4nCANn7q6YVtdlgIQ.png" /><figcaption>Create SparkSession</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EFTMlAMsgpiXk0tyTAC-LA.png" /><figcaption>Generates 20,000 dummy transaction rows, saves them as CSV, converts them to Parquet, and compares file sizes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*NxTaj4ZC9ZQ2Dr_XOyQ99A.png" /><figcaption>Calculates total file size of CSV and Parquet to prove Parquet compresses data significantly smaller</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*p2m0rSN6N6vQY4dnbbUfbw.png" /><figcaption>Loads data directly from the Parquet folder</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A0Eys5Uur8S0Z7Zc_NBAJg.png" /><figcaption>Splits dataset into sub-folders based on values in the kota column</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Smm5YbvnmOKfh2PEHUq8w.png" /><figcaption>Creates 20 dummy customer records in JSON Lines format</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*yQ9J17S-pARCwh1K_SeeNw.png" /><figcaption>reads JSON into PySpark</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*utuE3jCX1B4F71pyP3Yn0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*f6YErZB-ijBg5bNNy7k7UQ.png" /><figcaption>Reads raw CSV transactions and JSON customer details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*S4hXThJleYOMXEnhUcSxUw.png" /><figcaption>Merges datasets using join() on customer_id and uses when() conditions to categorize revenue levels (Tinggi, Sedang, Rendah)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*YUX_xbdBC21Xp8oiTLVCDw.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*mdHZwwRzKKTzhOQLCJMBwQ.png" /><figcaption>Writes clean output to Hadoop HDFS in Parquet format partitioned by product category (kategori)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*vqNTCeBiGXioQ6JeloFa6w.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*SGc1jtNdQKo20xFJC7CLFw.png" /><figcaption>Reads from HDFS to verify the output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*bRv6MqER7OT_y4ChtfkmLw.png" /><figcaption>Closing SparkSession</figcaption></figure><h3>Practice</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*eTw-ENj02pisEn-W0-sN2g.png" /><figcaption>Create SparkSession</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*30y6TgAYQH2akEaRIX0Pxg.png" /><figcaption>Reads Parquet files to verify data types</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*UidcvwH57oxO8abTi8OxTw.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fd-KlNK1Rs3mDD95JDnzrw.png" /><figcaption>Joins transactions (CSV) with customers (JSON) on customer_id and counts orders per membership tier</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wflJVJ5tVN9DTr_5BDkG4Q.png" /><figcaption>Saves the joined dataset locally as Parquet partitioned by membership, creating sub-folders for each tier</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Bu0io_F6leFyn6p2Jc_Jw.png" /></figure><h3>Assignment</h3><p>Run this to get 3 dataset</p><pre>import numpy as np<br>import pandas as pd<br>import json<br><br>np.random.seed(33)<br><br># Sumber 1: Transaksi (CSV) — 5000 baris<br>n_trx = 5000<br>kategori_list = [&quot;Elektronik&quot;, &quot;Fashion&quot;, &quot;Makanan&quot;, &quot;Rumah Tangga&quot;, &quot;Kesehatan&quot;]<br>df_trx_tugas6 = pd.DataFrame({<br>    &quot;order_id&quot;: [f&quot;TX{i}&quot; for i in range(n_trx)],<br>    &quot;product_id&quot;: np.random.randint(1, 31, size=n_trx),<br>    &quot;unit_terjual&quot;: np.random.randint(1, 8, size=n_trx),<br>    &quot;tanggal&quot;: np.random.choice(pd.date_range(&quot;2026-10-01&quot;,&quot;2026-10-31&quot;), size=n_trx).astype(str),<br>})<br>df_trx_tugas6.to_csv(&quot;tugas6_transaksi.csv&quot;, index=False)<br><br># Sumber 2: Data produk (JSON Lines) — 30 produk<br>produk = [<br>    {&quot;product_id&quot;: i, &quot;nama_produk&quot;: f&quot;Produk-{i}&quot;, &quot;kategori&quot;: np.random.choice(kategori_list),<br>     &quot;harga&quot;: int(np.random.choice([25000,50000,75000,100000,150000,250000]))}<br>    for i in range(1, 31)<br>]<br>with open(&quot;tugas6_produk.json&quot;, &quot;w&quot;) as f:<br>    for p in produk:<br>        f.write(json.dumps(p) + &quot;\n&quot;)<br><br># Sumber 3: Data ulasan (CSV) — tidak semua transaksi memiliki ulasan (realistis)<br>n_review = 3500<br>df_review = pd.DataFrame({<br>    &quot;order_id&quot;: np.random.choice(df_trx_tugas6[&quot;order_id&quot;], size=n_review, replace=False),<br>    &quot;rating&quot;: np.random.randint(1, 6, size=n_review),<br>})<br>df_review.to_csv(&quot;tugas6_ulasan.csv&quot;, index=False)<br><br>print(f&quot;Tiga sumber data berhasil dibuat:&quot;)<br>print(f&quot;- tugas6_transaksi.csv : {len(df_trx_tugas6)} baris&quot;)<br>print(f&quot;- tugas6_produk.json   : {len(produk)} baris&quot;)<br>print(f&quot;- tugas6_ulasan.csv    : {len(df_review)} baris (tidak seluruh transaksi memiliki ulasan)&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*bayevjV4JAwFBHJw0kRTiA.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/1*lPGKJiTr6LLk9bHWZ8gcUQ.png" /><figcaption>Here’s the dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*xEqsL8gQCe-M-7Y73HJGCw.png" /><figcaption>Create SparkSession</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*DhKtr_oHzwYRJXNwxh50xw.png" /><figcaption>Reads transactions (CSV), products (JSON), and reviews (CSV) into 3 Spark DataFrames</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*zspQ69k8sRDghocwRt-IlQ.png" /><figcaption>Output</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*u0ypBQIGriUf2oV8RB9poA.png" /><figcaption>Combines the DataFrames using an Inner Join for products and a Left Join for reviews, then calculates total_pendapatan (unit_terjual * harga)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*oxsPkZGk4ApYMKfGd_eu0w.png" /><figcaption>Adds ada_ulasan (True/False)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IW7fRp8vUXS_KDPKj35a-A.png" /><figcaption>fills null ratings with 0 to prevent calculation errors without skewing valid ratings (1–5)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iQaMkdijZeZt6dLPa4pNiQ.png" /><figcaption>Saves the clean data to HDFS in Parquet format, partitioned by kategori</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*5Q53jsgErYeOQnl9yRncNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*eWe7TPSzOyjpW0cTYTpUAA.png" /><figcaption>Calculates review percentages per category so marketing can target low-engagement areas with review incentives</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cW2N8YX8DHfvNNcoqvYvhA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*jTEt3psKELxmfyCtztblHw.png" /><figcaption>Closing SparkSession</figcaption></figure><h3>Conclusion</h3><p>Parquet compresses files and preserves data types much better than CSV. We used partitioning to optimize queries, read JSON files, and built a complete ETL pipeline with Hadoop HDFS.</p><h3>Reference</h3><ul><li>Modul Praktikum Big Data — Pertemuan 6: Format Data Besar (Parquet &amp; JSON) dan Pipeline ETL Sederhana, Program Studi Teknologi Informasi, Universitas Tidar.</li></ul><h3>GitHub Link</h3><ul><li><a href="https://github.com/KristianAgungN/Big_Data/blob/main/Pertemuan_6/Modul_Praktikum_BigData_Pertemuan6.ipynb">Big_Data/Pertemuan_6/Modul_Praktikum_BigData_Pertemuan6.ipynb at main · KristianAgungN/Big_Data</a></li><li><a href="https://github.com/KristianAgungN/Big_Data/blob/main/Pertemuan_6/Tugas6_2505060044_Kristian_Agung_Nugroho.ipynb">Big_Data/Pertemuan_6/Tugas6_2505060044_Kristian_Agung_Nugroho.ipynb at main · KristianAgungN/Big_Data</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6491602d72a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>