<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohammed Galalen on Medium]]></title>
        <description><![CDATA[Stories by Mohammed Galalen on Medium]]></description>
        <link>https://medium.com/@mgalalen?source=rss-35c46c11872d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y4KvF2Cf7NowVjN4Dh4tJQ.jpeg</url>
            <title>Stories by Mohammed Galalen on Medium</title>
            <link>https://medium.com/@mgalalen?source=rss-35c46c11872d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 02:10:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mgalalen/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why Cassandra Won’t Let You Change a Column’s Type (And Why That’s a Feature)]]></title>
            <link>https://medium.com/@mgalalen/why-cassandra-wont-let-you-change-a-column-s-type-and-why-that-s-a-feature-744423d6fc1c?source=rss-35c46c11872d------2</link>
            <guid isPermaLink="false">https://medium.com/p/744423d6fc1c</guid>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[design-systems]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[cassandra]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Mohammed Galalen]]></dc:creator>
            <pubDate>Mon, 11 May 2026 17:56:42 GMT</pubDate>
            <atom:updated>2026-06-09T19:51:17.747Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*alXQF2IOtsyALnzb3VxklQ.png" /></figure><p>One Cassandra limitation that confused me for a while: you can’t simply change a column’s data type, not because it was forgotten, but because the storage engine makes it fundamentally unsafe in most cases.</p><p>At first glance, it feels like a limitation. In most databases, you can simply run:</p><pre>ALTER TABLE users ALTER age TYPE text;</pre><p>But in Cassandra, that’s not allowed for most type changes, and in the new versions, the command is effectively <strong>disabled</strong>. There were historically a few narrow exceptions for byte-compatible changes like ascii → text, but even those have been removed in modern Cassandra versions — precisely because they were error-prone.</p><p>To understand why, you need to understand how Cassandra actually stores data.</p><h3>How Cassandra Stores Data</h3><p>Cassandra is built on an LSM-tree (log-structured storage engine). The write path looks like this:</p><pre>Write → Commit Log → Memtable → Flush → Immutable SSTable</pre><p>Cassandra doesn’t update data in place. Once data is flushed to an SSTable, it is never modified. Updates aren’t overwritten — instead, new cells are written with newer timestamps, and Cassandra resolves conflicts at read time using <strong>last-write-wins</strong>. Older, shadowed cells are eventually cleaned up through <strong>compaction</strong>.</p><p>When Cassandra writes data, it serializes values into bytes using the column’s declared type. Simplified, the on-disk cell looks like:</p><pre>[timestamp] + [serialized value bytes]</pre><p>The serialization depends entirely on the column type. If you change the type, Cassandra has no way to reinterpret those existing bytes correctly.</p><h3>The Bytes Are the Problem</h3><p>Consider int 13 vs the text &quot;13&quot;:</p><ul><li>As an int, it&#39;s 4 fixed bytes: 0x00 0x00 0x00 0x0D</li><li>As text, it&#39;s 2 UTF-8 bytes: 0x31 0x33</li></ul><p>Different length. Different encoding. Different meaning.</p><p>To migrate between them, Cassandra would have to:</p><pre>Read every SSTable → Deserialize using old type → Re-encode using new type → Rewrite across all nodes</pre><p>That’s not a schema change. That’s a full cluster-wide data migration — across potentially terabytes of immutable files, on every replica.</p><h3>The Schema Is Distributed, Not Centralized</h3><p>Unlike traditional databases like Postgres, Cassandra runs across multiple nodes, where schema changes propagate asynchronously.</p><p>This means:</p><ul><li>Nodes may temporarily have different schema versions.</li><li>Reads may involve merging results from multiple replicas.</li><li>Data is eventually consistent, not globally locked.</li></ul><p>If column types could change, different nodes might interpret the same data differently (e.g., int vs text), leading to <strong>inconsistent reads and potential corruption</strong>.</p><p>Cassandra avoids this by allowing only <strong>forward-compatible schema changes</strong>.</p><h3>Why Adding a Column Works</h3><p>When you run:</p><pre>ALTER TABLE users ADD email text;</pre><p>Cassandra doesn’t rewrite anything. Old SSTables stay exactly as they were:</p><p>Old SSTables:</p><pre>Row A → {name: &quot;A&quot;, age: 20}<br>Row B → {name: &quot;B&quot;, age: 25}</pre><p>New writes include the new column:</p><pre>Row C → {name: &quot;C&quot;, age: 30, email: &quot;c@mail.com&quot;}</pre><p>On reads, Cassandra merges data from memtables and multiple SSTables; missing columns are simply treated as NULL. This works because adding a column is forward-compatible: it doesn’t require reinterpreting any existing bytes.</p><p><strong>This is why Cassandra is so fast.</strong> Because it never has to go back and rewrite old files when you change the schema, there is zero performance hit. While other databases might lock your table for hours to update every row, Cassandra’s schema changes are instant. You aren’t changing the old data; you’re just giving the database new instructions for the future.</p><h3>Dropping a Column Doesn’t Delete the Bytes</h3><p>If you run into this limitation, your first thought is probably:</p><pre>ALTER TABLE users DROP age;<br>ALTER TABLE users ADD age text;</pre><p>Same column name, new type. Problem solved, right?</p><p>It isn’t. And depending on the version of Cassandra, it can either fail outright or — worse — succeed, corrupting your reads.</p><p>Dropping a column in Cassandra doesn’t physically remove the data from existing SSTables. It just records a <em>drop timestamp</em> in the schema. Cells written before that timestamp are filtered out at read time; cells written after are kept. The old bytes are still sitting on disk, waiting for compaction to clean them up eventually.</p><p>Now you re-add a column with the same name but a different type. Cassandra is strict about this for exactly that reason — modern versions will reject the re-add if the new type isn’t compatible with the old one’s serialized bytes, because:</p><ul><li>Old SSTables still contain cells under the original column name, encoded with the old type.</li><li>A read that touches both old and new SSTables would have to merge cells with incompatible encodings under the same column name.</li></ul><p>The short version: drop-and-re-add is not a workaround for type changes. It’s a way to turn a clean error into a subtle one.</p><p>If you genuinely want to reuse a column name with a new type, you need to drop it, run a full major compaction on every node to physically purge the old data, and wait out gc_grace_seconds, and only then re-add.</p><h3>What You Should Do Instead</h3><p>If you actually need to change a type, the safe pattern is:</p><ul><li><strong>Add</strong> a new column.</li><li>Update code to <strong>dual-write</strong>.</li><li><strong>Backfill</strong> (using a custom script or specialized tools like DataStax Bulk Loader).</li><li>Update code to <strong>read from the new, fallback to the old</strong> (optional but safer).</li><li>Update code to <strong>read/write only new</strong>.</li><li><strong>Drop</strong> the old column.</li></ul><h3>Final Thoughts</h3><p>This isn’t unique to Cassandra. Any append-only or LSM-based system runs into the same constraint: immutable storage enables high write throughput and horizontal scale, but it also means schema evolution must move forward.</p><h3>References</h3><ul><li><a href="https://cassandra.apache.org/doc/latest/cassandra/architecture/storage-engine.html">https://cassandra.apache.org/doc/latest/cassandra/architecture/storage-engine.html</a></li><li><a href="https://cassandra.apache.org/doc/latest/cassandra/reference/cql-commands/alter-table.html">https://cassandra.apache.org/doc/latest/cassandra/reference/cql-commands/alter-table.html</a></li><li><a href="https://cassandra.apache.org/doc/4.1/cassandra/architecture/dynamo.html">https://cassandra.apache.org/doc/4.1/cassandra/architecture/dynamo.html</a></li><li><a href="https://issues.apache.org/jira/browse/CASSANDRA-11820">https://issues.apache.org/jira/browse/CASSANDRA-11820</a></li><li><a href="https://issues.apache.org/jira/browse/CASSANDRA-12443">https://issues.apache.org/jira/browse/CASSANDRA-12443</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=744423d6fc1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Databases Work Under the Hood: Building a Key-Value Store in Go]]></title>
            <link>https://medium.com/@mgalalen/how-databases-work-under-the-hood-building-a-key-value-store-in-go-2af9a772c10d?source=rss-35c46c11872d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2af9a772c10d</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Mohammed Galalen]]></dc:creator>
            <pubDate>Sat, 01 Feb 2025 09:56:01 GMT</pubDate>
            <atom:updated>2025-02-02T07:58:14.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bb6e2B2JRCsXtdd5k4nTtQ.jpeg" /><figcaption>Photo by <strong>Glen Carrie </strong>on<strong> </strong>Unsplash</figcaption></figure><p>Have you ever wondered how databases like <strong>Postgres</strong>, <strong>MySQL</strong>, or <strong>Cassandra</strong> store and retrieve data so efficiently? I was amazed when I first realized that databases — these powerful, complex systems — are, at their core, just files on disk. <strong>Yes, files! </strong>Structured and optimized for fast and efficient operations. Two key components make this possible:</p><ol><li><strong>Query Layer</strong>: Handles user queries, optimizes them, and interacts with the storage engine.</li><li><strong>Storage Engine</strong>: Manages how data is stored, retrieved, and updated on disk or in memory.</li></ol><h4>What is a Storage engine?</h4><p>A <strong>storage engine</strong> is a core component of a database system responsible for managing how data is stored, retrieved, and updated on disk or in memory. It handles low-level details such as file organization, indexing, and concurrency control. At its heart, a storage engine relies on efficient data structures designed specifically for disk-based access.</p><p>Storage engines can be broadly classified into two types based on how they handle data modifications:</p><ol><li><strong>Mutable Storage Engines:</strong> These engines, like B-Trees, are optimized for read-heavy workloads. They update data in place, meaning existing data is overwritten when modified. Examples include Postgres, InnoDB (MySQL), and SQLite.</li><li><strong>Immutable Storage Engines:</strong> These engines, like Log-Structured Merge Trees (LSM Trees) and Log-Structured Hash Tables (LSHT), are optimized for write-heavy workloads. Instead of overwriting data, they append new data to a log file and periodically clean up old or deleted data through a process called compaction. Examples include LevelDB, RocksDB, Bitcask, and Cassandra.</li></ol><p>In this post, we’ll build a simple immutable key-value store<strong>,</strong> inspired by the Bitcask model. Bitcask is based on a Log-Structured Hash Table (LSHT), which appends data sequentially to a log file and maintains an in-memory hash table for fast lookups. LSHT-based engines are simpler to implement compared to other designs like LSM-Trees, as they avoid the complexity of sorting, multi-level merging, and additional data structures.</p><p>By the end of this post, you’ll have a foundational understanding of how storage engines work and a working implementation of a minimal key-value store. In future posts, we’ll enhance this implementation by adding features like compaction, advanced indexing, and handling large datasets efficiently.</p><blockquote>Full source code available on <a href="https://github.com/galalen/minkv">GitHub</a></blockquote><h3>Design Overview</h3><p>Our key-value store uses an append-only binary file format with a simple yet efficient record structure:</p><p>Each record contains:</p><ul><li>Timestamp (4 bytes)</li><li>Key length (4 bytes)</li><li>Value length (4 bytes)</li><li>Tombstone flag (1 byte)</li><li>Key data (variable length)</li><li>Value data (variable length)</li></ul><p><em>Record format:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Eey_W2hrj7BQfTzdbBYyg.png" /></figure><h4>Why Binary Format for Serialization?</h4><p>While JSON and XML are human-readable and widely used for data interchange, most modern storage systems use binary formats for their core storage. Here’s why:</p><ol><li><strong>Performance</strong>: The system can read and write binary formats directly without parsing overhead.</li><li><strong>Space Efficiency</strong>: Binary representations typically require significantly less storage space than text-based formats.</li><li><strong>Cross-Platform Compatibility</strong>: Using standardized binary formats ensures data can be read consistently across different systems.</li></ol><h3>Implementation</h3><h4>Types</h4><pre>type Record struct {<br>   Key       []byte<br>   Value     []byte<br>   Timestamp uint32<br>   Tombstone bool<br>}<br><br>type Store struct {<br>   filename string<br>   mu       sync.RWMutex<br>   file     *os.File<br>   index    map[string]int64<br>}</pre><p>The MinKV structure maintains a persistent file handle and an in-memory index for efficient lookups.</p><h4>Creating the Store</h4><pre>func Open(filename string) (*Store, error) {<br>   file, err := os.OpenFile(filename, os.O_RDWR|os.O_CREATE, 0666)<br>   if err != nil {<br>      return nil, fmt.Errorf(&quot;failed to open file: %w&quot;, err)<br>   }<br>   store := &amp;Store{<br>      filename: filename,<br>      file:     file,<br>      index:    make(map[string]int64),<br>   }<br>  <br>   if err := store.buildIndex(); err != nil {<br>      file.Close()<br>      return nil, fmt.Errorf(&quot;failed to rebuild index: %w&quot;, err)<br>   }<br>  <br>   return store, nil<br>}</pre><h4>Index Rebuilding</h4><p>To support efficient key lookups, the buildIndex function scans the file and populates the in-memory index with the latest offsets for each key. (But if the file is a large building the index will be slow).</p><pre>func (s *Store) buildIndex() error {<br>   s.mu.Lock()<br>   defer s.mu.Unlock()<br>  <br>   var offset int64<br>   for {<br>      record, err := s.readRecord(offset)<br>      if err == io.EOF {<br>         break<br>      }<br>      if err != nil {<br>         return fmt.Errorf(&quot;failed to read record: %v&quot;, err)<br>      }<br>      <br>      // mark tombstoned records as deleted<br>      if record.Tombstone {<br>         s.index[string(record.Key)] = tombstoneOffset<br>      } else {<br>         s.index[string(record.Key)] = offset<br>      }<br>      offset += int64(headerSize + len(record.Key) + len(record.Value))<br>   }<br>   return nil<br>}</pre><h4>Binary Serialization</h4><p>The writeRecord function efficiently serializes records into binary format:</p><pre>func (s *Store) writeRecord(record *Record) (int64, error) {<br>   totalSize := headerSize + len(record.Key) + len(record.Value)<br>   buf := make([]byte, totalSize)<br>  <br>   // serialize timestamp (4 bytes)<br>   binary.BigEndian.PutUint32(buf[0:4], record.Timestamp)<br>  <br>   // serialize key len (4 bytes)<br>   binary.BigEndian.PutUint32(buf[4:8], uint32(len(record.Key)))<br>  <br>   // serialize value len (4 bytes)<br>   binary.BigEndian.PutUint32(buf[8:12], uint32(len(record.Value)))<br>  <br>   // serialize tombstone (1 byte)<br>   if record.Tombstone {<br>      buf[12] = 1<br>   } else {<br>      buf[12] = 0<br>   }<br>  <br>   // copy key data<br>   copy(buf[headerSize:headerSize+len(record.Key)], record.Key)<br>  <br>   // copy value data<br>   copy(buf[headerSize+len(record.Key):], record.Value)<br>  <br>   // write record to file<br>   offset, err := s.file.Seek(0, io.SeekEnd)<br>   if err != nil {<br>      return 0, err<br>   }<br>  <br>   if _, err := s.file.Write(buf); err != nil {<br>      return 0, err<br>   }<br>  <br>   return offset, nil<br>}</pre><p>The readRecord function reads binary-encoded records from the file:</p><pre>func (s *Store) readRecord(offset int64) (*Record, error) {<br>   if _, err := s.file.Seek(offset, io.SeekStart); err != nil {<br>      return nil, fmt.Errorf(&quot;failed to seek: %w&quot;, err)<br>   }<br>  <br>   record := &amp;Record{}<br>  <br>   // read timestamp (4 bytes)<br>   timestampBuf := make([]byte, 4)<br>   if _, err := s.file.Read(timestampBuf); err != nil {<br>      return nil, err<br>   }<br>   record.Timestamp = binary.BigEndian.Uint32(timestampBuf)<br>  <br>   // read key len (4 bytes)<br>   keyLenBuf := make([]byte, 4)<br>   if _, err := s.file.Read(keyLenBuf); err != nil {<br>      return nil, err<br>   }<br>   keyLen := binary.BigEndian.Uint32(keyLenBuf)<br>  <br>   // read value len (4 bytes)<br>   valueLenBuf := make([]byte, 4)<br>   if _, err := s.file.Read(valueLenBuf); err != nil {<br>      return nil, err<br>   }<br>   valueLen := binary.BigEndian.Uint32(valueLenBuf)<br>  <br>   // read tombstone (1 byte)<br>   tombstoneBuf := make([]byte, 1)<br>   if _, err := s.file.Read(tombstoneBuf); err != nil {<br>      return nil, err<br>   }<br>   record.Tombstone = tombstoneBuf[0] == 1<br>  <br>   // read key data<br>   key := make([]byte, keyLen)<br>   if _, err := s.file.Read(key); err != nil {<br>      return nil, err<br>   }<br>   record.Key = key<br>  <br>   // read value data<br>   value := make([]byte, valueLen)<br>   if _, err := s.file.Read(value); err != nil {<br>      return nil, err<br>   }<br>   record.Value = value<br>  <br>   return record, nil<br>}</pre><h4>Put Operation</h4><p>The Put function writes a new record to the file and updates the in-memory index. It ensures thread safety by acquiring a write lock:</p><pre>func (s *Store) Put(key, value []byte) error {<br>   if len(key) == 0 {<br>      return fmt.Errorf(&quot;key cannot be empty&quot;)<br>   }<br>  <br>   s.mu.Lock()<br>   defer s.mu.Unlock()<br>  <br>   record := &amp;Record{<br>      Key:       key,<br>      Value:     value,<br>      Timestamp: uint32(time.Now().Unix()),<br>   }<br>  <br>   offset, err := s.writeRecord(record)<br>   if err != nil {<br>      return fmt.Errorf(&quot;failed to write record: %w&quot;, err)<br>   }<br>  <br>   s.index[string(key)] = offset<br>   return nil<br>}</pre><h4>Get Operation</h4><p>The Get function retrieves a value by its key using the in-memory index. It acquires a read lock to ensure thread safety:</p><pre>func (s *Store) Get(key []byte) ([]byte, error) {<br>   if len(key) == 0 {<br>      return nil, fmt.Errorf(&quot;key cannot be empty&quot;)<br>   }<br>  <br>   s.mu.RLock()<br>   defer s.mu.RUnlock()<br>  <br>   offset, exists := s.index[string(key)]<br>   if !exists || offset == tombstoneOffset {<br>      return nil, fmt.Errorf(&quot;key not found: %s&quot;, key)<br>   }<br>  <br>   record, err := s.readRecord(offset)<br>   if err != nil {<br>      return nil, fmt.Errorf(&quot;failed to read record: %w&quot;, err)<br>   }<br>  <br>   return record.Value, nil<br>}</pre><h4>Delete Operation</h4><p>The Delete function writes a tombstoned record to the file to ensure deletions are persisted across restarts then updates the in-memory index marking the record as deleted.</p><p><em>(Notice the record data still exists in the file we’ll be removing it later when implementing compaction)</em>.</p><pre>func (s *Store) Delete(key []byte) error {<br>   if len(key) == 0 {<br>      return fmt.Errorf(&quot;key cannot be empty&quot;)<br>   }<br>  <br>   s.mu.Lock()<br>   defer s.mu.Unlock()<br>   record := &amp;Record{<br>      Key:       key,<br>      Tombstone: true,<br>      Timestamp: uint32(time.Now().Unix()),<br>   }<br>  <br>   _, err := s.writeRecord(record)<br>   if err != nil {<br>      return fmt.Errorf(&quot;failed to write record: %w&quot;, err)<br>   }<br>  <br>   s.index[string(key)] = tombstoneOffset<br>   return nil<br>}</pre><h4>Getting All the Records</h4><p>We&#39;ve implemented an iterator pattern to efficiently scan all records from the file without loading all the records into memory at once and skipping tombstoned records.</p><pre>type Iterator interface {<br>    Next() bool<br>    Record() (*Record, error)<br>}</pre><p>Example usage:</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br><br> &quot;github.com/galalen/minkv&quot;<br>)<br><br>func main() {<br>   store, _ := minkv.Open(&quot;store.db&quot;)<br>   defer store.Close()<br>  <br>   // Put data<br>   _ = store.Put([]byte(&quot;os&quot;), []byte(&quot;mac&quot;))<br>   _ = store.Put([]byte(&quot;db&quot;), []byte(&quot;kv&quot;))<br>   _ = store.Put([]byte(&quot;lang&quot;), []byte(&quot;go&quot;))<br>  <br>   // Retrieve data<br>   value, _ := store.Get([]byte(&quot;os&quot;))<br>   fmt.Printf(&quot;os =&gt; %s\n&quot;, value)<br>  <br>   // update value<br>   _ = store.Put([]byte(&quot;os&quot;), []byte(&quot;linux&quot;))<br>  <br>   // Delete data<br>   _ = store.Delete([]byte(&quot;db&quot;))<br>  <br>   // Iterate over records<br>   iter, _ := store.Iterator()<br>   for iter.Next() {<br>      record, _ := iter.Record()<br>      fmt.Printf(&quot;Key: %s, Value: %s\n&quot;, record.Key, record.Value)<br>   }<br>}</pre><h4>Future Optimizations</h4><p>Here are some areas where this key-value store can be further improved:</p><ol><li><strong>Compaction:</strong> Currently, deleted or overwritten records remain in the file until compaction is performed. Implementing a compaction mechanism will help reduce file size and improve performance.</li><li><strong>Hintfiles:</strong> Reduce disk seeks and speed up index rebuilding.</li><li><strong>Batching Writes</strong>: Group multiple writes into a single batch to reduce the number of disk sync operations.</li></ol><h3>Conclusion</h3><p>We’ve built a simple yet powerful key-value store that demonstrates core database storage principles. Key-value stores are the backbone of many real-world databases and systems. For example:</p><ul><li><strong>Riak:</strong> A distributed NoSQL database that implements the Bitcask log structure to handle write-heavy operations efficiently.</li><li><strong>Cassandra:</strong> Uses key-value principles for distributed data storage.</li><li><strong>RocksDB:</strong> A key-value store used in systems like MySQL and Kafka, is built on an LSM-based architecture.</li></ul><blockquote>Full source code available on <a href="https://github.com/galalen/minkv">GitHub</a></blockquote><h3><strong>References</strong></h3><ul><li><a href="https://tech-lessons.in/en/blog/bitcask/">https://tech-lessons.in/en/blog/bitcask/</a></li><li><a href="https://riak.com/assets/bitcask-intro.pdf">https://riak.com/assets/bitcask-intro.pdf</a></li><li><a href="https://tikv.org/deep-dive/key-value-engine/introduction/">https://tikv.org/deep-dive/key-value-engine/introduction/</a></li><li><a href="https://tikv.org/deep-dive/key-value-engine/b-tree-vs-lsm/">https://tikv.org/deep-dive/key-value-engine/b-tree-vs-lsm/</a></li><li><a href="https://github.com/avinassh/py-caskdb">https://github.com/avinassh/py-caskdb</a></li><li><a href="https://www.amazon.com/Database-Internals-Deep-Distributed-Systems/dp/1492040347">https://www.amazon.com/Database-Internals-Deep-Distributed-Systems/dp/1492040347</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2af9a772c10d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leverage Mongodb change stream to build a strong ETL pipeline]]></title>
            <link>https://medium.com/analytics-vidhya/leverage-mongodb-change-stream-to-build-a-strong-etl-pipeline-a0deb557e63b?source=rss-35c46c11872d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0deb557e63b</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <dc:creator><![CDATA[Mohammed Galalen]]></dc:creator>
            <pubDate>Wed, 30 Dec 2020 16:57:10 GMT</pubDate>
            <atom:updated>2020-12-30T16:57:10.191Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P_F6BYYqkqi3sKjI" /><figcaption>Photo by <a href="https://unsplash.com/@sortino?utm_source=medium&amp;utm_medium=referral">Joshua Sortino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Before diving into the change stream, let’s know why we want to migrate data from Mongo to another SQL database specifically Postgresql. The main issue raised when we connected an analytical tool like Metabase with mongo, and since we are dealing with document datastore, most of the relationships are embedded, and finding relationships between these documents can be hard enough, and it’s doesn’t fit our analytical pipeline.</p><p>Another thing we have to put in mind, we needed that tool to work in near-realtime (always giving the most recent analytics).</p><p>The best solution was to migrate the data to a SQL database and connect any analytical tool directly to it. Also, this is can be used as a secondary backup for the data because the update is real-time (Thanks Mongo change stream).</p><p>So what is Mongo change Stream?</p><blockquote>According to the <a href="https://docs.mongodb.com/manual/changeStreams/">official Mongo website</a>: “<strong>Change streams allow applications to access real-time data changes without the complexity and risk of tailing the oplog.</strong> <strong>Applications can use change streams to subscribe to all data changes on a single collection</strong>, a database, or an entire deployment, and immediately react to them. Because change streams use the aggregation framework, applications can also filter for specific changes or transform the notifications at will”.</blockquote><h4>How to use it:</h4><p><a href="https://medium.com/riow/mongodb-data-collection-change-85b63d96ff76">I followed this amazing article</a> in order to setup mongo to handle the change stream. It provides golden information about this topic.</p><p>But the most important one is mongo by default is running as a standalone application. “In order to use <strong>changeStreams</strong> you must convert your “<strong>Standalone</strong>” database into a “<strong>Replica Set</strong>” database.”</p><p>With that been said, I highly advise you to check this <a href="https://medium.com/riow/mongodb-data-collection-change-85b63d96ff76">article</a>.</p><p>Mongo changeStreams can be used in a lot of use cases to enable real-time apps to work effectively. The main docs reference that its save time for developers when implementing ETL</p><p><strong>System overview:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*_COQYGXsQnlTW0Dcv9uusg.png" /><figcaption>Created using <a href="https://app.diagrams.net/">https://app.diagrams.net/</a></figcaption></figure><p>This code example is taken from mongo <a href="https://docs.mongodb.com/manual/changeStreams">official website</a>:</p><pre>const collection = db.collection(‘inventory’);<br>const changeStream = collection.watch();</pre><pre>let newChangeStream;<br>changeStream.once(‘change’, next =&gt; {<br> const resumeToken = changeStream.resumeToken;<br> changeStream.close();</pre><pre>newChangeStream = collection.watch({ resumeAfter: resumeToken });<br> newChangeStream.on(‘change’, next =&gt; {<br> processChange(next);<br> });<br>});</pre><p>The above example is using nodejs to listen for changes in the inventory document.</p><ul><li>Another thing added to this system recently is a batch pull, a background service to pull data from MongoDB directly. It runs periodically (5 to 7 days) to ensure data integrity.</li></ul><p><strong>But unfortunately</strong>, there is no free lunch<strong>, </strong>the main downside of this approach adds a maintainability headache to the team. Every new field added or removed from the main schema in MongoDB needs to be reflected in the ETL service and the SQL schema as well. Making it a bit harder to maintain.</p><p>In conclusion, the mongo change stream is a really powerful feature, you will get rid of many third-party tools to enable real-time in your application. And it&#39;s perfect to implement most of the common use cases like ETL, collaborative apps, notifications, and chats. The possibilities are endless.</p><p><strong>References:</strong></p><ul><li><a href="https://docs.mongodb.com/manual/changeStreams/">https://docs.mongodb.com/manual/changeStreams/</a></li><li><a href="https://www.mongodb.com/blog/post/an-introduction-to-change-streams">https://www.mongodb.com/blog/post/an-introduction-to-change-streams</a></li><li><a href="https://medium.com/riow/mongodb-data-collection-change-85b63d96ff76">https://medium.com/riow/mongodb-data-collection-change-85b63d96ff7</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0deb557e63b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/leverage-mongodb-change-stream-to-build-a-strong-etl-pipeline-a0deb557e63b">Leverage Mongodb change stream to build a strong ETL pipeline</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Life-saving Git Commands I Recently Knew]]></title>
            <link>https://medium.com/analytics-vidhya/life-saving-git-commands-i-recently-knew-f69e9c973701?source=rss-35c46c11872d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f69e9c973701</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Mohammed Galalen]]></dc:creator>
            <pubDate>Wed, 22 Jul 2020 20:27:32 GMT</pubDate>
            <atom:updated>2020-08-01T16:05:42.051Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*24ML0sLmHdTwXZQRGT_VMQ.jpeg" /></figure><p>Git is an essential tool to work on any project, it’s enables collaborative work and keeps your work organized.</p><p>I want to show you commands I got to know them recently that saved me a lot.</p><h4>Squash Commits Into 1 Commit</h4><p>Who doesn’t write a long series of commits sometimes? Imagine you are writing docs after committing your changes to your PR, you realized that you miss spelled a word then you will have to make another commit for only this change, to be honest, I made two non-sense commits on spelling a word before. You will find yourself having a lot of commits some goods and others like my 2 non-sense commits. And you want to organize your PR to have one concise commit with a good message.</p><p>Here is <strong><em>git rebase -i HEAD~n</em></strong> came to rescue. Its squash the top n commits down into one.</p><p>Here we create a simple project and added the README file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*JRpRKNjQI3gdBYWftruu8Q.png" /></figure><p>And still, keep adding commits to the README</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*waQWutyV3eBsy4fPEHfv9Q.png" /></figure><p>Then we used the magical command to squash the top three commits</p><pre>git rebase -i HEAD~3</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*At5c3Xkvm1T9LTmqWXuVAQ.png" /></figure><p>This window will popup showing the commits will be affected with other options. If we want to include the commit in the rebase or the squashing operation we share to change <strong>pick </strong>word in the front of the commit to <strong>squash </strong>or <strong>s</strong>. Then save the file, I’m using vim here so my I’ll press <strong>ESC </strong>and then <strong>:wq</strong>.</p><ul><li><em>You should leave the first commit with </em><strong><em>pick</em></strong><em> option because the commits will be squashed into it.</em></li></ul><p>And ta-da the commits squashed</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*g-vsqnF_-8NVSKNxawBpew.png" /></figure><p>After that rename the commit message to something appropriate</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*lSPylvDqA8DYXlIRX1TV1g.png" /></figure><h4>Stashing Changes on Patches</h4><p>Let’s imagine you are working on task X, and you wrote a lot of lines to implement that task, so far you want to review what you have done and what changes before committing the changes.</p><pre>git add -p</pre><p>According to the git help docs: <em>Interactively choose hunks of patch between the index and the work tree and add them to the index. This gives the user a chance to review the difference before adding modified contents to the index.</em></p><p>Here I made some changes to main.py file and I can control the patches with the [y, n, q, a, d, e, ?] options</p><pre>y - stage this hunk</pre><pre>n - do not stage this hunk</pre><pre>q - quit; do not stage this hunk or any of the remaining ones</pre><pre>a - stage this hunk and all later hunks in the file</pre><pre>d - do not stage this hunk or any of the later hunks in the file</pre><pre>e - manually edit the current hunk</pre><pre>? - print help</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*avEYQp_IQrMMabsq2pzTCg.png" /></figure><p>To write a solid commit message, it&#39;s very helpful to review your changes before making a commit.</p><p>Git still has a lot of commands that need a deep dive to discover and understand them. You just need to be in that case that lets you search for them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f69e9c973701" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/life-saving-git-commands-i-recently-knew-f69e9c973701">Life-saving Git Commands I Recently Knew</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>