<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by aibunny on Medium]]></title>
        <description><![CDATA[Stories by aibunny on Medium]]></description>
        <link>https://medium.com/@aibunny?source=rss-e4afca7aa6b6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*q4s5mGQAqlIQYGEDkYxkLg.jpeg</url>
            <title>Stories by aibunny on Medium</title>
            <link>https://medium.com/@aibunny?source=rss-e4afca7aa6b6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:26:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aibunny/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Evolution of Trust: From Centralized Control to Blockchain Technology]]></title>
            <link>https://aibunny.medium.com/the-evolution-of-trust-from-centralized-control-to-blockchain-technology-14b1965991fe?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/14b1965991fe</guid>
            <category><![CDATA[decentralization]]></category>
            <category><![CDATA[centralization]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[ethereum]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Wed, 15 Jan 2025 20:41:19 GMT</pubDate>
            <atom:updated>2025-01-21T16:08:21.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OsuCmjQBj8GNncvqYrBvAQ.png" /></figure><h3>The Age of Information Gatekeepers</h3><p>For most of human history, information flow was limited by geographic boundaries. Communication between continents was not just slow, it was often impossible. Fast forward to the digital age, and we find ourselves in a world where information travels at the speed of light. Yet, this transformation came with an unforeseen cost: the centralization of power in the hands of a few key players.</p><p>Today, almost all of our critical information is controlled by centralized authorities:</p><ul><li>Governments managing citizen identities and records</li><li>Banks overseeing financial transactions</li><li>Healthcare providers maintaining medical histories</li><li>Tech giants storing and processing personal data</li></ul><p>This centralization creates a paradox: while we’ve gained unprecedented access to information and services, we’ve become increasingly dependent on these institutions to verify, validate, and secure our data. We trust these organizations not by choice, but by necessity.</p><h3>The Fundamental Role of Trust in Human Networks</h3><p>In the pre-industrial era, village economies operated primarily through personal relationships, with trade constrained by physical proximity. Communities relied on word-of-mouth reputation systems, and those who violated trust could face exile. A system that, while limited in scale, proved highly effective within local communities.</p><p>Entered the Industrial Revolution which brought major changes, introducing formal institutions like banks, paper money backed by precious metals, and letters of credit for international trade. This period also saw the rise of contract law and enforcement, allowing trust to scale through institutional guarantees.</p><p>In this Digital Age, trust mechanisms have transformed further with the emergence of the internet whuch led to electronic payment systems, credit card networks, online banking, and digital identities, and trust now primarily maintained through centralized authorities.</p><h3>The Hidden Costs of “Modern” Financial Trust</h3><p>The hidden costs of “modern” financial trust systems reveal significant inefficiencies despite our digital age advancement. Traditional banking comes with substantial monetary expenses, including international wire transfer fees ranging from $2 to $5, credit card processing fees between 1.5% and 3.5% per transaction, monthly bank account maintenance charges of $10 to $15, ATM withdrawal fees of $2 to $5, and currency exchange markups of 2% to 5%.</p><p>Beyond these direct costs, time delays create another layer of inefficiency, with bank transfers taking 2 to 5 business days, check clearing requiring up to 7 business days, international transfers spanning 3 to 7 business days, and securities settlements typically taking T+2.</p><h3>The Digital Paradox</h3><p>In an age where we can instantly send messages, videos and images worldwide for free and in seconds, why does moving money remain:</p><ul><li>Expensive</li><li>Slow</li><li>Restricted by business hours</li><li>Bound by geographical borders</li><li>Dependent on multiple intermediaries</li></ul><p>This inefficiency stems from:</p><ul><li>Legacy systems maintaining backwards compatibility lol.</li><li>Regulatory compliance costs</li><li>Profit-taking by intermediaries</li><li>Lack of technological innovation</li><li>Resistance to change from established players</li></ul><h3>The Trust Crisis</h3><p>The problem with centralized trust isn’t just about control — it’s about vulnerability and accountability. When we rely on central authorities:</p><ul><li>A single point of failure can compromise millions of users’ data</li><li>Decision-making power concentrates in the hands of few</li><li>Innovation can be stifled by gatekeepers</li><li>Individual privacy becomes increasingly fragile</li><li>Organizations have become “too big to fail” or “too big to jail”.</li></ul><p>This last point is particularly troubling. Major financial institutions and tech giants have grown so large and become so deeply embedded in our economic infrastructure that they operate with virtual impunity. When they fail, taxpayers bear the burden; when they break laws, they often face minimal consequences.</p><pre>******************** Visa and Mastercard situation. ************************</pre><p>This creates a dangerous moral hazard where these institutions can take enormous risks or engage in questionable practices knowing they’re essentially immune to serious consequences, and there’s nothing at least the most of us can do about it.</p><p>Moreover, these institutions aren’t infallible. Financial crises, data breaches, and abuse of power have repeatedly shown the limitations and risks of centralized trust systems.</p><h3>The Blockchain Revolution: Programmatic Trust</h3><p><strong>The blockchain </strong>— a revolutionary response to the trust crisis. What makes blockchain unique is its fundamental reimagining of how trust works in the digital age. Instead of requiring faith in institutions, blockchain creates trust through mathematics, cryptography, and game theory.</p><p><strong>Understanding the First Waves: </strong>Bitcoin and Ethereum</p><p>Before we start on their differences, it’s important to understand what Bitcoin and Ethereum represent in the blockchain space.</p><p>Bitcoin, launched in 2009, emerged as the world’s first decentralized cryptocurrency, designed primarily as a peer-to-peer electronic cash system that operates without intermediaries. Ethereum, introduced in 2015, built upon Bitcoin’s foundation but expanded blockchain’s potential by introducing smart contracts — self-executing agreements that live on the blockchain. These innovations marked two distinct generations of blockchain technology, each serving different purposes in the digital economy.</p><blockquote>Bitcoin is like a spreadsheet and ethereum is like a spreadsheet with macros</blockquote><h3>Bitcoin: The Digital Ledger</h3><p>Bitcoin “the spreadsheet” — a distributed ledger that tracks who owns what. Its primary function is to maintain a single, immutable record of financial transactions. This simplicity is intentional and provides:</p><ul><li>Maximum security for financial transactions</li><li>Clear focus on being digital gold</li><li>Resistance to tampering or manipulation</li></ul><h3>Ethereum: The Programmable Ledger</h3><p>Ethereum “the spreadsheet with macros”— in the form of smart contracts. This transforms the blockchain from a simple spreadsheet into a programmable computing platform. Smart contracts are:</p><ul><li>Self-executing programs that live on the blockchain</li><li>Capable of automatically enforcing agreements</li><li>Able to create complex financial instruments and applications</li><li>The foundation for decentralized applications (dApps)</li></ul><p>This programmability enables:</p><ul><li>Decentralized Finance (DeFi) protocols</li><li>Non-Fungible Tokens (NFTs)</li><li>Decentralized Autonomous Organizations (DAOs)</li><li>Complex business logic without intermediaries</li></ul><h3>Why This Matters</h3><p>The shift from centralized to decentralized trust systems represents more than just technological innovation — it’s a fundamental change in how society can organize itself. Blockchain technology offers:</p><ul><li>Transparency without compromising security</li><li>Trust without central authorities</li><li>Ownership of digital assets without intermediaries</li><li>Innovation without gatekeepers</li><li>Protection against the “too big to fail” paradigm</li></ul><p>As we continue to build on these foundations, we’re not just creating new technologies — we’re reimagining the very nature of trust and organizational structure in the digital age. The combination of Bitcoin’s proven security model and Ethereum’s programmable flexibility provides a robust foundation for a more equitable and efficient digital future.</p><h3>Technical Deep Dive: Bitcoin vs Ethereum Architecture</h3><h4><strong><em>Transaction Models: UTXO vs Account-Based</em></strong></h4><p><strong><em>Bitcoin’s UTXO Model</em></strong></p><p>Bitcoin uses the Unspent Transaction Output (UTXO) model, which functions similar to physical cash. Here’s how it works, <strong>When</strong> you receive Bitcoin, you don’t actually get a “balance” in your wallet. Instead, you receive UTXOs — think of them as individual bills or coins. Each UTXO is like a dollar bill that can only be spent once and must be spent in its entirety.</p><p>For example:</p><ul><li>You receive 1 BTC (one UTXO)</li><li>To spend 0.3 BTC, you must “break” the 1 BTC UTXO</li><li>The transaction creates two new UTXOs:</li><li>0.3 BTC sent to the recipient</li><li>0.7 BTC returned to you as change</li></ul><p>Advantages of UTXOs:</p><ul><li>Better privacy as each UTXO is unique</li><li>Easier parallel processing of transactions</li><li>Simpler to verify transaction validity</li><li>Natural prevention of double-spending</li></ul><p>Limitations:</p><ul><li>More complex to program</li><li>Stateless nature makes smart contracts harder</li><li>Can lead to larger transaction sizes</li></ul><p><strong><em>Ethereum’s Account Model</em></strong></p><p>Ethereum uses an account-based model, similar to a traditional bank account. Each account maintains:</p><ul><li>A balance (for ETH)</li><li>A nonce (transaction counter)</li><li>Optional contract code</li><li>Optional contract storage</li></ul><p>This model enables:</p><ul><li>Simpler balance tracking</li><li>Easier implementation of complex smart contracts</li><li>More intuitive programming model</li><li>Direct state management</li></ul><h3>State Management</h3><p><strong><em>Bitcoin’s State</em></strong></p><p>Bitcoin maintains a set of UTXOs, representing all spendable “coins” in the system. The state transition is simple:</p><ol><li>Transaction inputs must reference valid UTXOs</li><li>UTXOs are consumed (destroyed) when spent</li><li>New UTXOs are created as outputs</li><li>The sum of outputs must not exceed inputs</li></ol><p>This simplicity contributes to Bitcoin’s security but limits functionality.</p><p><strong><em>Ethereum’s State</em></strong></p><p>Ethereum maintains a more complex state machine:</p><ul><li>Global state trie storing all account states</li><li>Transaction trie for each block</li><li>Receipt trie for transaction outcomes</li><li>Storage trie for contract data</li></ul><p>This enables:</p><ul><li>Complex smart contract interactions</li><li>Stateful programming</li><li>Rich application development</li><li>More sophisticated state transitions</li></ul><h3>Memory and Storage</h3><h3>Bitcoin’s Simple Storage</h3><p>Bitcoin nodes store:</p><ul><li>The UTXO set</li><li>Block headers</li><li>Transaction data</li><li>Optional indexes</li></ul><p>The system is optimized for:</p><ul><li>Minimal state bloat</li><li>Efficient transaction verification</li><li>Simple pruning capabilities</li></ul><h3>Ethereum’s Rich Storage Model</h3><p>Ethereum’s storage is more complex:</p><ul><li>Account state (balances, nonces)</li><li>Contract code</li><li>Contract storage</li><li>Event logs</li><li>State snapshots</li></ul><p>Storage features:</p><ul><li>Merkle Patricia Tries for efficient proofs</li><li>Account abstraction</li><li>Rich statefulness for smart contracts</li></ul><h3>Scripting and Smart Contracts</h3><h3>Bitcoin Script</h3><p>Bitcoin uses a simple, stack-based scripting language:</p><ul><li>Intentionally non-Turing complete</li><li>Limited set of operations</li><li>Focus on transaction validation</li><li>No loops or complex control flow</li></ul><p>Example Bitcoin Script operations:</p><pre>OP_DUP<br>OP_HASH160<br>&lt;pubKeyHash&gt;<br>OP_EQUALVERIFY<br>OP_CHECKSIG</pre><h3>Ethereum Virtual Machine (EVM)</h3><p>Ethereum provides a full Turing-complete environment:</p><ul><li>Rich instruction set</li><li>Complex control flow</li><li>Contract-to-contract calls</li><li>Persistent storage</li><li>Event emission</li></ul><p>Example Solidity contract:</p><pre>contract SimpleStorage {<br>    uint256 private value;  <br><br>  function store(uint256 _value) public {<br>        value = _value;<br>    }<br>  function retrieve() public view returns (uint256) {<br>        return value;<br>    }<br>}</pre><h3>Network Architecture</h3><p><strong><em>Bitcoin’s Network</em></strong></p><ul><li>Focused on financial transactions</li><li>Limited script functionality</li><li>Highly optimized for security</li><li>Conservative upgrade path</li></ul><p><strong><em>Ethereum’s Network</em></strong></p><ul><li>General-purpose computation platform</li><li>Rich smart contract environment</li><li>Faster evolution and upgrades</li><li>More complex consensus rules</li></ul><p>These architectural differences reflect the core philosophies of each project:</p><ul><li>Bitcoin: Digital gold, value storage, financial sovereignty</li><li>Ethereum: World computer, programmable value, decentralized applications</li></ul><h3>Consensus Mechanisms and Scaling Solutions</h3><h3>Consensus Mechanisms</h3><p><strong><em>Bitcoin’s Proof of Work (PoW)</em></strong></p><figure><img alt="Inside a Bitcoin mining factory Photo: AFP" src="https://cdn-images-1.medium.com/max/1024/0*75r5e-BjNzGAo7k9.jpeg" /><figcaption><a href="https://www.globaltimes.cn/page/202106/1226598.shtml">Inside a Bitcoin mining factory Photo: AFP</a></figcaption></figure><p>Bitcoin’s Proof of Work (PoW) establishes trust through computational work and energy expenditure. Miners compete to solve complex mathematical puzzles, requiring significant computing power and electricity, with the winner earning the right to add the next block and receive block rewards. This system makes it prohibitively expensive to attack the network, as an attacker would need to control 51% of the network’s total hash power. The immutable chain of blocks, each cryptographically linked to its predecessor and secured by this computational work, creates a trustless system where the longest chain with the most accumulated work represents the agreed-upon truth, making Bitcoin’s history practically impossible to alter without massive resource investment.</p><p><em>Key Properties:</em></p><ul><li>Energy-intensive by design</li><li>Highly secure against 51% attacks</li><li>No native staking or delegation</li><li>Clear economic incentives through block rewards</li></ul><p><em>Security Considerations:</em></p><ul><li>Cost of attack scales with network hashpower</li><li>Currently requires billions in hardware investment to attack</li><li>Naturally resistant to Sybil attacks</li><li>Incentivizes geographical distribution</li></ul><p><strong><em>Ethereum’s Proof of Stake (PoS)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*ZK9i7BOsdkUGA0aV.jpg" /><figcaption>Validation can be done on any decent computer.</figcaption></figure><p>It establishes trust through economic security and decentralized validation. Validators must stake 32 ETH as collateral, putting their own assets at risk of slashing if they act maliciously, which creates strong financial incentives for honest behavior. The system randomly selects validators to propose and attest to blocks, making it extremely costly and impractical to attack the network since an attacker would need to control a large portion of staked ETH. Further enhancing trust, the protocol requires multiple validators to attest to each block’s validity, creating a robust consensus mechanism where validators must reach agreement on the chain’s state, while the high cost of entry and slashing penalties help maintain network integrity and security.</p><p><em>Key Properties:</em></p><ul><li>Energy efficient</li><li>Economic security through staked assets</li><li>Complex game theory incentives</li><li>Native yield generation through staking</li></ul><p><em>Security Considerations</em>:</p><ul><li>Nothing-at-stake problem solved through slashing</li><li>Long-range attack prevention through checkpoints</li><li>Social coordination for contentious forks</li><li>Increased centralization risks through staking pools</li></ul><h3>Scaling Solutions</h3><h3>Bitcoin’s Scaling Approach</h3><p><strong><em>Layer 1 (Base Chain):</em></strong></p><ul><li>Conservative block size (1MB with Segwit adjustment)</li><li>Focused on decentralization and security</li><li>Limited transaction throughput (~7 TPS)</li><li>Relatively high fees during congestion</li></ul><p><strong><em>Layer 2 Solutions:</em></strong></p><ol><li>Lightning Network:</li></ol><ul><li>Payment channel network</li><li>Near-instant transactions</li><li>Microscopic fees</li><li>Complex channel management</li></ul><p>2. Sidechains:</p><ul><li>Liquid Network</li><li>RSK for smart contracts</li><li>Two-way pegs</li><li>Federation-based security</li></ul><h3>Ethereum’s Multi-Pronged Scaling Strategy</h3><p><strong><em>Layer 1 (Base Chain):</em></strong></p><ul><li>Variable block size through gas limits</li><li>EIP-1559 fee mechanism</li><li>Account abstraction</li><li>Higher throughput (~15–30 TPS)</li></ul><p><strong><em>Layer 2 Solutions:</em></strong></p><ol><li>Rollups:</li></ol><ul><li>Optimistic Rollups (Optimism, Arbitrum)</li><li>Fraud proof-based security</li><li>Week-long withdrawal period</li><li>High compatibility with existing tools</li><li>ZK Rollups (zkSync, StarkNet)</li><li>Zero-knowledge proof security</li><li>Fast finality</li><li>More complex programming model</li></ul><p><strong><em>2. State Channels:</em></strong></p><ul><li>Similar to Lightning Network</li><li>Specialized for specific applications</li><li>Limited adoption compared to rollups</li></ul><p><strong>3. Plasma Chains:</strong></p><ul><li>Child chains with independent consensus</li><li>Complex exit mechanisms</li><li>Largely superseded by rollups</li></ul><h3>Future Scaling Roadmap:</h3><ol><li><a href="https://ethereum.org/en/roadmap/danksharding/"><strong><em>Proto-danksharding:</em></strong></a></li></ol><ul><li>Blob transactions for rollup data</li><li>Reduced rollup costs</li><li>Maintained decentralization</li></ul><p><strong><em>2.Full Sharding:</em></strong></p><ul><li>Data availability sharding</li><li>Parallel transaction processing</li><li>Theoretical throughput of 100,000+ TPS</li></ul><h3>Comparative Analysis</h3><p>The philosophical differences between Bitcoin and Ethereum are reflected in their fundamental approaches to development and scaling. Bitcoin adheres to a “security first, scaling second” mindset, maintaining a conservative approach to changes while focusing intensely on base layer security and encouraging ecosystem innovation primarily through Layer 2 solutions.</p><p>In contrast, Ethereum embraces a “pragmatic scaling with security” philosophy, featuring active protocol development, a rich Layer 2 ecosystem, and a strong emphasis on developer experience. These distinct approaches to consensus and scaling mirror each network’s broader objectives: Bitcoin, positioning itself as digital gold, prioritizes maximum security and stability, while Ethereum, aiming to be a world computer, requires continuous evolution and scaling capabilities.</p><p>The coexistence and success of these divergent approaches offer a crucial lesson in blockchain development: there is no universal solution to the challenges of decentralized networks. Each platform’s unique vision demands its own carefully tailored balance of security, scalability, and functionality. As the blockchain ecosystem continues to evolve to even more complex solution like <a href="https://ieeexplore.ieee.org/abstract/document/9047484">Block DAG</a>, this diversity of approaches may prove to be one of the industry’s greatest strengths, fostering innovation while ensuring that different use cases can find their optimal technical foundation.</p><h3>Next Chapter in Trust</h3><p>The transformation of trust from personal to institutional to programmatic represents one of humanity’s most significant evolutionary leaps. For millennia, we relied on knowing our trading partners personally. Then for centuries, we trusted institutions to verify and secure our interactions. Now, blockchain technology offers something unprecedented: the ability to trust mathematics and code instead of people or institutions. Through mechanisms like Proof of Work and Proof of Stake, we’ve created systems where trust isn’t given — it’s guaranteed through cryptography, game theory, and economic incentives. This isn’t just a technological revolution; it’s a fundamental upgrade to how humans can cooperate and organize at scale. As we continue to build on these foundations, we’re not just creating new financial tools or decentralized applications. We’re rewriting the basic infrastructure of trust itself, moving from “trust us because we say so” to “trust this because we can prove it.” or better “Don’t trust verify!” In this new paradigm, trust isn’t about faith in people or institutions — it’s about verifiable certainty in open, transparent systems that anyone can inspect and everyone can trust.</p><p>“on chain verifiable trust”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14b1965991fe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Columnar Databases Are the OGs in Analytical Tasks]]></title>
            <link>https://aibunny.medium.com/why-columnar-databases-are-the-ogs-in-analytical-tasks-e5925a9e847a?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5925a9e847a</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Fri, 05 Apr 2024 11:51:36 GMT</pubDate>
            <atom:updated>2024-04-05T11:51:36.667Z</atom:updated>
            <content:encoded><![CDATA[<p>Working as an analytical engineer in fintech, I’m faced with the challenge of building models and using data engineering architectures that allow real time analytics. To achieve all this I have to use use the best database available that can support stream processing and with the least query time possible even for the most complex queries. Through out my research I have come across multiple real time analytical databases such as clickhouse, kinetica, rockset, parade db and hydra but one thing stands out for all of them and is that they’re all columnar databases.</p><p>This begs the question, what are columnar databases? and how are they different from traditional row based databases such as MYSQL and “Just Postgres” (I’ll come back to this later on in one of blogs).</p><h3>The Row vs. Column Dilemma: Understanding the Divide.</h3><p>To actually show the technical difference in this two database types let’s get to understand of how row based databases work.</p><p><strong>Row-oriented databases</strong> : this kind of databases store data row by row, with each row containing all the fields for a single record, similar to how data is arranged in a spreadsheet. Each row represents a single record or entry, and each column represents a specific attribute or field of that record.</p><p>Look at this spread sheet with data about three books.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*RWWO6okItZPRLVjwKXhYXA.png" /></figure><p>Let’s visualize how data for three books might be stored in a row based db :</p><pre>| book_id | title  | author   | genre   | publication_year | price |<br>|---------|--------|----------|---------|------------------|-------|<br>| 1       | Book A | Author A | Fiction | 2000             | 20.00 |<br>| 2       | Book B | Author B | Romance | 2015             | 15.00 |<br>| 3       | Book C | Author C | Mystery | 2020             | 25.00 |</pre><p><strong>Columnar database: </strong>this databases store data in columns instead of rows, with each column containing all the values for a specific field across multiple records.</p><p>Let’s visualize how data for three books might be stored in a column based db :</p><pre>| book_id | 1       | 2       | 3       |<br>| title   | Book A  | Book B  | Book C  |<br>| author  | Author A| Author B| Author C|<br>| genre   | Fiction | Romance | Mystery |<br>| pub_year| 2000    | 2015    | 2020    |<br>| price   | 20.00   | 15.00   | 25.00   |</pre><p>You will notice that in the table representation, all the values for the book_id column are stored together, followed by all the values for the title column, then the author column, and so on. This is the columnar storage format, where data is stored column by column, rather than row by row as in traditional row-oriented databases.</p><h3><strong>Advantages of this approach.</strong></h3><p><em>1. Efficient Data Storage:</em></p><p>For example, in the table of books, one column stores all the titles, another stores all the authors, and so forth. This structure allows for highly efficient storage and retrieval of data, especially for analytical queries that often access specific columns rather than entire rows.</p><p><em>2. Optimized Query Performance:</em></p><p>Analytical queries typically involve aggregating or analyzing specific attributes across large datasets. Columnar databases excel in this scenario because they can swiftly access and process only the columns relevant to the query. Unlike row-based databases, which may need to scan entire rows even for simple attribute-based queries, columnar databases can quickly skip irrelevant data, leading to significantly faster query performance.</p><p>Example:</p><p>Looking at this SQL query</p><pre><br>SELECT SUM(price)<br>FROM books<br>WHERE publication_year &gt; 2010;</pre><p>Though the syntax is the same in both the row based databases and columnar databases. This is what happens:</p><p>Row-Based Database Query:</p><ul><li>In a row-based database, the query engine needs to scan through the entire rows of the books table to evaluate the condition publication_year &gt; 2010.</li><li>Even though we are only interested in the price column, the database engine must read and process all columns for each row that meets the condition.</li><li>This can result in higher I/O overhead and longer query execution times, especially for large datasets.</li></ul><p>Columnar Database Query:</p><ul><li>In a columnar database, the query engine can efficiently access and process only the price column for instances where publication_year &gt; 2010.</li><li>Since the data is physically stored by column, the database engine can quickly skip over irrelevant columns, such as title, author, genre, etc., during query execution.</li><li>As a result, the query performance is significantly faster compared to the row-based approach, especially for analytical queries involving large datasets.</li></ul><p><em>3. Data Compression and Encoding:</em></p><p>Columnar databases employ sophisticated techniques such as data compression and encoding to further enhance storage efficiency and query performance. By compressing similar data values within a column and using encoding schemes tailored to specific data types, columnar databases minimize storage footprint and accelerate data access. This results in reduced storage costs and faster query execution, making them ideal for handling large volumes of analytical data.</p><p><em>4. Parallel Processing:</em></p><p>Another ace up the sleeve of columnar databases is their ability to leverage parallel processing capabilities. Analytical workloads often involve complex queries that can benefit from parallel execution across multiple CPU cores or distributed nodes. Columnar databases are designed to exploit parallelism efficiently, enabling them to tackle high-concurrency analytical tasks with ease and agility.</p><p><em>5. Scalability and Flexibility:</em></p><p>Scalability is a cornerstone of modern data infrastructure, and columnar databases are built to scale seamlessly with growing data volumes and user demands. Whether it’s scaling vertically by adding more resources to a single server or horizontally by distributing data across multiple nodes in a cluster, columnar databases offer flexibility and adaptability to evolving analytical needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5925a9e847a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on Guide to Data Modeling with dbt and Visualization with Lightdash]]></title>
            <link>https://aibunny.medium.com/hands-on-guide-to-data-modeling-with-dbt-and-visualization-with-lightdash-f6bf3c0a2b36?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6bf3c0a2b36</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[dbt-labs]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Wed, 20 Dec 2023 19:53:38 GMT</pubDate>
            <atom:updated>2023-12-22T05:06:34.848Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JHxcv1meXllDuphaAcAjpA.png" /></figure><p><strong>Introduction</strong></p><p>Welcome to our hands-on guide where we’ll explore the powerful combination of dbt (data build tool) for data modeling and Lightdash for visualization. In this blog, I’ll take you through the process of transforming raw data into meaningful insights using the well-known Microsoft Northwind database.</p><p><strong>Why dbt and Lightdash?</strong></p><p>Data modeling and visualization are crucial components of any data-driven decision-making process. dbt simplifies the data modeling process, allowing you to define transformations and relationships in your data warehouse effortlessly. Lightdash, on the other hand, brings these models to life with an intuitive interface for data exploration and visualization.</p><p><strong>What to Expect</strong></p><p>In this guide, I’ll walk you through setting up your environment, modeling data using dbt, and visualizing the results with Lightdash. Whether you’re a data analyst, engineer, or business user, this hands-on experience will provide you with practical insights into the seamless integration of dbt and Lightdash.</p><h4>Prerequisites</h4><p>Outline the tools and resources needed:</p><ul><li>PostgreSQL database</li><li>Microsoft Northwind database</li><li><a href="https://github.com/aibunny/dbt_northwind">Github</a></li><li>dbt (install instructions)</li><li>Lightdash (install instructions)</li></ul><h3>Section 1: Setup PostgreSQL and Create a Database</h3><p><strong>1.1 Create an Account on Supabase</strong></p><p>Visit <a href="https://supabase.io/">Supabase</a> and sign up for a new account.</p><p>Create a New Project:</p><ul><li>Once logged in, click on the “New Project” button.</li><li>Provide a name for your project give it a few minutes to start the project and proceed.</li></ul><p><strong>1.2 Access the PostgreSQL Database on Supabase</strong></p><p>Navigate to the settings Tab:</p><ul><li>After creating your project, go to the “settings” tab in the left menu then database, here you’ll find the details for your PostgreSQL database (host, user, password, port, dbname).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6oThf9hHtTzfz_9yQTWjoA.png" /></figure><p><strong>1.3 Connect to Supabase PostgreSQL Database Locally</strong></p><p>Install psql:</p><ul><li>Ensure you have PostgreSQL’s command-line tool (psql) installed locally.</li></ul><p>Connect to Supabase PostgreSQL:</p><ul><li>Open your terminal and run the following command:</li></ul><pre>psql -h &lt;YOUR_SUPABASE_DB_HOST&gt; -p &lt;YOUR_SUPABASE_DB_PORT&gt; -d &lt;YOUR_SUPABASE_DB_NAME&gt; -U &lt;YOUR_SUPABASE_DB_USER&gt;</pre><ul><li>Replace &lt;YOUR_SUPABASE_DB_HOST&gt;, &lt;YOUR_SUPABASE_DB_PORT&gt;, &lt;YOUR_SUPABASE_DB_NAME&gt;, and &lt;YOUR_SUPABASE_DB_USER&gt; with the details from your Supabase project.</li><li>Enter your password when prompted.</li></ul><p><strong>1.4 Upload Northwind Database to Supabase</strong></p><p>Download Northwind SQL Dump:</p><ul><li>Download the Northwind SQL dump file from <a href="https://github.com/aibunny/bot/blob/master/northwind.sql">this link</a>.</li></ul><p>Execute SQL Dump:</p><ul><li>Once connected to the Supabase PostgreSQL database, run the following command:</li></ul><pre>\i /path/to/downloaded/northwind.sql</pre><ul><li>Replace /path/to/downloaded/northwind.sql with the actual path where you downloaded the Northwind SQL dump file.</li></ul><h3>Section 2: Setting Up dbt with PostgreSQL and Creating Model Files</h3><p>Now let’s install dbt and initialize a dbt project connected to our above the postgres database above.</p><p><strong>1.1: Create and Activate a Virtual Environment</strong></p><p>Choose a directory for your project and create a virtual environment. Open your terminal and run:</p><pre>mkdir your_project_directory<br>cd your_project_directory<br>python -m venv venv<br>source venv/bin/activate   # On Windows, use `venv\Scripts\activate`</pre><p><strong>1.2 Install dbt</strong></p><p>For this project we will use dbt-core this is dbt’s python library for local development, we’ll also install dbt&#39;s postgres adapterdbt-postgres . Open your teminal and run</p><pre>pip install dbt-core dbt-postgres</pre><p>Verify the installation:</p><pre>dbt --version</pre><p><strong>1.3 Initialize a New dbt Project</strong></p><p>In this folder run dbt init project_name this command creates the necessary structure and configuration files for your dbt project, ensuring a seamless setup for your data transformations.This command creates the necessary structure and configuration files for your dbt project, ensuring a seamless setup for your data transformations.</p><pre>project_name/<br>├── dbt_project.yml<br>├── models/<br>├── seeds/<br>├── sources/<br>├── tests/<br>├── macros/<br>├── analysis/<br>└── docs/</pre><p>For the purpose of this hands on experience let’s name our project `dbt_northwind`</p><pre>dbt init dbt_northwind</pre><p>dbt will start your project and will also prompt you to select a database you would like to use this mainly depends on the adaptors you have installed in your projects environment but since we only have postgres, that will the only one listed from our choices.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*U2V-GqtLMe1xeKsW56GOIQ.png" /></figure><p>enter number one then enter the details for your postgres database</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*zchF1da7HEFjMpWkjwfdew.png" /></figure><p>This step creates the the profiles.yml file in your home directory at ~/.dbt which centralizes dbt connection information. You can view/edit the file by running on linux.</p><pre>nano ~/.dbt/profiles.yml</pre><p>Now let’s run `dbt debug` to confirm that our PSQL connection works</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*FgV5vuwX80IX-Rh3EXbMVA.png" /></figure><h3>Section 3: Creating dbt Models for Northwind Database</h3><p>This step-by-step walkthrough will help you set up meaningful dbt models, enabling powerful analytics on your data. Below, we define models with clear purposes and dimensions for the Northwind database.</p><p><strong>Step 1: Define dbt Models</strong></p><p>In your models directory you’ll find an example direcory with sample models by dbt delete this directory it’s just there for demonstation purposes. , create SQL files for each model. We&#39;ll define four models, each serving a distinct purpose, but before that we’ll create a schema file in the models directory which will instruct dbt on which schema to use.</p><pre>#models/northwind/schema.yml<br>version: 2<br><br>sources:<br>  - name: northwind_data<br>    schema: public<br>    tables:<br>      - name: order_details<br>      - name: customers<br>      - name: products<br>      - name: orders</pre><p>This ensures that your models are part of the specified schema (public) and are well-organized within your dbt project.</p><p>Now let’s define our models</p><ol><li>Customer Orders:</li></ol><ul><li>Purpose: Analyzing customer orders to derive insights into customer purchasing behavior.</li><li>SQL Model:</li></ul><pre>-- models/northwind/customer_orders.sql<br>WITH customer_orders AS (<br>  SELECT<br>    o.order_id,<br>    c.customer_id,<br>    c.company_name,<br>    COUNT(od.product_id) AS num_products_ordered,<br>    SUM(od.unit_price * od.quantity * (1 - od.discount)) AS total_order_amount<br>  FROM<br>    {{ source(&#39;northwind_data&#39;, &#39;orders&#39;) }} o<br>    JOIN {{ source(&#39;northwind_data&#39;, &#39;customers&#39;) }} c ON o.customer_id = c.customer_id<br>    LEFT JOIN {{ source(&#39;northwind_data&#39;, &#39;order_details&#39;) }} od ON o.order_id = od.order_id<br>  GROUP BY<br>    o.order_id, c.customer_id, c.company_name<br>)<br><br>SELECT *<br>FROM customer_orders;</pre><p>2. Customer Segmentation:</p><ul><li>Purpose: Segmenting customers based on their order count and total spend for targeted marketing strategies.</li><li>SQL Model:</li></ul><pre>-- models/northwind/customer_segmentation.sql<br>WITH customer_segmentation AS (<br>  SELECT<br>    c.customer_id,<br>    c.company_name,<br>    COUNT(o.order_id) AS num_orders,<br>    SUM(od.unit_price * od.quantity * (1 - od.discount)) AS total_spent<br>  FROM<br>    {{ source(&#39;northwind_data&#39;, &#39;customers&#39;) }} c<br>    LEFT JOIN {{ source(&#39;northwind_data&#39;, &#39;orders&#39;) }} o ON c.customer_id = o.customer_id<br>    LEFT JOIN {{ source(&#39;northwind_data&#39;, &#39;order_details&#39;) }} od ON o.order_id = od.order_id<br>  GROUP BY<br>    c.customer_id, c.company_name<br>)<br><br>SELECT *,<br>  CASE<br>    WHEN num_orders &gt;= 10 AND total_spent &gt;= 1000 THEN &#39;High Value&#39;<br>    WHEN num_orders &gt;= 5 THEN &#39;Mid Value&#39;<br>    ELSE &#39;Low Value&#39;<br>  END AS customer_segment<br>FROM customer_segmentation;</pre><p>3. Product Sales:</p><ul><li>Purpose: Analyzing product sales to identify top-performing products and optimize inventory.</li><li>SQL Model:</li></ul><pre>-- models/northwind/product_sales.sql<br>WITH product_sales AS (<br>  SELECT<br>    p.product_id,<br>    p.product_name,<br>    COUNT(od.order_id) AS num_orders,<br>    SUM(od.unit_price * od.quantity * (1 - od.discount)) AS total_sales<br>  FROM<br>    {{ source(&#39;northwind_data&#39;, &#39;products&#39;) }} p<br>    LEFT JOIN {{ source(&#39;northwind_data&#39;, &#39;order_details&#39;) }} od ON p.product_id = od.product_id<br>  GROUP BY<br>    p.product_id, p.product_name<br>)<br><br>SELECT *<br>FROM product_sales;</pre><p>4. Revenue by Region:</p><ul><li>Purpose: Understanding revenue distribution across different shipping regions.</li><li>SQL Model:</li></ul><pre>-- models/northwind/revenue_by_region.sql<br>WITH revenue_by_region AS (<br>  SELECT<br>    o.ship_region,<br>    COUNT(o.order_id) AS num_orders,<br>    SUM(od.unit_price * od.quantity * (1 - od.discount)) AS total_sales<br>  FROM<br>    {{ source(&#39;northwind_data&#39;, &#39;orders&#39;) }} o<br>    LEFT JOIN {{ source(&#39;northwind_data&#39;, &#39;order_details&#39;) }} od ON o.order_id = od.order_id<br>  GROUP BY<br>    o.ship_region<br>)<br><br>SELECT *<br>FROM revenue_by_region;</pre><p><strong>Step 2: Define Dimensions for our models</strong></p><p>Create a models.yml file to define dimensions for each model, providing context and enhancing the data modeling process.</p><pre># models/northwind/models.yml<br>version: 2<br><br>models:<br>  - name: customer_orders<br>    columns:<br>      - name: customer_id<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Customer ID&#39;<br>            description: &#39;The unique identifier for a customer.&#39;<br>      - name: company_name<br>        meta:<br>          dimension:<br>            type: string<br>            label: &#39;Company Name&#39;<br>            description: &#39;The name of the customer company.&#39;<br>      - name: num_products_ordered<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Number of Products Ordered&#39;<br>            description: &#39;The total number of products ordered.&#39;<br>      - name: total_order_amount<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Total Order Amount&#39;<br>            description: &#39;The total amount of the order.&#39;<br><br>  - name: customer_segmentation<br>    columns:<br>      - name: customer_id<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Customer ID&#39;<br>            description: &#39;The unique identifier for a customer.&#39;<br>      - name: company_name<br>        meta:<br>          dimension:<br>            type: string<br>            label: &#39;Company Name&#39;<br>            description: &#39;The name of the customer company.&#39;<br>      - name: num_orders<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Number of Orders&#39;<br>            description: &#39;The total number of orders for the customer.&#39;<br>      - name: total_spent<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Total Spent&#39;<br>            description: &#39;The total amount spent by the customer.&#39;<br>      - name: customer_segment<br>        meta:<br>          dimension:<br>            type: string<br>            label: &#39;Customer Segment&#39;<br>            description: &#39;Segmentation based on order count and total spend.&#39;<br><br>  - name: product_sales<br>    columns:<br>      - name: product_name<br>        meta:<br>          dimension:<br>            type: string<br>            label: &#39;Product Name&#39;<br>            description: &#39;The name of the product.&#39;<br>      - name: num_orders<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Number of Orders&#39;<br>            description: &#39;The total number of orders for the product.&#39;<br>      - name: total_sales<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Total Sales&#39;<br>            description: &#39;The total sales amount for the product.&#39;<br><br>  - name: revenue_by_region<br>    columns:<br>      - name: ship_region<br>        meta:<br>          dimension:<br>            type: string<br>            label: &#39;Ship Region&#39;<br>            description: &#39;The region to which the order is shipped.&#39;<br>      - name: num_orders<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Number of Orders&#39;<br>            description: &#39;The total number of orders for the ship region.&#39;<br>      - name: total_sales<br>        meta:<br>          dimension:<br>            type: number<br>            label: &#39;Total Sales&#39;<br>            description: &#39;The total sales amount for the ship region.&#39;</pre><p>Now, with defined models, dimensions, and schema, you’re ready to run dbt and explore the analytical possibilities within the Northwind database.</p><p>Let’s now run dbt create the views on our database using dbt run</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*reE1lCIjOprKd6d2R4bRkQ.png" /></figure><p>since the project is running successfully now we can got to the next step.</p><h3>Section 4: Upload Your dbt Project to GitHub</h3><p>To enhance collaboration and facilitate seamless integration with Lightdash, we encourage you to upload your dbt project to GitHub. By doing so, you&#39;ll leverage version control, enable collaborative development, and ensure a smooth transition to Lightdash for impactful data visualization.</p><p>Follow these simple steps:</p><ol><li>Create a GitHub Repository:</li></ol><ul><li>Head to <a href="https://github.com/">GitHub</a> and create a new repository for your dbt project.</li></ul><p>2. Upload Your dbt Project:</p><ul><li>Push your dbt project files, including the dbt_project.yml, models, and any associated configurations, to your GitHub repository.</li></ul><h3>Section 5: Connect Lightdash to your project</h3><p>Now that your dbt project is securely hosted on GitHub, let’s seamlessly integrate it with Lightdash Cloud for powerful and interactive data visualization. Follow these steps:</p><ol><li>Sign Up for Lightdash Cloud:</li></ol><ul><li>Head to <a href="https://www.lightdash.com/">Lightdash Cloud </a>and sign up for a Lightdash account.</li></ul><p>Once you’ve signed up select postgres on the get started page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*4RCfIbzD0uMzt5l4plJwAQ.png" /></figure><p>Then select manually so you can upload your github project from github</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*QB1f09tP9Q4QNhOdqfKVFw.png" /></figure><p>Choose I’ve define them on the next screen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*3nwABKwAV_o3rflovItyVQ.png" /></figure><p>On this screen fill in with your psql details and ensure you disable the ssl mode this is only applicable for this project since we did not set up any ssl certificate for our database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ov_6sMDIzoUvAXkHr6BPSw.png" /></figure><p>Now connect to your GitHub repository that contains the dbt project we did above by providing the necessary permissions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_r_j3BPWMhxxJw27rZrqLw.png" /></figure><p>Finally click on test and compile button. If your project has compiled successfully you’ll and on this page where you can see you model tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FIFFQTAzm5DBrn_SDX2jIg.png" /></figure><p>Once you click on one table eg; product sale then select all dimension and run a query by clicking on the `run query`</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Pye4_eA-sPwr8Fid2MfTw.png" /></figure><p>You can be able to see all the results and also a sample chart by clicking the dropdown on the chart below are sample results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZw39SxgEFDCV7R7ExHw_w.png" /></figure><h3>Conclusion: Elevate Your Data Modeling and Visualization with dbt and Lightdash</h3><p>Congratulations on mastering the art of data modeling with dbt and creating stunning visualizations with Lightdash! As we wrap up this journey, here’s a quick recap and some insights for the road ahead:</p><p>Recap:</p><ul><li>You began by setting up a PostgreSQL database using Supabase, effortlessly importing the Microsoft Northwind database to kickstart your modeling journey.</li><li>The installation and initialization of dbt paved the way for an organized project structure, complete with models, seeds, sources, and tests.</li><li>We delved into creating meaningful models, each designed to uncover valuable insights — from customer segmentation to product sales and revenue by region.</li></ul><p>Next Steps:</p><ul><li>Leverage the flexibility of dbt to connect to other databases effortlessly. Whether it’s BigQuery, Snowflake, or Redshift, dbt’s adaptability makes cross-database modeling a breeze.</li><li>Lightdash opens up a world of possibilities for creating compelling visuals. Dive deeper into the rich feature set of Lightdash to enhance your analytics game. Explore advanced functionalities such as custom metrics, filters, and drill-downs for more nuanced insights.</li></ul><p>Powerful Collaboration:</p><ul><li>Hosting your dbt project on GitHub ensures seamless collaboration and version control. Lightdash Cloud’s integration with GitHub allows for effortless synchronization, enabling you and your team to stay on top of the latest changes.</li><li>Collaborate with fellow data enthusiasts, share insights, and iterate on models to continually refine your data strategy.</li></ul><p>Nerdy Thrills Await:</p><ul><li>Dive into the world of advanced analytics with dbt and Lightdash. Leverage dbt’s ability to create powerful transformations, and let Lightdash unleash the full potential of your data through interactive dashboards.</li><li>Embrace the nerdiness of fine-tuning SQL queries, optimizing data models, and crafting visuals that tell compelling data stories.</li></ul><p>Stay Curious, Stay Data-Driven: Your journey into the realms of dbt and Lightdash has just begun. Stay curious, explore new datasets, experiment with advanced features, and continue refining your skills. As the data landscape evolves, your ability to adapt and innovate will set you apart.</p><p>checkout this project on <a href="https://github.com/aibunny/dbt_northwind">github</a></p><p>Happy Modeling and Visualizing! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6bf3c0a2b36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Storing Media and Static Files in an Amazon S3 Bucket with Django]]></title>
            <link>https://awstip.com/storing-media-and-static-files-in-an-amazon-s3-bucket-with-django-a42db08736f2?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/a42db08736f2</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Sun, 10 Dec 2023 07:22:57 GMT</pubDate>
            <atom:updated>2023-12-11T13:42:32.432Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction</h4><p>In modern web development, the efficient management of static and media files is crucial for a seamless user experience. Amazon S3 (Simple Storage Service) provides a scalable and cost-effective solution for storing these files, offering a reliable and high-performance storage infrastructure. In this comprehensive guide, we will walk through the entire process, from creating an S3 bucket on AWS to setting permissions and integrating it with a Django project. We’ll cover scenarios for both public and private access, and explore URL pre-signing for added security.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*rulnyD2t1ehhLmS3FLEjBA.png" /></figure><h3>Step 1: Create an S3 Bucket on AWS</h3><ol><li>Log in to AWS Console: Open the <a href="https://aws.amazon.com/">AWS Management Console</a> and log in to your account.</li><li>Change your aws region to where you want your bucket to reside.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*sA9v1Sjffoh1MZJN_e3gcQ.png" /></figure><p>3. Navigate to S3:</p><p>Search for the S3 service from the AWS Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*l7Ttr4R5j-OdR1Jpo7Y8rg.png" /></figure><p>4. Create a New Bucket:</p><ul><li>Click the Create bucket button.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*M6Pz1pGRNvGT7i3JIrNOIg.png" /></figure><ul><li>Enter a unique and meaningful name for your bucket I named mine aibunny1.</li></ul><p>5. Configure Options:</p><ul><li>Skip the Configure options section for now. You can customize these settings based on your needs.</li></ul><h3>Step 2: Set Permissions — Public Access Example</h3><h3>Making Your Bucket Public (Not Recommended for Sensitive Data)</h3><ol><li>Bucket Policy:</li></ol><ul><li>Navigate to your S3 bucket and select the Permissions tab.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*yktgUqUhTM0kIrz2srITqA.png" /></figure><ul><li>Under Bucket policy, add the following policy to make your bucket public (replace aibunny1 with your actual bucket name):</li></ul><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: &quot;*&quot;,<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::aibunny1/*&quot;<br>        }<br>    ]<br>}</pre><p>This policy allows anyone (Principal: &quot;*&quot;) to read (Action: &quot;s3:GetObject&quot;) objects in your bucket.</p><p>2. Access Control List (ACL):</p><ul><li>Under Access control list (ACL), set the List objects permission to Everyone.</li></ul><p>Now, your S3 bucket is configured for public access. Keep in mind the security implications, especially for sensitive data.</p><h3>Step 3: Set Permissions — Private Access Example</h3><h3>Restricting Access to Your Bucket</h3><ol><li>Bucket Policy:</li></ol><ul><li>Navigate to your S3 bucket and select the Permissions tab.</li><li>Add a bucket policy to restrict access (replace aibunny1 with your actual bucket name and your-aws-account-id with your AWS account ID) which is found at by clicking the top right corner of your aws console under your aws username:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*UnIdn8qgWKHAONrzTYoVJQ.jpeg" /></figure><pre>{<br>    &quot;Version&quot;: &quot;2012-10-17&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Effect&quot;: &quot;Deny&quot;,<br>            &quot;Principal&quot;: &quot;*&quot;,<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::aibunny1/*&quot;,<br>            &quot;Condition&quot;: {<br>                &quot;StringNotEquals&quot;: {<br>                    &quot;aws:PrincipalAccount&quot;: &quot;your-aws-account-id&quot;<br>                }<br>            }<br>        }<br>    ]<br>}</pre><p>This policy denies access to anyone except your AWS account (aws:PrincipalAccount).</p><p>2. Access Control List (ACL):</p><ul><li>Under Access control list (ACL), ensure that the List objects permission is set to Owner.</li></ul><p>Now, your S3 bucket is configured for private access, allowing only your AWS account to retrieve objects.</p><h3>Step 4: Integrate with Django</h3><ol><li>Install Required Packages:</li></ol><pre>pip install django-storages boto3</pre><p>2. Configure Django Settings: Update your settings.py with the following configurations:</p><pre># settings.py<br><br>AWS_ACCESS_KEY_ID = &#39;your-access-key-id&#39;<br>AWS_SECRET_ACCESS_KEY = &#39;your-secret-access-key&#39;<br>AWS_STORAGE_BUCKET_NAME = &#39;your-s3-bucket-name&#39;<br>AWS_S3_REGION_NAME = &#39;your-s3-region-name&#39;  # The region you created your bucket e.g., ap-south-1<br><br># Use Amazon S3 for storage for static files.<br>STATIC_URL = f&#39;https://{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com/static/&#39;<br>STATICFILES_STORAGE = &#39;storages.backends.s3boto3.S3Boto3Storage&#39;<br><br># Use Amazon S3 for storage for uploaded media files.<br>DEFAULT_FILE_STORAGE = &#39;storages.backends.s3boto3.S3Boto3Storage&#39;</pre><p>Replace placeholder values with your actual Amazon S3 credentials and bucket information.</p><p>3. Update Model for Media Files: If you have a model for storing media files (e.g., images), ensure it uses the FileField or ImageField:</p><pre># models.py<br><br>from django.db import models<br><br>class User(models.Model):<br>    &#39;&#39;&#39;<br>    the upload to path is the folder in the bucket you want to store your images<br>    &#39;&#39;&#39;<br>    avatar = models.ImageField(upload_to=&#39;avatars/&#39;)<br>    # ... other fields ...</pre><p>4. Apply Migrations: After updating your models, run the following commands:</p><pre>python manage.py makemigrations<br>python manage.py migrate</pre><h3>Conclusion</h3><p>Congratulations! You’ve successfully configured an Amazon S3 bucket, set permissions for both public and private access, and integrated it with a Django project. This setup ensures efficient handling of files, scalability, and reliability, making it suitable for production-grade applications.</p><p>By utilizing Amazon S3, you leverage the power of cloud storage to enhance the performance and scalability of your Django application. Feel free to explore additional features offered by Amazon S3, such as versioning, access control, and lifecycle policies, to tailor the setup to your specific requirements.</p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a42db08736f2" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/storing-media-and-static-files-in-an-amazon-s3-bucket-with-django-a42db08736f2">Storing Media and Static Files in an Amazon S3 Bucket with Django</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Environments Unveiled: My Hands-On Exploration with Pipenv and Pip with Virtualenv at…]]></title>
            <link>https://python.plainenglish.io/python-environments-unveiled-my-hands-on-exploration-with-pipenv-and-pip-with-virtualenv-at-f9ecbc7e515a?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9ecbc7e515a</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[pip]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Wed, 06 Dec 2023 09:43:36 GMT</pubDate>
            <atom:updated>2023-12-21T11:53:14.632Z</atom:updated>
            <content:encoded><![CDATA[<h3>Python Environments Unveiled: My Hands-On Exploration with Pipenv and Pip with Virtualenv at SpaceYaTech</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*q0hq_AdaI3DA-WALbGW2Cg.jpeg" /></figure><h4>Introduction</h4><p>Hey Python enthusiasts! Today, I’m taking you on a personal journey through the practical side of Python environments, as experienced in the day-to-day coding adventures within <a href="https://spaceyatech.com/">SpaceYaTech</a>. Let’s dive into the world of Pipenv and Pip with Virtualenv, keeping things real and relatable.</p><h4>A Python Developer’s Odyssey at <a href="https://spaceyatech.com/">SpaceYaTech</a></h4><p>I embarked on my journey with <a href="https://spaceyatech.com/">SpaceYaTech</a>, a vibrant open organization in Kenya, around August 2022. At that time, the community was young and growing. In the heart of my role at <a href="https://spaceyatech.com/">SpaceYaTech</a>, we initiated a project called <a href="https://github.com/SpaceyaTech/mastori.git">Mastori</a>, inspired by Medium. As the community burgeoned, we set sail on a new venture — the development of the <a href="https://spaceyatech.com/">SpaceYaTech</a> website <a href="https://github.com/SpaceyaTech/SYT-APIs">API</a> backend using Django. This project aimed to showcase who we are and the projects we were passionately working on, emphasizing the need for a robust environment and dependency management system.</p><p>In the early stages of Mastori, we relied on the familiar terrain of Pip and virtual environments. The process was customary: create a virtual environment, activate it, and install dependencies using a requirements.txt file. However, as our projects evolved, so did the challenges.</p><p>Working with multiple laptops, each running a specific Python version, transformed into a logistical puzzle. The simplicity of Pip became a double-edged sword. While effective for smaller projects, managing dependencies across different machines proved to be an intricate dance of version numbers, compatibility checks, and errors.</p><p>Then came a turning point — a git commit introducing Pipenv. This tool promised a more elegant and structured approach. Intrigued by its potential, we decided to give it a shot for our Django project. The journey from Pip to Pipenv was not just a technical transition but an exploration of a more mindful and organized development process.</p><p>Python Environments with Pip:</p><p><em>Setting Up with Pip:</em> Creating a virtual environment with Pip is a straightforward process:</p><pre># Create a virtual environment<br>python -m venv env<br># Activate the virtual environment (Linux)<br>source env/bin/activate<br># Activate the virtual environment (Windows)<br>env\Scripts\activate</pre><p><em>Installing Dependencies:</em> Using Pip to install dependencies from a requirements.txt file:</p><pre># Install required packages<br>pip install -r requirements.txt</pre><p><em>Advantages of Pip:</em></p><ol><li>Simplicity: Pip is widely known and used, making it an accessible choice for developers.</li><li>Extensive Package Repository: With a vast collection of packages, Pip ensures you have access to a broad spectrum of tools.</li></ol><p><em>Disadvantages of Pip:</em></p><ol><li>Dependency Mess: Managing dependencies manually can lead to a cluttered and confusing environment.</li><li>Lack of Deterministic Builds: Pip lacks a clear mechanism for ensuring deterministic builds.</li></ol><p>Introducing Pipenv:</p><p><em>Setting Up with Pipenv:</em> Getting started with Pipenv involves a few extra steps but offers a more holistic approach:</p><pre># Install Pipenv<br>pip install pipenv<br># Create a new virtual environment<br>pipenv shell<br></pre><p><em>Installing Dependencies:</em> Using Pipenv to install dependencies:</p><pre># Install required packages<br>pipenv install</pre><p><em>Advantages of Pipenv:</em></p><ol><li>Dependency Management: Pipenv uses a Pipfile to clearly define project dependencies, reducing the likelihood of conflicts.</li><li>Deterministic Builds: Pipenv generates a Pipfile.lock, ensuring consistent and reproducible builds.</li><li>Streamlined Commands: Pipenv consolidates environment and package management into a single tool.</li></ol><p><em>Disadvantages of Pipenv:</em></p><ol><li>Learning Curve: Beginners might find Pipenv’s additional features a bit overwhelming initially.</li><li>Slower Installation: Pipenv can be slower due to its feature-rich nature.</li><li>Compatibility Issues: Some packages may not be fully compatible with Pipenv.</li></ol><h4>Conclusion</h4><p>Choosing between Pip and Pipenv ultimately depends on the specific needs of your project. For simpler endeavors, Pip may suffice, but as projects grow in complexity and scale, Pipenv provides a more structured and deterministic environment. Join us at <a href="https://spaceyatech.com/community">Space Ya Tech Community</a> to share your experiences and insights.</p><p>Happy coding!</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9ecbc7e515a" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/python-environments-unveiled-my-hands-on-exploration-with-pipenv-and-pip-with-virtualenv-at-f9ecbc7e515a">Python Environments Unveiled: My Hands-On Exploration with Pipenv and Pip with Virtualenv at…</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Fortified Banking API: Security Practices with Django Rest Framework]]></title>
            <link>https://awstip.com/building-a-fortified-banking-api-security-practices-with-django-rest-framework-581b87cd57e2?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/581b87cd57e2</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[django-rest-framework]]></category>
            <category><![CDATA[jwt-token]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Thu, 02 Nov 2023 13:08:51 GMT</pubDate>
            <atom:updated>2023-12-11T13:42:22.316Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LJjHN_BItP6jxAOYgbefGg.png" /><figcaption><a href="https://www.invicti.com/blog/web-security/rest-api-web-service-security/">invicti</a></figcaption></figure><h4><strong>Introduction :</strong></h4><p>In the fast-evolving world of banking and fintech, the stakes are higher than ever. With the shift towards digitalization, security isn’t just a feature; it’s an absolute necessity. As someone deeply invested in the world of software development, I can’t stress enough how paramount it is to implement top-notch security practices in our applications, especially when it comes to handling sensitive financial data. This blog will be your guide, your compass in the realm of secure API development.</p><p>Let’s face it; the consequences of a security breach in a banking or fintech application can be catastrophic. From financial losses to loss of trust, the implications are far-reaching. But fear not, because I’m here to help you navigate this complex landscape and build an ironclad API. Today, we’ll embark on a journey into the world of Django Rest Framework, a robust and versatile framework for building RESTful APIs.</p><p>Our mission? To build a secure banking API that can stand up to the stringent security requirements of the financial sector. We’ll explore every aspect of API security, from user authentication to data encryption, error handling, and everything in between. By the end of this blog, you’ll have the knowledge and the code to fortify your applications, ensuring they meet the high standards required in the banking and fintech industry.</p><p>So, fasten your seatbelts, for this is not just a tutorial, but a transformative experience in crafting APIs with an unwavering focus on security. Let’s dive into the world of secure banking APIs, where every line of code is a shield against potential threats.</p><p>The first step in our journey to build a secure banking API with Django Rest Framework is to set up the project. This foundational phase is critical for laying a strong, secure, and organized groundwork.</p><p><strong>Step 1: Create a Virtual Environment</strong></p><p>Before we dive into the project, it’s essential to create a controlled environment for our development work. I highly recommend using a virtual environment to isolate our project’s dependencies from the system-wide Python packages. This ensures a clean and organized development environment.</p><pre># Create a virtual environment<br>python -m venv secure_bank_api_env<br><br># Activate the virtual environment (on Windows)<br>secure_bank_api_env\Scripts\activate<br><br># Activate the virtual environment (on macOS and Linux)<br>source secure_bank_api_env/bin/activate</pre><p>Now, we’re in a clean slate environment ready to install the necessary packages for our project.</p><p><strong>Step 2: Install Django and Django Rest Framework</strong></p><p>Our choice of framework is Django, a powerful and secure web framework, and Django Rest Framework (DRF), an extension for building RESTful APIs with Django. Let’s install them.</p><pre># Install Django and DRF<br>pip install django djangorestframework</pre><p>Add rest_framework to your INSTALLED_APPS setting.</p><pre>INSTALLED_APPS = [<br>    ...<br>    &#39;rest_framework&#39;,<br>]</pre><p>With Django and DRF in place, we’re ready to start building our secure banking API.</p><p><strong>Step 3: Create a New Django Project and App</strong></p><p>In Django, a project is a collection of settings for an instance of a website, while an app is a web application that does something. Let’s create both.</p><pre># Create a new Django project<br>django-admin startproject secure_bank_api_project<br><br># Navigate to the project directory<br>cd secure_bank_api_project<br><br># Create a new Django app (replace &#39;banking&#39; with your preferred app name)<br>python manage.py startapp banking</pre><p><strong>Step 4: Set Up the Database and Configure Basic Project Settings</strong></p><p>Our secure banking API will rely on a robust database to store and manage financial data. Let’s configure the database settings in our Django project’s settings.py file. You can choose from various database engines, such as PostgreSQL or SQLite, depending on your requirements.</p><p>Here’s an example of configuring the database to use SQLite</p><pre># In secure_bank_api_project/settings.py<br>DATABASES = {<br>    &#39;default&#39;: {<br>        &#39;ENGINE&#39;: &#39;django.db.backends.sqlite3&#39;,<br>        &#39;NAME&#39;: BASE_DIR / &#39;db.sqlite3&#39;,<br>  </pre><p>Additionally, you’ll want to configure other essential project settings, such as security, secret key, and allowed hosts. These settings lay the foundation for secure application behavior.</p><h4>User Authentication .</h4><p>User authentication is the cornerstone of security in any application, especially when it comes to banking and fintech systems. In our journey to build a secure banking API with Django Rest Framework, implementing robust user authentication is a fundamental step. Django provides a built-in authentication system that makes this process straightforward.</p><p><strong>Implementing User Authentication with Django:</strong></p><p>Django’s built-in authentication system takes care of many security concerns for us. To implement user authentication, follow these steps:</p><ol><li><strong>User Model Configuration:</strong></li></ol><p>First, configure the user model in your app’s models.py. You can extend the built-in User model or create a custom user model with additional fields as needed.</p><pre>from django.contrib.auth.models import AbstractUser  <br><br>class CustomUser(AbstractUser):   <br>  # Add custom fields here</pre><p><strong>2. Authentication Views:</strong></p><p>Django provides authentication views that you can use out of the box. To enable them, add the following URLs to your project’s urls.py.</p><pre>#url.py<br><br>from django.contrib.auth import views as auth_views  <br><br>urlpatterns = [<br>     path(&#39;login/&#39;, auth_views.LoginView.as_view(), name=&#39;login&#39;),<br>     path(&#39;logout/&#39;, auth_views.LogoutView.as_view(), name=&#39;logout&#39;),<br> ]</pre><p><strong>3. Strong Password Policies and Secure Storage:</strong></p><p>Strong password policies are vital for protecting user accounts. Django’s authentication system encourages this by default. Passwords are hashed and stored securely in the database, making it extremely difficult for attackers to retrieve the original password.</p><ul><li>Password Hashing: Django uses a strong password hashing algorithm (e.g., PBKDF2 by default) to hash and store passwords in the database. This ensures that even if the database is compromised, an attacker won’t obtain plain-text passwords.</li><li>Password Policies: You can enforce strong password policies by configuring the password validators in your project’s settings.py. These policies may include requirements for password length, complexity, and uniqueness.</li></ul><pre>AUTH_PASSWORD_VALIDATORS = [ <br>  # ... <br>  {    <br>   &#39;NAME&#39;: &#39;django.contrib.auth.password_validation.MinimumLengthValidator&#39;,     <br>  },<br>  { <br>   &#39;NAME&#39;: &#39;django.contrib.auth.password_validation.CommonPasswordValidator&#39;,<br>   }, <br>  {<br>   &#39;NAME&#39;: &#39;django.contrib.auth.password_validation.NumericPasswordValidator&#39;,<br>  },<br>]</pre><p><em>The Advantage of Using Secure Passwords like Bcrypt:</em></p><p>Implementing secure password hashing is crucial, and Django takes care of it. By using strong hashing algorithms like bcrypt, Django ensures that even if the database is compromised, an attacker would have an incredibly hard time cracking the hashed passwords. This offers a robust layer of protection for user accounts.</p><p><em>The Disadvantage of Using Custom Django Authentication:</em></p><p>While Django’s built-in authentication system is robust and secure, some projects might have specific authentication requirements that cannot be fully met by Django’s default behavior. In such cases, custom authentication can become necessary. However, the disadvantage is that custom authentication systems require careful design and thorough security auditing. Any misconfigurations or vulnerabilities in custom implementations can lead to security breaches.</p><h4>Token-Based Authentication and MFA using Middlewares.</h4><p>In this segment of our blog, we will dive deeper into the realm of security by introducing token-based authentication using JWT (JSON Web Tokens) in Django Rest Framework. JWTs provide a powerful and secure way to authenticate users and implement Multi-Factor Authentication (MFA) using middlewares. Let’s explore this important aspect of securing our banking API with code samples.</p><p><strong>Introducing Token-Based Authentication with JWT:</strong></p><p>Token-based authentication using JWTs is a robust approach to ensure secure API access. JWTs are stateless, self-contained tokens that store user information and are signed to prevent tampering. Here’s how to introduce this approach in our Django Rest Framework project:</p><p>Step 1: Install Required Packages:</p><p>First, install the necessary packages for JWT token authentication:</p><pre>pip install djangorestframework-jwt</pre><p>Step 2: Configuration:</p><p>Configure Django Rest Framework and JWT settings in your project’s settings.py file:</p><pre>#settings.py<br><br>INSTALLED_APPS = [<br>    &#39;rest_framework&#39;,<br>    &#39;rest_framework_jwt&#39;,<br>]<br><br># Authentication settings<br>REST_FRAMEWORK = {<br>    &#39;DEFAULT_AUTHENTICATION_CLASSES&#39;: [<br>        &#39;rest_framework_jwt.authentication.JSONWebTokenAuthentication&#39;,<br>    ],<br>}<br># JWT settings<br>from datetime import timedelta<br>JWT_AUTH = {<br>    &#39;JWT_EXPIRATION_DELTA&#39;: timedelta(seconds=3600),  # Set your token expiration time<br>    #....more custom settings<br>}</pre><p>Step 3: Token Generation and Authentication Views:</p><p>Let’s implement token generation and authentication views using DRF’s built-in views.</p><pre># views.py<br>from rest_framework.views import APIView<br>from rest_framework.response import Response<br>from rest_framework import status<br>from rest_framework_jwt.views import obtain_jwt_token<br><br><br>class CustomObtainJWTToken(obtain_jwt_token):<br>    def post(self, request):<br>        response = super().post(request)<br>        if response.status_code == status.HTTP_200_OK:<br>            # Implement MFA logic here before returning the token<br>            if not request.session.get(&#39;mfa_verified&#39;):<br>                response.status_code = status.HTTP_401_UNAUTHORIZED<br>                response.data = {&#39;detail&#39;: &#39;Multi-Factor Authentication required.&#39;}<br>        return response<br><br><br>class SecureBankingView(APIView):<br>    authentication_classes = [JSONWebTokenAuthentication]<br>    permission_classes = [IsAuthenticated]<br>    <br>    def get(self, request):<br>        # Your secure API view</pre><p>The Benefits of Token-Based Authentication in a Banking Context:</p><ol><li>Enhanced Security: JWT-based authentication offers a secure way to access banking APIs. Tokens are digitally signed, making it difficult for attackers to forge them. This adds an extra layer of protection for sensitive financial data.</li><li>Stateless and Scalable: Token-based authentication is stateless, meaning it doesn’t rely on server-side sessions. This makes it highly scalable, essential for banking applications with a large user base and high traffic.</li><li>Fine-Grained Access Control: JWTs can carry user permissions and roles, allowing for precise access control. In a banking context, different users may have varying levels of access to financial data, and JWTs enable this control.</li></ol><p><strong>MFA Using Middleware:</strong></p><p>Multi-Factor Authentication (MFA) is a powerful security measure that we can easily add to our JWT-based authentication using middleware.</p><p>1. Middleware Registration:</p><p>Register a custom middleware class in your project’s settings.</p><pre># In settings.py<br>MIDDLEWARE = [<br>   # ...<br>   &#39;banking.middleware.MultiFactorAuthMiddleware&#39;,<br>]</pre><p>2. Custom Middleware:</p><p>Create a custom middleware class that checks for additional authentication factors and enforces MFA.</p><pre># In middleware.py<br>class MultiFactorAuthMiddleware:<br>   def __init__(self, get_response):<br>       self.get_response = get_response<br>  def __call__(self, request):<br>         if not request.user.is authenticated:<br>             return JsonResponse({&#39;error&#39;: &#39;Authentication required.&#39;}, status=401)<br>         # Implement MFA logic here<br>         if not request.session.get(&#39;mfa_verified&#39;):<br>             return JsonResponse({&#39;error&#39;: &#39;MFA required.&#39;}, status=401)<br>         response = self.get_response(request)<br>         return response</pre><p>By combining JWT-based token authentication and MFA using middleware, we’ve created a formidable security framework for our banking API. Only authorized users with valid tokens and MFA verification can access sensitive financial data, ensuring a secure and trustworthy banking experience.</p><h4>Rate Limiting and Throttling.</h4><p>Rate limiting and throttling are two other essential techniques for safeguarding our API against unauthorized access, abuse, and potential threats.</p><p>Implementing Rate Limiting and Throttling:</p><p>Rate limiting and throttling are mechanisms that control the number and frequency of requests made to your API endpoints. This ensures that clients, including potential attackers, cannot overwhelm your server with a flood of requests. To implement these measures in Django Rest Framework, follow these steps:</p><ol><li>Rate Limiting:</li></ol><p>Rate limiting restricts the number of requests a client can make within a specified time window. You can configure rate limiting classes and set limits based on your application’s requirements. For example, you might want to allow a user to make a maximum of 100 requests per hour.</p><pre><br>REST_FRAMEWORK = {<br>     &#39;DEFAULT_THROTTLE_RATES&#39;: {<br>         &#39;user&#39;: &#39;100/hour&#39;, <br>        &#39;anonymous&#39;: &#39;50/hour&#39;, <br>    } <br>}</pre><p>2. Throttling:</p><p>Throttling, on the other hand, limits the rate of requests on a per-client basis. You can define different throttling classes for different views. For example, you might want to allow more requests to public endpoints compared to highly sensitive financial transactions.</p><pre>from rest_framework.throttling import UserRateThrottle<br><br>class HighSecurityTransactionView(APIView): <br>    throttle_classes = [UserRateThrottle]</pre><p>The Role of Rate Limiting in Protection:</p><p>Rate limiting plays a crucial role in protecting your API against various types of attacks, including:</p><ol><li>Brute Force Attacks:</li></ol><p>In a brute force attack, an attacker attempts to guess a user’s password or access token by making numerous login requests. Rate limiting limits the number of login attempts within a specific time window, making it impractical for attackers to guess credentials through sheer trial and error.</p><p>2. DDoS (Distributed Denial of Service) Attacks:</p><p>DDoS attacks flood your server with an excessive volume of requests, overwhelming it and causing service disruption. Rate limiting can mitigate the impact of DDoS attacks by capping the number of requests a client can make, reducing the potential damage caused by a flood of incoming requests.</p><p>By implementing rate limiting and throttling, we enhance the security of our API, ensuring that even in the face of malicious actors attempting to abuse our resources, our API remains responsive and available to legitimate users. In the ever-evolving landscape of online threats, these defensive mechanisms are indispensable.</p><h4>Data Encryption.</h4><p>Data encryption is a linchpin of security in our banking API, serving as a safeguard both in transit and at rest. The importance of encryption cannot be overstated, especially when dealing with sensitive financial information.</p><p><strong>Importance of Data Encryption:</strong></p><ol><li><strong>In Transit:</strong></li></ol><p>Encrypting data in transit, such as during communication between clients and the server, is crucial to prevent eavesdropping and interception by malicious actors. The use of HTTPS (Hypertext Transfer Protocol Secure) ensures that data exchanged between a client and the server is encrypted, making it extremely challenging for attackers to access or tamper with the information during transmission.</p><p><strong>2. At Rest:</strong></p><p>Data at rest refers to information stored on servers, databases, or other storage devices. Encrypting data at rest safeguards it from unauthorized access in case of a security breach, physical theft, or unauthorized access to the storage infrastructure. This is especially pertinent for databases housing sensitive financial data.</p><p><strong>Implementing HTTPS for Secure Data Transmission:</strong></p><p>To implement secure data transmission using HTTPS, follow these steps:</p><ol><li>SSL/TLS Certificate:</li></ol><p>You need an SSL/TLS certificate to enable HTTPS. You can obtain a certificate from a trusted Certificate Authority (CA), or you can use Let’s Encrypt, which provides free certificates.</p><p>2. Configure Your Web Server:</p><p>Configure your web server (e.g., Apache, Nginx) to use the obtained SSL certificate. Ensure that your server is set up to use HTTPS for all incoming requests. Additionally, you may want to configure HTTP to automatically redirect to HTTPS to ensure secure connections.</p><p>3. Update Your API Endpoints:</p><p>Ensure that all API endpoints are accessible via HTTPS. Update the client-side code to make requests to the secure, HTTPS endpoints.</p><p><strong>Encrypting Sensitive Data in the Database:</strong></p><p>To encrypt sensitive data stored in the database, you can follow these best practices:</p><ol><li>Use Field-Level Encryption:</li></ol><p>Implement field-level encryption for specific database columns that store sensitive information, such as account numbers, personal identification details, and transaction records. You can use libraries like django-encrypted-fields to achieve this in Django.</p><p>2. Key Management:</p><p>Properly manage encryption keys. Keys should be securely stored and accessible only to authorized personnel. Regularly rotate encryption keys to enhance security.</p><p>3. Database Encryption at Rest:</p><p>Many database systems offer encryption at rest features. For example, you can enable Transparent Data Encryption (TDE) in Microsoft SQL Server or use encrypted storage engines in databases like PostgreSQL or MySQL. These features ensure that data stored in the database files is encrypted.</p><p>By encrypting sensitive data at rest, you add an additional layer of protection in case unauthorized access occurs, whether it’s due to a security breach or physical access to the database server.</p><h4>Error Handling and Logging.</h4><p>As we shape our secure banking API, one vital aspect is robust error handling for API responses and the significance of logging for monitoring and incident response. Let’s explore these aspects in-depth and see how to create log files for your Django project.</p><p>1. Implementing Robust Error Handling:</p><p>Error handling is about managing unexpected situations and delivering informative error responses. It’s crucial for user experience and troubleshooting. In Django Rest Framework, you can customize error handling with custom exception handlers and serializers. Here’s how you can create an error response with a custom serializer:</p><pre><br># serializers.py<br>from rest_framework import serializers<br><br>class CustomErrorSerializer(serializers.Serializer):<br>    error_code = serializers.IntegerField()<br>    error_message = serializers.CharField()</pre><pre># views.py<br>from rest_framework.views import APIView<br>from rest_framework.response import Response<br>from . serializer import CustomErrorSerializer<br><br>class SecureBankingView(APIView):<br>    def get(self, request):<br>        try:<br>            # Your secure API logic<br>            result = perform_secure_transaction()<br>            return Response({&#39;result&#39;: result})<br>        except Exception as e:<br>            error_data = {<br>                &#39;error_code&#39;: 500,<br>                &#39;error_message&#39;: &#39;An unexpected error occurred.&#39;<br>            }<br>            error_serializer = CustomErrorSerializer(error_data)<br>            return Response(error_serializer.data, status=500)</pre><p>2. The Importance of Logging for Monitoring and Incident Response:</p><p>Logging is your eyes and ears in the world of application behavior. Here’s why it’s crucial:</p><ul><li>Monitoring: Logs provide insights into your API’s behavior. You can monitor for anomalies and patterns, crucial for early detection of security threats or operational issues.</li><li>Incident Response: In the event of an incident, logs offer a timeline of events, helping you understand the cause and impact. They are essential for root cause analysis, vital for preventing future incidents.</li></ul><p>3. How to Create Log Files in Django:</p><p>Django provides robust logging capabilities. You can configure loggers in your project’s settings.py:</p><pre># settings.py<br><br>LOGGING = {<br>    &#39;version&#39;: 1,<br>    &#39;disable_existing_loggers&#39;: False,<br>    &#39;handlers&#39;: {<br>        &#39;file&#39;: {<br>            &#39;level&#39;: &#39;DEBUG&#39;,<br>            &#39;class&#39;: &#39;logging.FileHandler&#39;,<br>            &#39;filename&#39;: &#39;your_api.log&#39;,<br>        },<br>    },<br>    &#39;loggers&#39;: {<br>        &#39;django&#39;: {<br>            &#39;handlers&#39;: [&#39;file&#39;],<br>            &#39;level&#39;: &#39;DEBUG&#39;,<br>            &#39;propagate&#39;: True,<br>        },<br>    },<br>}</pre><p>In the above example, we create a log file named your_api.log to store log entries. You can then use Python&#39;s built-in logging module to log messages at different severity levels:</p><pre>import logging<br><br>logger = logging.getLogger(__name__)<br><br>def some_function():<br>    try:<br>        # Your code here<br>    except Exception as e:<br>        logger.error(f&quot;An error occurred: {e}&quot;)</pre><p>By configuring loggers and creating log files, you ensure that you have a record of what’s happening in your API, allowing you to respond promptly to issues and maintain a secure and stable banking API.</p><h4>API Versioning.</h4><p>API versioning is a strategic practice in our mission to build a secure banking API with Django Rest Framework. It ensures backward compatibility and a seamless transition when evolving your API. In the dynamic landscape of financial technology, versioning is essential to accommodate changing requirements and to deliver consistent and reliable services.</p><p>Best Practices for API Versioning in a Financial Context:</p><ol><li>Semantic Versioning: Follow semantic versioning (SemVer) principles. Version numbers should consist of major, minor, and patch components, and changes in each version should convey the type of modifications made.</li><li>URL Versioning: Include version information in the API’s URL. For example, use a URL structure like /v1/endpoint/ to signify version 1 of an endpoint.</li><li>Maintain Deprecated Versions: When introducing a new version, continue to support and maintain older versions for a reasonable period. This helps users transition to newer versions without disruption.</li><li>Documentation: Maintain comprehensive documentation to help users understand the differences between API versions and to guide them on the transition from older versions to newer ones.</li><li>Deprecation Notices: Provide clear deprecation notices in your documentation and responses to inform users about upcoming changes or the removal of specific features in future versions.</li><li>Testing: Rigorously test version updates to ensure that existing functionality remains intact. Automated testing is crucial to verify backward compatibility.</li></ol><p>API versioning in the financial sector is more than just a technical concern; it’s a commitment to serving clients and partners with stability, consistency, and trust. By adopting best practices, we ensure that changes to the API are handled with transparency and care, allowing for a smooth and secure transition as the financial landscape continues to evolve.</p><h4>Conclusion and Next Steps.</h4><p>In our journey to create a fortified banking API with Django Rest Framework, we’ve fortified our application with a robust set of security practices that are paramount in the financial sector. Here’s a quick recap of key security practices we’ve implemented:</p><ol><li>User Authentication: We’ve established user authentication using Django’s built-in system, emphasizing strong password policies and secure storage. This is the foundation of user security.</li><li>Token-Based Authentication and MFA: We’ve implemented token-based authentication for API access, coupled with Multi-Factor Authentication (MFA) using middlewares to add an extra layer of security, ensuring only authorized users access sensitive data.</li><li>Rate Limiting and Throttling: We’ve enforced rate limiting and throttling to safeguard our API against abuse, brute force attacks, and DDoS threats.</li><li>Data Encryption: We’ve secured data both in transit and at rest by implementing HTTPS for secure data transmission and encrypting sensitive data in the database.</li><li>Error Handling and Logging: We’ve ensured robust error handling for API responses and emphasized the importance of logging for monitoring and incident response.</li><li>API Versioning: We’ve introduced API versioning to ensure backward compatibility and provided best practices for managing evolving APIs in a financial context.</li></ol><p>As we conclude this journey, it’s important to remember that security is an ongoing process. The landscape of technology and security threats is ever-changing. I encourage you to continue your research and learning, keeping pace with the latest security practices, regulations, and advancements in the financial technology sector.</p><p>Remember that the security of your banking API is not just about code and practices; it’s also about building a culture of security awareness within your team and organization. Together, we can create a banking API that not only meets the high-security standards of the financial industry but also adapts and thrives in an ever-evolving digital world. Thank you for joining me on this journey, and I wish you continued success in your pursuit of secure and innovative banking solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=581b87cd57e2" width="1" height="1" alt=""><hr><p><a href="https://awstip.com/building-a-fortified-banking-api-security-practices-with-django-rest-framework-581b87cd57e2">Building a Fortified Banking API: Security Practices with Django Rest Framework</a> was originally published in <a href="https://awstip.com/">AWS Tip</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlining Business Verification with Smile ID and Django REST: A Celery-Powered Solution]]></title>
            <link>https://python.plainenglish.io/streamlining-business-verification-with-smile-id-and-django-rest-a-celery-powered-solution-340bdc96d998?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/340bdc96d998</guid>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 13:22:43 GMT</pubDate>
            <atom:updated>2023-12-11T02:27:45.292Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4IgNzc_GRen02OFeo-tuw.png" /></figure><h4>Introduction</h4><p>In an age of rapid digital transformation, verifying business details efficiently is paramount for many applications. Smile ID offers a powerful solution for business verification, and by integrating it with your Django REST project, you can streamline your verification process.</p><p>In this blog, we will walk through the steps to integrate Smile ID’s Business Verification with your Django REST project, leveraging the Celery task queue for asynchronous job scheduling. This ensures that your application’s responsiveness remains intact while robust verifications are performed.</p><h3>Preparing for the Adventure</h3><p>Before embarking on this adventure, you’ll need to prepare your environment. Here’s a checklist of what you’ll need:</p><ul><li><strong>Django Project</strong>: Create a DRF project to host your business verification functionality.</li><li><strong>Smile ID Account</strong>: Ensure you have a Smile ID account and obtain the necessary<em> API key</em> and <em>partner ID</em> from the smile ID portal more details can be found at <a href="https://docs.usesmileid.com/integration-options/server-to-server/python/products/business-verification#submit-the-job-to-smile-id">smile ID docs </a>.</li><li><strong>Celery</strong>: Install and configure Celery in your Django project. Checkout how to setup celery in my previous <a href="https://medium.com/@aibunny/django-and-celery-supercharging-your-web-app-with-asynchronous-tasks-7ad989848d08">blog</a></li><li><strong>smile ID python library</strong>: Install the smile id python library for submiting the jobs by running</li></ul><pre>pip install smile_id_core</pre><h3>The Django project</h3><ol><li><strong>Initialize apps</strong></li></ol><p>Our journey begins with defining two Django apps: business and smileid. The former stores Business related code, while the latter represents Smile ID.</p><pre>django-admin startapp business smileid</pre><p><strong>2. The django models</strong></p><p>Define two Django models in the business app: BusinessAccount and BusinessJob. The former stores business details, while the latter represents jobs submitted for verification. The post_save function in the BusinessJobupdates the BusinessAccount table when a new BusinessJob is created, ensuring that the KYB verification progress and associated details are recorded for the business.</p><pre>from django.db import models<br>import uuid<br>from django.db import models<br>from django.db.models.signals import post_save<br>from django.dispatch import receiver<br><br># Create your models here<br>class BusinessAccount(models.Model):<br>    <br>    KYC_STATUS_CHOICES = (<br>        (&quot;Failed&quot;, &quot;Failed&quot;),<br>        (&quot;Pending&quot;, &quot;Pending&quot;),<br>        (&quot;Not Started&quot;, &quot;Not Started&quot;),<br>        (&quot;Completed&quot;, &quot;Completed&quot;),<br>    )<br>    business_id = models.UUIDField(<br>        primary_key=True,<br>        default=uuid.uuid4,<br>        editable=False<br>        )    <br>    legal_name = models.CharField(max_length=100,null=True )<br>    business_tax_id = models.CharField(max_length=20,null=True )<br>    country = models.CharField(max_length=20,default=&#39;Kenya&#39;)<br>    postal_address = models.CharField(max_length=50,null=True)<br>    postal_code = models.CharField(max_length=15)<br>    kyb_verified = models.BooleanField(default=False)<br>    kyb_status = models.CharField(<br>        max_length=20,<br>        choices=KYC_STATUS_CHOICES,<br>        default=&quot;Not Started&quot;<br>        )    <br>    kyc_job = models.ForeignKey(<br>        &#39;BusinessJob&#39;,<br>        on_delete=models.PROTECT,<br>        null=True<br>        )<br>    created_at = models.DateTimeField(auto_now_add=True)<br>    updated_at = models.DateTimeField(auto_now=True)<br><br>    class Meta:<br>        db_table = &#39;Business Accounts&#39;<br>        <br>        <br>class BusinessJob(models.Model):<br>    job_id = models.UUIDField(<br>        primary_key=True,<br>        default=uuid.uuid4,<br>        editable=False<br>        )<br>    country = models.CharField(max_length=5, default=&#39;KE&#39;)<br>    id_type = models.CharField(max_length=25, default=&#39;BUSINESS_REGISTRATION&#39;)<br>    registration_number = models.CharField(max_length=55)<br>    kra_pin = models.CharField(max_length=55)<br>    postal_address = models.CharField(max_length=50)<br>    postal_code = models.CharField(max_length=15)<br>    business_name = models.CharField(max_length=150)<br>    business_id = models.CharField(max_length=150)<br>    <br>    class Meta:<br>        db_table = &#39;kyb_jobs&#39;<br><br>@receiver(post_save, sender = BusinessJob)<br>def fill_business_details_post_save(sender, instance, created, **kwargs):<br>    if created:<br>        try:<br>            business_instance = BusinessAccount.objects.get(<br>                business_id = instance.business_id<br>                )<br>            <br>            business_instance.legal_name = instance.business_name<br>            business_instance.business_tax_id = instance.kra_pin<br>            business_instance.postal_address = instance.postal_address <br>            business_instance.postal_code = instance.postal_code<br>            business_instance.kyc_status = &#39;Pending&#39;<br>            business_instance.kyc_job = instance<br>            <br>            business_instance.save()<br>        except BusinessAccount.DoesNotExist:<br>            pass<br>post_save.connect(fill_business_details_post_save, sender=BusinessJob)</pre><p>Define another model in the smileid app: BusinessVerificationResponse . The models stores business registartion details received from smile ID, It also has a save_response method which is responsible for saving this response in the database. It checks whether the verification was successful based on the ResultCode and ResultText. If the verification is successful, it updates the KYB status of the associated BusinessAccount and saves the response data in the BusinessVerificationResponse model.</p><pre>from django.db import models<br>from business.models import BusinessAccount<br><br><br>class BusinessVerificationResponse(models.Model):<br>    JSONVersion = models.CharField(max_length=10, blank=True, null=True)<br>    SmileJobID = models.CharField(max_length=255, blank=True, null=True)<br>    <br>    # PartnerParams<br>    job_id = models.CharField(max_length=255, blank=True, null=True)<br>    user_id = models.CharField(max_length=255, blank=True, null=True)<br>    job_type = models.IntegerField(null=True)<br>    <br>    ResultType = models.CharField(max_length=100, blank=True, null=True)<br>    ResultText = models.CharField(max_length=255, blank=True, null=True)<br>    ResultCode = models.CharField(max_length=10, blank=True, null=True)<br>    IsFinalResult = models.BooleanField(null=True)<br>    <br>    # Actions<br>    Verify_Business = models.CharField(max_length=255, blank=True, null=True)<br>    Return_Business_Info = models.CharField(max_length=255, blank=True, null=True)<br>    <br>    # company_information<br>    company_type = models.CharField(max_length=255, blank=True, null=True)<br>    country = models.CharField(max_length=100, blank=True, null=True)<br>    address = models.TextField(blank=True, null=True)<br>    registration_number = models.CharField(max_length=100, blank=True, null=True)<br>    search_number = models.CharField(max_length=100, blank=True, null=True)<br>    authorized_shared_capital = models.CharField(max_length=100, blank=True, null=True)<br>    industry = models.CharField(max_length=255, blank=True, null=True)<br>    tax_id = models.CharField(max_length=100, blank=True, null=True)<br>    registration_date = models.DateTimeField(null=True)<br>    phone = models.CharField(max_length=20, blank=True, null=True)<br>    legal_name = models.CharField(max_length=255, blank=True, null=True)<br>    state = models.CharField(max_length=100, blank=True, null=True)<br>    email = models.EmailField(blank=True, null=True)<br>    status = models.CharField(max_length=100, blank=True, null=True)<br>    <br>    # Fiduciaries (as JSON)<br>    fiduciaries = models.JSONField(null=True, blank=True)<br>    <br>    # Proprietors (as JSON)<br>    proprietors = models.JSONField(null=True, blank=True)<br>    <br>    # Documents (as JSON)<br>    documents = models.JSONField(null=True, blank=True)<br>    <br>    # Directors (as JSON)<br>    directors = models.JSONField(null=True, blank=True)<br>    <br>    class Meta:<br>        db_table = &#39;kyb_details&#39;<br>    <br>    @classmethod<br>    def save_response(cls, response_data):<br>        result_code = response_data.get(&#39;ResultCode&#39;)<br>        result_text = response_data.get(&#39;ResultText&#39;)<br>        if result_code == &quot;1012&quot; &amp; result_text == &#39;Business Verified&#39;:<br>            try:<br>                business_id = response_data.get(&#39;user_id&#39;)<br>                business_instance = BusinessAccount.objects.get(<br>                    business_id =business_id<br>                )<br>                business_instance.kyb_verified = True<br>                business_instance.kyb_status = &#39;Completed&#39;<br>                business_instance.save()<br>                                <br>            except BusinessAccount.DoesNotExist:<br>                pass<br>                            <br>            response_instance = cls(**response_data)<br>            response_instance.save()</pre><p><strong>3. The drf serializer</strong></p><p>Before we dive into the KYB view, we need to create a serializer to validate and prepare incoming data. Let’s call it BusinessKYBDetails</p><pre># serializers.py in your business app<br>from rest_framework import serializers<br><br>class BusinessKYBDetails(serializers.Serializer):<br>    &#39;&#39;&#39;kyb details from user&#39;&#39;&#39;<br>    business_name = serializers.CharField(required=True)<br>    registration_number = serializers.CharField(required=True)<br>    kra_pin = serializers.CharField(required=True)<br>    postal_address = serializers.CharField(required=True)<br>    postal_code = serializers.CharField(required=True)<br>    business_id = serializers.CharField(required=True)</pre><p><strong>4. The view</strong></p><p>The next stop is the BusinessKYBView, a Django view that receives user details, saves a verification job, and initiates the KYB process. In this view, we create a new BusinessJob instance and start the KYB task using Celery.</p><pre># views.py in your business app<br><br>from rest_framework import generics ,status<br>from rest_framework.response import Response<br>from rest_framework.permissions import IsAuthenticated<br>from rest_framework.views import APIView<br><br><br>from smileid.tasks import handle_kyb_task<br>from . utils import update_kyb_status<br>from . serializers import *<br>from . models import *<br><br>class BusinessKYBView(APIView):<br>    permission_classes = [IsAuthenticated,]<br><br>    def post(self, request):<br>        <br>        serializer = BusinessKYBDetails(data=request.data)<br><br>        if serializer.is_valid():<br>            # Create and save a BusinessJob instance<br>            business_job, created = BusinessJob.objects.get_or_create(<br>                registration_number=serializer.validated_data[&#39;registration_number&#39;],<br>                postal_address=serializer.validated_data[&#39;postal_address&#39;],<br>                postal_code=serializer.validated_data[&#39;postal_code&#39;],<br>                kra_pin = serializer.validated_data[&#39;kra_pin&#39;],<br>                business_name = serializer.validated_data[&#39;business_name&#39;],<br>                business_id = serializer.validated_data[&#39;business_id&#39;],                <br>            )<br>            <br>            business_job.save()<br><br>            job_id = business_job.job_id<br>            user_id = business_job.business_id <br>            id_number = business_job.registration_number<br>            postal_address = business_job.postal_address<br>            postal_code = business_job.postal_code<br>            business_id = business_job.business_id<br>            <br>            update_kyb_status(<br>                business_id<br>                )#update the business kyb status as &#39;pending&#39;<br>            <br>            handle_kyb_task.apply_async(<br>                (<br>                user_id,<br>                job_id,<br>                id_number,<br>                postal_address,<br>                postal_code<br>                )<br>                )#submit async kyb job<br>        <br>            <br>            return Response({&quot;message&quot;: &quot;KYB started&quot;}, status=status.HTTP_200_OK)<br>        else:<br>            <br>            return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)<br><br></pre><p><strong>5. The util</strong></p><pre># utils.py in your business app<br><br>from . models import BusinessAccount<br><br>def update_kyb_status(business_id : str):<br>    <br>    business_instance = BusinessAccount.objects.get(<br>        business_id = business_id<br>        )<br>    <br>    #set kyb_status == pending    <br>    business_instance.kyb_status = &#39;Pending&#39;<br>    business_instance.save()</pre><p>The update_kyb_status function called in the BusinessKYBView, updates the kyb status for the business as ‘Pending’ while awaiting smile ID response.</p><p><strong>6. The KYB Task: Your Trusty Companion</strong></p><p>Our journey wouldn’t be complete without our trusty companion, the Celery task that handles the KYB verification. The handle_kyb_task function submits a job for business verification, tracks the job’s progress, and updates the status based on the response received from Smile ID. If there are any errors or if the verification fails, it retries the process up to a maximum number of times (as specified by max_retries) before handling the error. If the maximum number of retries is reached, it exits the loop. If the verification is successful, the response is saved. .</p><pre># tasks.py in your smileid app<br><br>import time<br>from celery import shared_task<br>from django.conf import settings<br>from smile_id_core import BusinessVerification<br>from smile_id_core import ServerError<br>#local<br>from . models import BusinessVerificationResponse<br>from business.models import BusinessAccount<br><br><br>@shared_task<br>def handle_kyb_task(user_id, job_id, id_number, postal_address, postal_code):<br>    partner_id = settings.PARTNER_ID<br>    api_key = settings.API_KEY<br>    sid_server = settings.SID_SERVER<br><br>    connection = BusinessVerification(partner_id, api_key, sid_server)<br><br>    # Create partner_params<br>    partner_params = {<br>        &quot;user_id&quot;: str(user_id),<br>        &quot;job_id&quot;: str(job_id),<br>        &quot;job_type&quot;: 7,<br>    }<br><br>    max_retries = 2  # Maximum number of retries<br><br>    for retry_count in range(max_retries + 1):  # Include the initial attempt<br>        # Create id params<br>        id_params = {<br>            &quot;country&quot;: &quot;KE&quot;,<br>            &quot;id_type&quot;: &quot;BUSINESS_REGISTRATION&quot;,<br>            &quot;id_number&quot;: id_number, <br>            &quot;postal_address&quot;: postal_address,    <br>            &quot;postal_code&quot;: postal_code,  <br>        }<br><br>        # Submit the job<br>        try:<br>            response = connection.submit_job(partner_params, id_params)<br>        except ValueError:<br>            # Some params for a job are not valid or missing<br>            try:<br>                business_instance = BusinessAccount.objects.get(business_id=user_id)<br>                business_instance.kyb_status = &#39;Failed&#39;<br>                business_instance.save()<br>            except BusinessAccount.DoesNotExist:<br>                pass<br>            break<br>        except ServerError:<br>            # SmileID Server returned an error<br>            if retry_count &lt; max_retries:<br>                time.sleep(5)  # Wait for 5 seconds before resubmitting the job<br>            else:<br>                break  # Break out of the loop after all retries are exhausted<br>        else:<br>            BusinessVerificationResponse.save(response)<br>            break</pre><h3>Putting It All Together</h3><p>With these components in place, you can now initiate KYB verifications asynchronously. When a user submits their data, a BusinessJob instance is created, and the KYB task is scheduled using Celery. This means your application can continue serving requests without waiting for the verification to complete, ensuring a seamless user experience.</p><h3>Conclusion</h3><p>By combining Django, Celery, and Smile ID, you can efficiently handle server-to-server business verifications for Kenyan businesses. This approach not only improves the responsiveness of your application but also provides a robust and scalable solution for KYB processes.</p><p>To dive deeper into the Smile ID API and customization options, you can refer to the official <a href="https://docs.usesmileid.com/products/for-businesses-kyb/business-verification">Smile ID documentation</a>. Remember to obtain your partner ID and user ID from the Smile ID portal to enable seamless verification.</p><p>The source code for this project is available on <a href="https://github.com/aibunny/smileid-business">GitHub,</a> so feel free to explore and adapt it for your specific needs.</p><p>We hope this blog helps you in building a more efficient and responsive business verification process for your application.</p><p>Happy coding!</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=340bdc96d998" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/streamlining-business-verification-with-smile-id-and-django-rest-a-celery-powered-solution-340bdc96d998">Streamlining Business Verification with Smile ID and Django REST: A Celery-Powered Solution</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleashing the Power of GraphQL with Django and Graphene: A Deep Dive]]></title>
            <link>https://python.plainenglish.io/unleashing-the-power-of-graphql-with-django-and-graphene-a-deep-dive-945199929861?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/945199929861</guid>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[django-rest-framework]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 00:57:34 GMT</pubDate>
            <atom:updated>2023-12-18T04:37:14.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rRgXLv1mBX2W695CWV8biw.png" /><figcaption>@programmingwithmosh</figcaption></figure><p>In the ever-evolving landscape of web development, the demand for efficient and flexible APIs is paramount. As developers, we strive to empower our clients to request precisely the data they need while ensuring optimal performance. This is where GraphQL, a query language for APIs, and Django Graphene, a powerful library, come into play. Get ready for a deep dive into the world of GraphQL and Django Graphene.</p><h3>Understanding the Need for GraphQL</h3><p>Traditional RESTful APIs, while effective, often suffer from over-fetching or under-fetching of data. Clients must accept the structure of the data provided by the server, leading to either unnecessary data transfer or not having access to vital information. GraphQL revolutionizes this paradigm by letting clients define the structure of the response they desire. It offers the flexibility to request exactly what’s needed and nothing more, reducing data transfer and improving efficiency.</p><h3>The Power of Django and Graphene</h3><p>Django is a popular web framework known for its robust and rapid development capabilities. When combined with Graphene, a GraphQL library for Python, it becomes a formidable duo for building APIs.</p><p>Here are some advantages of using Django Graphene:</p><ol><li><strong>Rapid Development:</strong> Django’s ORM (Object-Relational Mapping) and Graphene’s type system make it quick and easy to define your data models and corresponding GraphQL schema.</li><li><strong>Strongly Typed:</strong> Graphene enforces a strong type system, ensuring that your API is well-structured and consistent.</li><li><strong>Optimized Queries:</strong> With GraphQL, clients can request only the data they need, which reduces the over-fetching of data. This leads to more efficient queries and faster response times.</li><li><strong>Introspection:</strong> GraphQL’s introspective nature allows clients to discover the API’s capabilities, making it easy to explore and query the schema.</li></ol><h3>Building a GraphQL API with Django Graphene</h3><p>Let’s dive into the process of building a GraphQL API with Django Graphene. Here are the fundamental steps:</p><p><strong>Installation:</strong> Start by installing Django and Graphene. You can use <strong>pip</strong> to install the necessary packages.</p><pre>pip install django graphene-django</pre><p><strong>Schema Definition:</strong></p><pre># schema.py<br>import graphene<br>from graphene_django.types import DjangoObjectType<br>from .models import YourModel<br><br>class YourModelType(DjangoObjectType):<br>    class Meta:<br>        model = YourModel<br><br>class Query(graphene.ObjectType):<br>    all_your_models = graphene.List(YourModelType)<br>    def resolve_all_your_models(self, info, **kwargs):<br>        return YourModel.objects.all()</pre><p>In this part, we define the schema for your GraphQL API using Graphene-Django.</p><p>We import necessary modules and your Django model (<strong>YourModel</strong>) to define a corresponding GraphQL type (<strong>YourModelType</strong>) that represents the model in the schema.</p><p>The <strong>Query</strong> class represents the entry points for querying data. In this example, it provides a single query called <strong>all_your_models</strong> that returns all instances of <strong>YourModel</strong> from the database when queried.</p><p>The <strong>resolve_all_your_models</strong> method is a resolver function. It tells GraphQL how to retrieve the data for the <strong>all_your_models</strong> query. In this case, it fetches all instances of <strong>YourModel</strong> from the database</p><p><strong>Queries and Mutations:</strong></p><pre># schema.py<br>class Mutation(graphene.ObjectType):<br>    create_your_model = CreateYourModel.Field()<br><br>class CreateYourModel(graphene.Mutation):<br>    class Arguments:<br>        data = YourModelInput()<br>    your_model = graphene.Field(YourModelType)<br>    def mutate(self, info, data):<br>        your_model = YourModel(**data)<br>        your_model.save()<br>        return CreateYourModel(your_model=your_model)</pre><p>In this section, we define mutations. Mutations are similar to queries but are used to modify data, such as creating, updating, or deleting records.</p><p>We create a <strong>Mutation</strong> class that contains one mutation called <strong>create_your_model</strong>. This mutation allows creating new instances of <strong>YourModel</strong>.</p><p><strong>CreateYourModel</strong> is the mutation itself. It specifies the arguments it accepts (<strong>data</strong>) and the resulting data type (<strong>your_model</strong>).</p><p>The <strong>mutate</strong> method defines what happens when this mutation is called. It creates a new <strong>YourModel</strong> instance with the provided data, saves it to the database, and then returns the newly created instance.</p><p><strong>GraphQL View:</strong></p><pre># views.py<br><br>from graphene_django.views import GraphQLView <br> <br>class CustomGraphQLView(GraphQLView):<br>     pass</pre><p>This part defines a custom GraphQL view that extends <strong>GraphQLView</strong> provided by Graphene-Django.</p><p>The <strong>CustomGraphQLView</strong> class is used to handle incoming GraphQL queries and mutations.</p><p><strong>URL Configuration:</strong></p><pre># urls.py<br><br>from django.urls import path<br>from .views import CustomGraphQLView<br><br>urlpatterns = [<br>    path(&#39;graphql/&#39;, CustomGraphQLView.as_view(graphiql=True, schema=schema)),<br>]</pre><p>Here, we configure the URL mapping for GraphQL queries and mutations.</p><p>The <strong>path</strong> function maps the URL endpoint <strong>/graphql/</strong> to the <strong>CustomGraphQLView</strong>.</p><p>The <strong>graphiql=True</strong> parameter enables the GraphiQL interface, which is a user-friendly tool for testing GraphQL queries and mutations during development.</p><p>The <strong>schema=schema</strong> parameter specifies the GraphQL schema to be used, which we defined in the <strong>schema.py</strong> file.</p><h3>Benefits of Using Django Graphene for GraphQL</h3><p>Django Graphene offers several advantages for building GraphQL APIs:</p><ol><li><strong>Seamless Integration:</strong> If you’re already familiar with Django, adding Graphene to your project is straightforward. It fits naturally into the Django ecosystem.</li><li><strong>Authentication and Permissions:</strong> Leverage Django’s built-in authentication and permission system to secure your GraphQL API.</li><li><strong>Data Validation:</strong> Graphene helps ensure that input data adheres to the specified types and constraints, reducing the risk of malformed requests.</li><li><strong>Scalability:</strong> As your application grows, Django’s scalability features can be utilized in tandem with Graphene to meet increasing demands.</li></ol><h3>Conclusion: Unleash the Full Potential of GraphQL</h3><p>By integrating GraphQL into your Django application with Graphene, you empower clients to request data in a way that suits their needs. This flexibility not only improves the client experience but also leads to more efficient and high-performing APIs. With the rich ecosystem of Django and the capabilities of Graphene, you can build powerful and adaptable APIs that stand at the forefront of modern web development. So, embrace GraphQL and Django Graphene to unleash the full potential of your APIs. Dive deeper into GraphQL and Django, and watch your API flourish.</p><p>Happy Coding !!​</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=945199929861" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/unleashing-the-power-of-graphql-with-django-and-graphene-a-deep-dive-945199929861">Unleashing the Power of GraphQL with Django and Graphene: A Deep Dive</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Django and Celery: Supercharging Your Web App with Asynchronous Tasks]]></title>
            <link>https://python.plainenglish.io/django-and-celery-supercharging-your-web-app-with-asynchronous-tasks-7ad989848d08?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ad989848d08</guid>
            <category><![CDATA[kenya]]></category>
            <category><![CDATA[celery]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[asynchronous]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Sun, 29 Oct 2023 00:28:37 GMT</pubDate>
            <atom:updated>2023-12-18T04:37:09.818Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-prTPSdlAslqcjTwAvVjw.png" /><figcaption>courtersy @saadali18</figcaption></figure><p>Now, let’s put this concept into action by building a Kenyan Weather App using Django and Celery. This app will fetch real-time weather data for different Kenyan locations. Users can get up-to-the-minute weather forecasts without causing delays in the application’s response time.</p><h4>But How Does Celery Coordinate All These Tasks?</h4><p>This is where the “message broker” comes into play. In the context of our Kenyan Weather App, the message broker is like a central command post for managing tasks. When a user requests weather data for a specific location, this request is handed over to Celery. Celery, in turn, communicates with the message broker.</p><p>The message broker’s job is to organize and distribute tasks to Celery workers. It ensures that each task gets processed by the right worker and in a timely manner. For example, fetching weather data for Nairobi is one task, while fetching data for Mombasa is another. The message broker makes sure that these tasks are processed independently and efficiently.</p><p>GitHub Repository: <a href="https://github.com/aibunny/celery-weather-app">Kenyan Weather App</a></p><h3>Getting Started: Setting Up Django and Celery</h3><ol><li><strong>Create a Django Project</strong></li></ol><pre>django-admin startproject core</pre><p><strong>2. Install Required Packages</strong></p><pre>pip install django celery[redis] requests</pre><p><strong>3. Create a Django App for Weather Data</strong></p><pre>python manage.py startapp weather</pre><p><strong>4. Configure Your Django Project:</strong></p><p><em>4.1 setting.py</em></p><p>In your Django project’s settings.py, add configurations for Celery and configure your database settings</p><pre># settings.py<br><br>INSTALLED_APPS = [<br>    # ...<br>    &#39;weatherapp&#39;,<br>    &#39;django_celery_results&#39;,<br>]<br><br># Celery Configuration<br>CELERY_BROKER_URL = &#39;redis://localhost:6379/0&#39;<br>CELERY_RESULT_BACKEND = &#39;django-db&#39;<br>CELERY_ACCEPT_CONTENT = [&#39;json&#39;]<br>CELERY_RESULT_SERIALIZER = &#39;json&#39;<br>CELERY_TASK_SERIALIZER = &#39;json&#39;</pre><p>4.2. Celery Configuration:</p><p>Create a celery.py file in your project&#39;s main directory (where settings.py is located):</p><pre># celery.py<br><br>from __future__ import absolute_import, unicode_literals<br>import os<br>from celery import Celery<br><br># Set the default Django settings module for the &#39;celery&#39; program.<br>os.environ.setdefault(&#39;DJANGO_SETTINGS_MODULE&#39;, &#39;core.settings&#39;)<br><br>#your projects name<br>app = Celery(&#39;core&#39;)<br><br># Using a string here means the worker doesn&#39;t have to serialize the configuration object to child processes.<br>app.config_from_object(&#39;django.conf:settings&#39;, namespace=&#39;CELERY&#39;)<br><br># Load task modules from all registered apps in different Django app configs.<br>app.autodiscover_tasks()</pre><h4><strong>5. Build the app:</strong></h4><p>5.1 In your weather/models.py:</p><p>Define the model for weather data:</p><pre># weather/models.py<br><br>from django.db import models<br><br>class WeatherData(models.Model):<br>    location = models.CharField(max_length=100)<br>    temperature = models.DecimalField(max_digits=5, decimal_places=2)<br>    humidity = models.DecimalField(max_digits=5, decimal_places=2)<br>    conditions = models.CharField(max_length=100)<br>    timestamp = models.DateTimeField(auto_now_add=True)<br><br>    def __str__(self):<br>        return self.location</pre><p>After defining the model, create and apply migrations:</p><pre>python manage.py makemigrations weatherapp<br>python manage.py migrate</pre><p>5.2 Forms for User Input</p><p>In your weather/forms.py, create a form for user input:</p><pre># weather/forms.py<br><br>from django import forms<br><br>class WeatherLocationForm(forms.Form):<br>    location = forms.CharField(label=&#39;Location&#39;, max_length=20)</pre><p>5.3 Views</p><p>Define views in your weather/views.py to interact with the Celery tasks and manage user input:</p><pre># weather/views.py<br><br>from django.shortcuts import render<br>from .forms import WeatherLocationForm<br>from .models import WeatherData<br>from .tasks import fetch_weather<br><br>def weather_view(request):<br>    if request.method == &#39;POST&#39;:<br>        form = WeatherLocationForm(request.POST)<br>        if form.is valid():<br>            location = form.cleaned_data[&#39;location&#39;]<br>            # Replace &#39;user_api_key&#39; with the actual user&#39;s API key.<br>            api_key = &#39;user_api_key&#39;<br>            result = fetch_weather.apply_async((location, api_key))<br>    else:<br>        form = WeatherLocationForm()<br>        result = None<br><br>    weather_data = WeatherData.objects.all().order_by(&#39;-timestamp&#39;)[:10]<br><br>    return render(request, &#39;weather/weather.html&#39;, {&#39;form&#39;: form, &#39;result&#39;: result, &#39;weather_data&#39;: weather_data})</pre><p>5.4 Templates</p><p>Create HTML templates in your templates directory, organizing them in a weather subdirectory:</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>&lt;head&gt;<br>    &lt;title&gt;Kenyan Weather App&lt;/title&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>    &lt;h1&gt;Kenyan Weather App&lt;/h1&gt;<br>    &lt;form method=&quot;post&quot;&gt;<br>        {% csrf_token %}<br>        {{ form.as_p }}<br>        &lt;button type=&quot;submit&quot;&gt;Get Weather&lt;/button&gt;<br>    &lt;/form&gt;<br><br>    {% if result %}<br>        &lt;h2&gt;Weather Data:&lt;/h2&gt;<br>        &lt;p&gt;Location: {{ result.location }}&lt;/p&gt;<br>        &lt;p&gt;Temperature: {{ result.temperature }} °C&lt;/p&gt;<br>        &lt;p&gt;Humidity: {{ result.humidity }} %&lt;/p&gt;<br>        &lt;p&gt;Conditions: {{ result.conditions }}&lt;/p&gt;<br>    {% endif %}<br><br>    &lt;h2&gt;Recent Weather Data:&lt;/h2&gt;<br>    &lt;ul&gt;<br>        {% for data in weather_data %}<br>            &lt;li&gt;<br>                Location: {{ data.location }}, Temperature: {{ data.temperature }} °C, Humidity: {{ data.humidity }} %, Conditions: {{ data.conditions }}<br>            &lt;/li&gt;<br>        {% endfor %}<br>    &lt;/ul&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;</pre><p>5.5 urls</p><p>Include URL patterns for your view in your core/urls.py:</p><pre># core/urls.py<br><br>from django.urls import path<br>from weather.views import weather_view<br><br>urlpatterns = [<br>    path(&#39;weather/&#39;, weather_view, name=&#39;weather_view&#39;),<br>]</pre><p><strong>6. Celery Task</strong></p><p>In your weather/tasks.py, define the Celery task to fetch weather data from a weather API:</p><pre># weatherapp/tasks.py<br><br>from celery import Celery<br>import requests<br>from .models import WeatherData<br><br>app = Celery(&#39;weather&#39;, broker=&#39;redis://localhost:6379/0&#39;)<br><br>@app.task<br>def fetch_weather(location, api_key):<br>    # Replace &#39;your_api_key&#39; and use the correct endpoint for your chosen weather API.<br>    WEATHER_API_URL = &#39;https://api.openweathermap.org/data/2.5/weather&#39;<br><br>    params = {<br>        &#39;q&#39;: location,  # Example: &#39;Nairobi,KE&#39;<br>        &#39;appid&#39;: api_key,  # Replace with your actual API key<br>    }<br><br>    response = requests.get(WEATHER_API_URL, params=params)<br><br>    if response.status_code == 200:<br>        weather_data = response.json()<br><br>        # Save the weather data to the database<br>        WeatherData.objects.create(<br>            location=location,<br>            temperature=weather_data[&#39;main&#39;][&#39;temp&#39;],<br>            humidity=weather_data[&#39;main&#39;][&#39;humidity&#39;],<br>            conditions=weather_data[&#39;weather&#39;][0][&#39;description&#39;]<br>        )<br><br>        return weather_data<br>    else:<br>        return {&#39;error&#39;: &#39;Unable to fetch weather data&#39;}</pre><p><strong>7. Running Your Celery Worker</strong></p><p>To ensure that Celery can execute your tasks, start a Celery worker process using the following command:</p><pre>celery -A core worker --loglevel=info</pre><p><strong>8. Run the Development Server</strong></p><p>Finally, start your Django development server:</p><pre>python manage.py runserver</pre><h4>In Conclusion</h4><p>In this journey through Django and Celery, we’ve seen how to supercharge your Kenyan Weather App with the power of asynchronous tasks. By utilizing Celery and a message broker, we’ve made it possible for your app to fetch real-time weather data for various Kenyan locations, ensuring users receive up-to-the-minute weather forecasts without any delays.</p><p>We’ve explored the role of the message broker, which acts as a central command post for managing tasks, and seen how it coordinates tasks to be processed independently and efficiently. Whether it’s fetching weather data for Nairobi or Mombasa, the message broker, coupled with Celery, ensures everything runs smoothly.</p><p>Now, armed with this knowledge, you’re ready to enhance the responsiveness and functionality of your web app.</p><p>Happy coding !!</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ad989848d08" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/django-and-celery-supercharging-your-web-app-with-asynchronous-tasks-7ad989848d08">Django and Celery: Supercharging Your Web App with Asynchronous Tasks</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Separate Settings Files in a Django Project for Local Development and Production: A…]]></title>
            <link>https://python.plainenglish.io/using-separate-settings-files-in-a-django-project-for-local-development-and-production-a-6a6866866d01?source=rss-e4afca7aa6b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a6866866d01</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[django-rest-framework]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[django-settings]]></category>
            <category><![CDATA[backend-development]]></category>
            <dc:creator><![CDATA[aibunny]]></dc:creator>
            <pubDate>Mon, 25 Sep 2023 05:18:18 GMT</pubDate>
            <atom:updated>2023-12-11T02:27:38.735Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Separate Settings Files in a Django Project for Local Development and Production: A Beginner’s Guide</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*p_vjtGcZRsG1HiFu_wam1g.jpeg" /></figure><h3>Introduction</h3><p>Django is a versatile framework for web development, but when it comes to configuring settings, it’s essential to distinguish between local development and production environments. In this beginner-friendly guide, we’ll walk you through the process of managing two separate settings files within a single Django project, ensuring that your application is secure and optimized for both local development and production.</p><h3>Why Separate Settings Files?</h3><p>Imagine building a house where you need different blueprints for the foundation (local development) and the final structure (production). Similarly, you need separate settings files for your Django project to ensure:</p><p><strong>1. Security:</strong> Local settings often contain sensitive information like secret keys and database credentials. Separating these settings from production configurations reduces the risk of exposing crucial data.</p><p><strong>2. Configuration:</strong> Local and production environments may require different configurations, such as database engines, debugging settings, and third-party integrations. Managing these configurations separately streamlines the deployment process.</p><p><strong>3. Debugging:</strong> Local settings enable debugging and development tools, while production settings ensure that errors and debugging information are hidden from users, maintaining a professional appearance.</p><h3>Creating Separate Settings Files</h3><p>Let’s go step by step, creating and managing separate settings files for your Django project:</p><h3>1. Duplicate settings.py</h3><p>Start by duplicating your existing <strong>settings.py</strong> file to create two distinct files: <strong>local_settings.py</strong> and <strong>production_settings.py</strong>. Your project&#39;s file structure should now look like this:</p><p>You can do this by running the following commands in your project’s root directory:</p><pre>$ cp settings.py local_settings.py<br>$ cp settings.py production_settings.py</pre><h3>2. Customize Local Settings</h3><p>Open <strong>local_settings.py</strong> and tailor it to your local development environment. This is where you can configure settings like <strong>DATABASES</strong> and <strong>SECRET_KEY</strong> specifically for your local setup. Here&#39;s an example:</p><pre># local_settings.py<br>if DEBUG:<br>    DATABASES = {<br>        &#39;default&#39;: {<br>            &#39;ENGINE&#39;: &#39;django.db.backends.sqlite3&#39;,<br>            &#39;NAME&#39;: &#39;local_db.sqlite3&#39;,<br>        }<br>    }<br>    SECRET_KEY = &#39;your_secret_key_for_local&#39;</pre><h3>3. Customize Production Settings</h3><p>Now, open <strong>production_settings.py</strong> and configure settings appropriate for your live application. Typically, you&#39;ll set <strong>DEBUG</strong> to <strong>False</strong>, use a different database, and employ a secure <strong>SECRET_KEY</strong>. Here&#39;s an example:</p><pre># production_settings.py<br>if not DEBUG:<br>    DATABASES = {<br>        &#39;default&#39;: {<br>            &#39;ENGINE&#39;: &#39;django.db.backends.postgresql&#39;,<br>            &#39;NAME&#39;: &#39;production_db&#39;,<br>            &#39;USER&#39;: &#39;your_production_db_user&#39;,<br>            &#39;PASSWORD&#39;: &#39;your_production_db_password&#39;,<br>            &#39;HOST&#39;: &#39;localhost&#39;,<br>            &#39;PORT&#39;: &#39;5432&#39;,<br>        }<br>    }<br>    SECRET_KEY = &#39;your_secure_secret_key_for_production&#39;</pre><h3>4. Implement Dynamic Importing</h3><p>Next, return to your main <strong>settings.py</strong> file and import the appropriate settings based on your environment. This is done using the <strong>os.environ</strong> variable to determine whether you&#39;re running in a local or production environment. Here&#39;s what to add to <strong>settings.py</strong>:</p><pre># settings.py<br>import os<br><br># Set DEBUG based on the environment<br>DEBUG = os.environ.get(&#39;DJANGO_DEBUG&#39;, &#39;True&#39;) == &#39;True&#39;<br>if DEBUG:<br>    from local_settings import *<br>else:<br>    from production_settings import *</pre><h3>5. Environment Variable Configuration</h3><p>To specify which settings file to use, you need to set the <strong>DJANGO_DEBUG</strong> environment variable when running your Django project. For local development, open your terminal and execute the following command (adjust <strong>your_project_name</strong> to your project&#39;s actual name):</p><pre>$ export DJANGO_DEBUG=True</pre><p>For production, configure the environment variable accordingly on your server:</p><pre>$ export DJANGO_DEBUG=False</pre><p>This ensures that Django loads the appropriate settings file based on the environment.</p><h3>Benefits of Separate Settings Files</h3><p>By following this approach, you can enjoy several advantages:</p><p><strong>1. Enhanced Security:</strong> Sensitive information remains separate from your codebase, reducing the risk of accidental exposure.</p><p><strong>2. Streamlined Configuration:</strong> It’s easy to customize settings for different environments, simplifying the deployment process.</p><p><strong>3. Error Isolation:</strong> Local settings enable debugging features, aiding development, while production settings ensure errors remain hidden from users, maintaining professionalism.</p><h3>Conclusion</h3><p>Managing separate settings files for local development and production environments in your Django project is a fundamental practice for maintaining security, configurational flexibility, and effective debugging. By following the steps outlined in this beginner’s guide, you’ll have a well-organized and secure setup that facilitates smooth development and deployment processes.</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a6866866d01" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/using-separate-settings-files-in-a-django-project-for-local-development-and-production-a-6a6866866d01">Using Separate Settings Files in a Django Project for Local Development and Production: A…</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>