<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Francisco Escher on Medium]]></title>
        <description><![CDATA[Stories by Francisco Escher on Medium]]></description>
        <link>https://medium.com/@francisco-escher?source=rss-d9e16079e054------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-vVvt5q_gNhqwwcGxz3AiA.jpeg</url>
            <title>Stories by Francisco Escher on Medium</title>
            <link>https://medium.com/@francisco-escher?source=rss-d9e16079e054------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:52:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@francisco-escher/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DInjections: building a Python dependency injection framework]]></title>
            <link>https://itnext.io/dinjections-building-a-python-dependency-injection-framework-773ecc93b57e?source=rss-d9e16079e054------2</link>
            <guid isPermaLink="false">https://medium.com/p/773ecc93b57e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[dependency-injection]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Francisco Escher]]></dc:creator>
            <pubDate>Mon, 11 Dec 2023 04:23:13 GMT</pubDate>
            <atom:updated>2023-12-11T23:39:28.398Z</atom:updated>
            <content:encoded><![CDATA[<h3>DInjections: a better Python dependency injection framework</h3><h4>Inspired by Uber’s Golang FX framework</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pyc163RO3TFMpnKM7T1FSA.png" /></figure><h3>Introduction</h3><p>Dependency injection (DI) is a powerful design pattern that simplifies code structure and improves maintainability. While implementations of this pattern are already quite common in a language like Python, which has been around for a long time, sometimes I feel that the language misses a more straightforward one.</p><p>In this year of 2023, I’ve been doing a lot of code using <a href="https://uber-go.github.io/fx/">Go’s package fx</a>, which is a framework that provides DI using the functional options pattern. I find it really powerful and, for that reason, I wanted to create a similar one for when I am developing for Python.</p><p>So… I did it! I am calling this package DInjections, and its code can be found <a href="https://github.com/franciscoescher/dinjections">here</a>.</p><h3>Overview</h3><p>Let’s see how the framework looks first, and then we can go over some practical use cases:</p><pre>from dinjections import App, Lifecycle, Hook, Provide, Invoke<br><br><br># this is a dependency of your project. You can have as many as you need.<br>class MyDependency:<br>    def run():<br>        print(&quot;running dependency 1&quot;)<br><br># This is the code to be invoked by your application<br>def register_hooks(l: Lifecycle, d: MyDependency):<br>    l.append_hook(Hook(<br>        on_start=lambda: {<br>            d.run()<br>        },<br>    ))<br><br># This is your app&#39;s definition, which you can provide dependencies<br># and register lifecycle hooks<br>app = App(<br>    Provide(<br>        MyDependency,<br>    ),<br>    Invoke(<br>        register_hooks,<br>    ))<br><br># Run this o call the registered hooks<br>app.run()</pre><h4>The App Class</h4><p>The framework’s usage starts with the creation of an object of the class “App”. To this object, you will pass all the provided dependencies, as well as the functions to register hooks to the lifecycle (more on that later). All objects passed to this one are of the type “Option” or “Module”. An Option applies changes to your “App” (like adding dependencies to it). You can pass multiple options of the same kind. We will see how Modules work later.</p><h4>The Provide Option</h4><p>The “Provide” option registers all the dependencies you will need across other dependencies or the invoke options. You may pass a class with hints in its __init__ method, a function that returns an object — also with hints in ints arguments and return type, or the object “Provider” described bellow. Don’t forget to add the dependencies for your dependencies here as well, as shown below:</p><pre>from dinjections import App, Lifecycle, Hook, Provide, Invoke<br><br><br>class Mydependency2:<br>    def run():<br>        print(&quot;running dependency 2&quot;)<br><br>class MyDependency:<br>    def __init__(self, d: Mydependency2):<br>        self.d = d<br><br>    def run(self):<br>        print(&quot;running dependency 1&quot;)<br><br># This is the code to be invoked by your application<br>def register_hooks(l: Lifecycle, d: MyDependency):<br>    l.append_hook(Hook(<br>        on_start=lambda: {<br>            d.run()<br>        },<br>    ))<br><br># This is your app&#39;s definition, which you can provide dependencies<br># and register lifecycle hooks<br>app = App(<br>    Provide(<br>        Mydependency2,<br>        MyDependency,<br>    ),<br>    Invoke(<br>        register_hooks,<br>    ))<br><br># Run this o call the registered hooks<br>app.run()</pre><h4>The Provider Class</h4><p>The provider class allows you to set advanced options to a dependency. Use the same as hint in the receiving end (as below). The supported features now are:</p><ul><li>For different dependencies with the same type, you can pass the variable “name” so you can differentiate them. If this is not done, dependencies of the same type may override each other.</li></ul><pre>from dinjections import App, Lifecycle, Hook, Provide, Invoke<br><br><br>class MyDependency:<br>    def __init__(self, d: Mydependency2):<br>        self.d = d<br><br>    def run(self):<br>        print(&quot;running dependency&quot;)<br><br>def register_hooks(l: Lifecycle, <br>    d1: Provider(MyDependency, name=&quot;d1&quot;),<br>    d2: Provider(MyDependency, name=&quot;d2&quot;)):<br><br>    l.append_hook(Hook(<br>        on_start=lambda: {<br>            d1.run()<br>        },<br>    ))<br>    l.append_hook(Hook(<br>            on_start=lambda: {<br>                d2.run()<br>            },<br>        ))<br><br>app = App(<br>    Provide(<br>        Provider(MyDependency, name=&quot;d1&quot;),<br>        Provider(MyDependency, name=&quot;d2&quot;),<br>    ),<br>    Invoke(<br>        register_hooks,<br>    ))<br><br>app.run()</pre><ul><li>If you need to group dependencies of the same kind, and receive them as a list, set the “group” argument to “True”.</li></ul><pre>from dinjections import App, Lifecycle, Hook, Provide, Invoke<br><br>class MyDependency:<br>    def __init__(self, d: Mydependency2):<br>        self.d = d<br><br>    def run(self):<br>        print(&quot;running dependency&quot;)<br><br>def register_hooks(l: Lifecycle, d: Provider(MyDependency, group=True)):<br>    # d now will be a list with dependencies of type MyDependency<br>    for element in d:<br>      l.append_hook(Hook(<br>          on_start=lambda: {<br>              element.run()<br>          },<br>      ))<br><br>app = App(<br>    Provide(<br>        Provider(MyDependency, group=True),<br>        Provider(MyDependency, group=True),<br>    ),<br>    Invoke(<br>        register_hooks,<br>    ))<br><br>app.run()</pre><h4>The Invoke Option</h4><p>The invoke option will not provide a dependency to the stack but will receive a Lifecycle parameter, that can be used to add hooks to the application, for when the app is “run”. Currently supported hooks are “on_start” and “on_stop”.</p><h4>Modularity</h4><p>Thanks to the functional options pattern, the framework is completely modular, so I can use multiple “Provide” and “Invoke” options from different packages and it would work fine. In addition to that, a developer can also make an object of the type “Module”, that also receives options, and pass it to the main application, like in the following example:</p><pre># mypckg1.py<br>module = Module(<br>    Provide(<br>        # Dependencies...<br>    ),<br>    Invoke(<br>        # hook registration...<br>    )<br>)<br><br># mypckg2.py<br>module = Module(<br>    Provide(<br>        # Dependencies...<br>    ),<br>    Invoke(<br>        # hook registration...<br>    )<br>)<br><br>app = App(<br>    mypckg1.module,<br>    mypckg2.module)<br><br>app.run()</pre><p>Attention: Make sure there is no cyclic dependency in your options, or a “RecursionError” will be raised.</p><h3>Conclusion</h3><p>In conclusion, the DInjections framework brings the power and elegance of Uber’s Go FX framework to the Python world, adapting the strengths of Go’s dependency injection pattern to Python’s dynamic nature. By leveraging the functional options pattern, DInjections offers a flexible and modular approach to managing dependencies in Python applications.</p><p>Furthermore, the modularity enabled by the functional options pattern in DInjections ensures that developers can easily integrate multiple modules, even from different packages, without running into dependency conflicts. This makes it an ideal choice for large-scale projects where maintainability and scalability are of paramount importance.</p><p>Overall, I think this package may offer a robust and intuitive solution for dependency injection. Luckily it will be a valuable contribution for developers seeking to implement clean and efficient dependency management in their projects.</p><p>Please, feel free to contribute!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=773ecc93b57e" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/dinjections-building-a-python-dependency-injection-framework-773ecc93b57e">DInjections: building a Python dependency injection framework</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Code Locking]]></title>
            <link>https://blog.stackademic.com/understanding-code-locking-03e5c038c30f?source=rss-d9e16079e054------2</link>
            <guid isPermaLink="false">https://medium.com/p/03e5c038c30f</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Francisco Escher]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 14:17:08 GMT</pubDate>
            <atom:updated>2023-11-30T09:44:34.294Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Dive into Concurrency and Thread Safety</h4><h4>TLDR;</h4><p>This article delves into the critical concept of code locking in concurrent programming, essential for maintaining data integrity when multiple threads access shared resources. Key discussions include performance considerations like minimizing lock overhead and managing contention, advanced strategies involving lock-free and wait-free algorithms, and addressing common pitfalls such as deadlocks and priority inversion. The article also compares various synchronization mechanisms, including mutexes, semaphores, and read-write locks, to guide best practices in different scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gz-v1c9Idut3zEngvDDDzA.png" /></figure><h3>The Essence of Code Locking</h3><p>In today’s concurrent software environment, understanding the principles of concurrency and thread safety is crucial. Code locking is a basic mechanism, but still a fundamental concept that ensures data integrity and consistency when multiple threads access shared resources. This article dives into the general concept of code locking, performance considerations, common pitfalls, best practices, and alternatives. I will provide small examples of implementation using Golang and Python to illustrate the basic idea, however, my focus will be on the underlying principles, rather than the specifics of the languages.</p><h3>What is Code Locking?</h3><p>Code locking is a synchronization mechanism used to prevent concurrent access to a resource by multiple threads or processes. It’s like having a digital “lock” on data or code segment which ensures that only one thread can access the resource at a time. This is critical in preventing race conditions, where the outcome of processes depends on the sequence or timing of other uncontrollable events.</p><p>Most languages will have support for code locking. Let’s see how this works in Golang and Python.</p><h4>Code Locking in Golang</h4><p>Golang, with its emphasis on concurrency, provides built-in support for synchronization primitives like mutexes. Let’s look at a simple example</p><pre>package main<br><br>import (<br>    &quot;fmt&quot;<br>    &quot;sync&quot;<br>)<br><br>var (<br>    counter int<br>    lock    sync.Mutex<br>)<br><br>func main() {<br>    for i := 0; i &lt; 5; i++ {<br>        go increment()<br>    }<br>    // ...wait for all goroutines to complete...<br>    fmt.Println(&quot;Final Counter:&quot;, counter)<br>}<br><br>func increment() {<br>    // here, the code code will only proceede when there is no other lock<br>    // prior to this. If it proceeds, the code locks for other calls.<br>    lock.Lock()<br>    // we defer unlock, so when it&#39;s done other calls can also use this code.<br>    defer lock.Unlock()<br>    counter++<br>}</pre><p>In this Golang snippet, we use a sync.Mutex to ensure that only one goroutine can increment the counter at a time, thus avoiding a race condition.</p><h4>Code Locking in Python</h4><p>Python, being a multi-paradigm language, also supports thread synchronization. The threading module provides basic synchronization primitives, including locks:</p><pre>import threading<br><br>counter = 0<br>lock = threading.Lock()<br><br>def increment():<br>    global counter<br>    # here, the code code will only proceede when there is no other lock<br>    # prior to this. If it proceeds, the code locks for other calls.<br>    with lock:<br>        counter += 1<br><br>threads = []<br>for i in range(5):<br>    thread = threading.Thread(target=increment)<br>    thread.start()<br>    threads.append(thread)<br><br>for thread in threads:<br>    thread.join()<br><br>print(f&quot;Final Counter: {counter}&quot;)</pre><p>Here, we use a threading.Lock() to ensure that only one thread can modify the counter at a time.</p><h3>Performance Considerations in Code Locking</h3><p>Efficient handling of concurrency is not just about preventing errors; it’s also about optimizing performance. Code locking, while essential for thread safety, can have significant implications for the performance of your application. Below are key considerations and strategies to optimize performance when dealing with code locks.</p><h4>Minimizing Lock Overhead</h4><p>Every lock operation consumes system resources. Excessive locking can lead to high overhead, diminishing the benefits of concurrency.</p><p>Best Practice: Minimize lock usage. Only use locks around the critical section of the code that absolutely needs synchronization. Consider using finer-grained locks to reduce the locked code portion.</p><h4>Lock Contention and Scalability</h4><p>High lock contention can lead to threads waiting in line to access resources, reducing scalability.</p><p>Best Practice: Reduce lock contention by using more granular locks or different locking mechanisms like read-write locks. Design your system to spread the workload evenly across threads.</p><h4>Avoiding Long-Held Locks</h4><p>Holding a lock for an extended period can block other threads, leading to performance bottlenecks.</p><p>Best Practice: Keep lock holding times as short as possible. This can be achieved by optimizing the code within the lock and avoiding calling slow operations while holding a lock.</p><h4>Lock-Free and Wait-Free Algorithms</h4><p>Lock-free algorithms can provide better performance in highly concurrent environments by allowing threads to progress without getting blocked.</p><p>Best Practice: Where applicable, use lock-free data structures and algorithms. While more complex to implement, they can offer significant performance benefits in high-concurrency scenarios.</p><h3>Common Pitfalls and Best Practices in Code Locking</h3><p>But as always when working with concurrency, there are some pitfalls and best practices we should consider. Understanding these common mistakes and adhering to best practices can significantly enhance the reliability and performance of your applications.</p><h4>Deadlocks</h4><p>Deadlocks occur when two or more threads are waiting for locks in a circular chain, with each holding a lock that the other wants. This results in a standstill where none of the threads can proceed.</p><p>Best Practice: To avoid deadlocks, always acquire locks in a consistent order. Consider using a timeout for lock acquisition and implement a robust deadlock detection and recovery mechanism.</p><h4>Priority Inversion</h4><p>Priority inversion happens when a high-priority thread is waiting for a lock held by a lower-priority thread. This can lead to performance degradation, especially in real-time systems.</p><p>Best Practice: Use priority inheritance or ceiling protocols where the system temporarily elevates the priority of the lock-holding thread.</p><h4>Lock Contention</h4><p>Lock contention occurs when multiple threads are trying to acquire the same lock, leading to reduced parallelism and increased context switching.</p><p>Best Practice: Minimize lock contention by keeping the lock-holding time short, using more granular locks, or employing lock-free data structures where appropriate.</p><h4>Neglecting Lock-Free and Read-Write Locks</h4><p>Over-reliance on mutexes can overlook the benefits of other synchronization mechanisms like read-write locks or lock-free algorithms, which can offer better performance in certain scenarios.</p><p>Best Practice: Evaluate if read-write locks could be more efficient, especially when read operations far outnumber write operations. Explore lock-free data structures for highly concurrent scenarios.</p><h3>Comparison with Other Synchronization Mechanisms</h3><h4>Mutexes vs. Semaphores</h4><p>Mutexes (Mutual Exclusion): Mutexes are key for locking code sections to ensure that only one thread accesses a resource at a time. They are simpler and safer for beginners, as they prevent accidental release from a different thread.</p><p>Semaphores: Semaphores are more flexible than mutexes as they can allow multiple threads to access a resource up to a certain limit. They are useful for controlling access to a resource pool but are more complex and prone to errors, such as accidental release from a non-owning thread.</p><h4>Read-Write Locks</h4><p>These locks allow multiple readers or one writer at a time, optimizing scenarios where read operations are more frequent than write operations. They can significantly improve performance in read-heavy applications but add complexity in managing two different types of locks.</p><h4>Condition Variables</h4><p>Used in conjunction with mutexes, condition variables allow threads to wait for certain conditions to be met. They are ideal for scenarios where threads need to wait for specific states or events before proceeding. They add complexity but are powerful for coordinating thread activities.</p><h4>Atomic Operations</h4><p>These operations ensure that a set of operations complete as a single, indivisible step. They are used for simple operations (like incrementing a counter) and can be faster than mutexes for these scenarios. However, they are limited in scope and not suitable for complex synchronization tasks.</p><h4>Lock-Free and Wait-Free Algorithms</h4><p>Lock-Free Algorithms: These provide a form of synchronization without using traditional locking mechanisms, allowing other threads to continue execution even if one thread is blocked or slow. They are more complex to implement but can offer high performance in certain contexts.</p><p>Wait-Free Algorithms: A stronger form of lock-free programming, wait-free algorithms guarantee that every thread will complete its operation in a finite number of steps. They are extremely complex to implement but provide the highest level of performance and reliability.</p><h3>Conclusion: The Universality of Code Locking</h3><p>Though the syntax and specific implementations differ, the concept of code locking is universal across programming languages. Its correct application is vital for developing reliable and safe concurrent applications, irrespective of the language used. By understanding the principles of code locking, developers can ensure data integrity and prevent complex bugs in multi-threaded environments.</p><h3>Stackademic</h3><p><em>Thank you for reading until the end. Before you go:</em></p><ul><li><em>Please consider </em><strong><em>clapping</em></strong><em> and </em><strong><em>following</em></strong><em> the writer! 👏</em></li><li><em>Follow us on </em><a href="https://twitter.com/stackademichq"><strong><em>Twitter(X)</em></strong></a><em>, </em><a href="https://www.linkedin.com/company/stackademic"><strong><em>LinkedIn</em></strong></a><em>, and </em><a href="https://www.youtube.com/c/stackademic"><strong><em>YouTube</em></strong></a><strong><em>.</em></strong></li><li><em>Visit </em><a href="http://stackademic.com/"><strong><em>Stackademic.com</em></strong></a><em> to find out more about how we are democratizing free programming education around the world.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03e5c038c30f" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/understanding-code-locking-03e5c038c30f">Understanding Code Locking</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bloom Filters and Go]]></title>
            <link>https://itnext.io/bloom-filters-and-go-1d5ac62557de?source=rss-d9e16079e054------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d5ac62557de</guid>
            <category><![CDATA[bloom-filter]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[bloom]]></category>
            <dc:creator><![CDATA[Francisco Escher]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 17:00:43 GMT</pubDate>
            <atom:updated>2023-11-30T03:06:57.843Z</atom:updated>
            <content:encoded><![CDATA[<h3>Implementing a Bloom Filter in Go</h3><h4>A Practical Guide</h4><p>This article presents the concept of a simple bloom filter and a more scalable one. It implements them in Go. The entire code can be seen and used at <a href="https://github.com/franciscoescher/gobloom">https://github.com/franciscoescher/gobloom</a>. Also, while this article presents quite technical terms and discussions, I will try to explain them in a language a wider audience can understand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1Jt-99t1zwrsaR-FpYGfQ.png" /></figure><h3>Introduction</h3><p>Imagine you’re at a vast library, with millions of books. You want to know if a particular book is in the library, but checking each shelf is impractical. Wouldn’t it be nice to know beforehand when a book is not there, so you can avoid searching for it? Here’s where a Bloom Filter comes into play — think of it as a pre-filter, a first line of defence in data retrieval. It quickly sifts through massive volumes of data, indicating whether an item is possibly in the set or definitely not, saving you from exhaustive searches through every item.</p><p>What is a Bloom Filter? It’s not a physical filter, but a clever data structure. It allows us to check the presence of an element in a set with incredible speed and surprising space efficiency. The beauty of a Bloom Filter lies in its ability to handle massive data sets while occupying minimal space.</p><p>False Positives, But No False Negatives: Bloom Filters have an intriguing quirk — they can mistakenly tell you that a non-existent item is in the set (a false positive). Still, they’ll never miss an item that’s actually there. This unique trait makes them invaluable when speed and space efficiency are more critical than absolute accuracy, like internet routers checking IP addresses or spell checkers in word processors.</p><p>Why This Post? As a developer, I found the elegance of Bloom Filters captivating. However, implementing them efficiently can be challenging. This post aims to bridge that gap. I’ll take you through a practical implementation of a Bloom Filter in Go, using my own code as a guide.</p><h3>Understanding the Basics</h3><h4>The Mechanics of a Bloom Filter</h4><ul><li>Hash Functions at Work: A Bloom Filter uses multiple hash functions to map each element to several positions in a bit array. When you add an item, the Bloom Filter marks these positions as true. This multi-hash approach is what makes Bloom Filters highly space-efficient and quick in querying.</li><li>The Bit Array: At the heart of a Bloom Filter is a simple array of bits (or boolean flags). Initially, all bits are set to false. As elements are added, specific bits determined by the hash functions are set to true. The size of this array and the number of hash functions are critical for the efficiency of the filter.</li></ul><h4>Probability and Efficiency</h4><ul><li>False Positive Rate: Understanding the false positive rate is crucial. It’s the probability that the Bloom Filter incorrectly indicates the presence of an element. This rate is affected by the size of the bit array and the number of elements added.</li><li>Calculating Size and Hash Functions: The magic lies in balancing the size of the bit array (m) and the number of hash functions (k). A larger array with more hash functions reduces the false positive rate but requires more memory and computation. There’s an optimal balance based on the expected number of elements and the acceptable false positive rate.</li></ul><h4>The Trade-Offs</h4><ul><li>Space vs. Accuracy: Bloom Filters offer a trade-off between space and accuracy. They are incredibly space-efficient compared to other data structures like hash tables or lists, but this comes at the cost of occasional false positives.</li><li>Speed vs. Precision: They provide a quick way to eliminate non-existent elements, speeding up searches significantly. However, for applications where precision is paramount, additional checks might be necessary following a positive result from the Bloom Filter.</li></ul><h3>Implementation in Go</h3><p>Let&#39;s first define the interfaces:</p><pre>package gobloom<br><br>import &quot;hash&quot;<br><br>type Interface interface {<br> Add([]byte)<br> Test([]byte) bool<br>}<br><br>type Hasher interface {<br> GetHashes(n uint64) []hash.Hash64<br>}</pre><p>The “Hasher” interface will be used to generate hashes for our bloom filter implementation. This allows for the decoupling of responsibilities (and allows us to pass it as a dependency).</p><p>Why would I want to make the hashes configurable? The choice of hash functions is a critical decision that influences the filter’s accuracy, efficiency, and overall suitability for a given application. It’s a balance between collision rate, uniform distribution, computational efficiency, and the specific nature of the data being handled. The Hasher interface in your implementation provides the flexibility to experiment with different hash functions and find the optimal setup for a specific use case.</p><p>The “Interface” interface defines the core functionality expected from any Bloom Filter implementation: Adding an element, and then testing if it might already be there or not.</p><p>Let’s implement it below:</p><pre>package gobloom<br><br>import (<br> &quot;fmt&quot;<br> &quot;hash&quot;<br> &quot;math&quot;<br>)<br><br>// BloomFilter represents a single Bloom filter structure.<br>type BloomFilter struct {<br> bitSet []bool        // The bit array represented as a slice of bool<br> m      uint64        // The number of bits in the bit set (shortcut for len(bitSet)<br> hashes []hash.Hash64 // The hash functions to use<br> k      uint64        // The number of hash functions to use (shortcut for len(hashes)<br> mutex  sync.Mutex    // Mutex to ensure thread safety<br>}<br><br>// NewBloomFilterWithHasher creates a new Bloom filter with the given number <br>// of elements (n) and false positive rate (p).<br>func NewBloomFilterWithHasher(n uint64, p float64, h Hasher) (*BloomFilter, error) {<br> if n == 0 {<br>  return nil, fmt.Errorf(&quot;number of elements cannot be 0&quot;)<br> }<br> if p &lt;= 0 || p &gt;= 1 {<br>  return nil, fmt.Errorf(&quot;false positive rate must be between 0 and 1&quot;)<br> }<br> if h == nil {<br>  return nil, fmt.Errorf(&quot;hasher cannot be nil&quot;)<br> }<br> m, k := getOptimalParams(n, p)<br> return &amp;BloomFilter{<br>  m:      m,<br>  k:      k,<br>  bitSet: make([]bool, m),<br>  hashes: h.GetHashes(k),<br> }, nil<br>}<br><br>// getOptimalParams calculates the optimal parameters for the Bloom filter,<br>// the number of bits in the bit set (m) and the number of hash functions (k).<br>func getOptimalParams(n uint64, p float64) (uint64, uint64) {<br> m := uint64(math.Ceil(-1 * float64(n) * math.Log(p) / math.Pow(math.Log(2), 2)))<br> if m == 0 {<br>  m = 1<br> }<br> k := uint64(math.Ceil((float64(m) / float64(n)) * math.Log(2)))<br> if k == 0 {<br>  k = 1<br> }<br> return m, k<br>}</pre><p>The parameters used to instantiate your struct are crucial for calculating the optimal parameters for the Bloom Filter, namely the size of the bit array (m) and the number of hash functions (k). This optimization is based on the desired number of elements (n) you expect to store in the Bloom Filter and the acceptable false positive rate (p). mutex is used to make it thread-safe. Here are some considerations on how to decide them:</p><ul><li>Adjusting to Expected Data: The function tailors the Bloom Filter based on how much data you expect to store in it (n), and how often you&#39;re willing to accept false positives (p).</li><li>Balancing Size and Accuracy: The goal is to find a balance where the Bloom Filter is not too big (which wastes memory) and not too inaccurate (which leads to more false positives).</li><li>Based on Data and Accuracy: The size of the bit array is calculated based on the expected number of elements and the desired false positive rate.</li><li>Avoiding Overcrowding: A larger bit array can reduce the chance of false positives but uses more memory. The function aims to find a size that’s “just right.”</li><li>Optimizing Performance: The number of hash functions is crucial for how the data is spread across the bit array.</li><li>Balancing Spread and Workload: Too few hash functions might not spread out the data enough, leading to false positives. Too many, and the Bloom Filter slows down because it’s doing more work than necessary.</li></ul><h4>Now let’s implement the Add and Test functions:</h4><pre>// Add adds an item to the Bloom filter.<br>func (bf *BloomFilter) Add(data []byte) {<br> bf.mutex.Lock()<br> defer bf.mutex.Unlock()<br> for _, hash := range bf.hashes {<br>  hash.Reset()<br>  hash.Write(data)<br>  hashValue := hash.Sum64() % bf.m<br>  bf.bitSet[hashValue] = true<br> }<br>}<br><br>func (bf *BloomFilter) Test(data []byte) bool {<br> bf.mutex.Lock()<br> defer bf.mutex.Unlock()<br> for _, hash := range bf.hashes {<br>  hash.Reset()<br>  hash.Write(data)<br>  hashValue := hash.Sum64() % bf.m<br>  if !bf.bitSet[hashValue] {<br>   return false<br>  }<br> }<br> return true<br>}</pre><p>The Add function is used to insert elements into the Bloom Filter, hashing the value with each hash from the filter and writing it into the corresponding bits.</p><p>The Test function is used to check whether an element is possibly in the Bloom Filter, verifying if the corresponding hashed bits are set to true in the bitSet. If not, it is proved that the data was not yet written to the filter.</p><h4>Testing the filter</h4><p>Here is a small application to test our new filter:</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br><br> &quot;github.com/franciscoescher/gobloom&quot;<br>)<br><br>func main() {<br> bf, _ := gobloom.NewBloomFilter(1000, 0.01)<br> bf.Add([]byte(&quot;foo&quot;))<br> bf.Add([]byte(&quot;bar&quot;))<br> bf.Add([]byte(&quot;baz&quot;))<br><br> fmt.Println(bf.Test([]byte(&quot;foo&quot;))) // true<br> fmt.Println(bf.Test([]byte(&quot;bar&quot;))) // true<br> fmt.Println(bf.Test([]byte(&quot;baz&quot;))) // true<br> fmt.Println(bf.Test([]byte(&quot;qux&quot;))) // false<br>}</pre><h3>Implementing the hasher</h3><p>The hasher struct should simply return an n number for hashes. Let’s use the github.com/spaolacci/murmur3 package for this:</p><pre>package gobloom<br><br>import (<br> &quot;hash&quot;<br><br> &quot;github.com/spaolacci/murmur3&quot;<br>)<br><br>type MurMur3Hasher struct{}<br><br>func NewMurMur3Hasher() *MurMur3Hasher {<br> return &amp;MurMur3Hasher{}<br>}<br><br>func (h *MurMur3Hasher) GetHashes(n uint64) []hash.Hash64 {<br> hashers := make([]hash.Hash64, n)<br> for i := 0; uint64(i) &lt; n; i++ {<br>  hashers[i] = murmur3.New64WithSeed(uint32(i))<br> }<br> return hashers<br>}</pre><p>Note: if you want to write your own hasher, make sure to use an algorithm that allows you to seed your hashes. By using different seeds for each hash function, you ensure that each function behaves differently. This diversity in behavior significantly reduces the likelihood of collisions, as the same input will produce different hash outputs in each function.</p><p>For a Bloom Filter to be efficient, it’s essential that elements are uniformly distributed across the bit array. A uniform distribution ensures that each bit in the array has an equal probability of being set, minimizing the chances of false positives.</p><p>Different seeds help in creating hash functions that spread the elements more evenly across the bit array. If all hash functions were seeded the same, they might exhibit similar distribution patterns, leading to clustering of set bits and thus higher false positives.</p><h3>Scaling the Bloom Filter</h3><p>If you paid attention, you might have realized that this implementation is static for value cardinality, meaning that once you set the number of values, you are stuck with it.</p><p>What if you are not sure about these fields, instead you want to create a filter that grows as you use it?</p><h4>Why Scalable?</h4><p>Traditional Bloom Filters have a fixed size, which means their efficiency and accuracy are predetermined by their initial configuration. However, in real-world applications, data often grows unpredictably, and a fixed-size filter might either waste space or become less accurate over time. I will present a solution that addresses this by dynamically adjusting its capacity and false positive rate as more elements are added, making it an ideal solution for applications dealing with evolving data sets.</p><p>Let’s see how this goes:</p><h4>Components of the Scalable Bloom Filter</h4><pre>// ScalableBloomFilter combines multiple BloomFilter slices to adapt to a growing number of elements.<br>type ScalableBloomFilter struct {<br> filters  []*BloomFilter // A slice of BloomFilter pointers, representing each layer of the scalable filter<br> n        uint64         // The number of items that have been added<br> fpRate   float64        // The false positive rate for the current filter slice<br> fpGrowth float64        // Factor by which the false positive probability should increase for each additional filter slice<br>}</pre><ul><li>filters: A slice of BloomFilter pointers, each representing a layer in the scalable structure. These layers allow the filter to expand its capacity.</li><li>n: This field tracks the total number of items that have been added across all filter layers.</li><li>fpRate: The false positive rate for the current filter layer, dictating the likelihood of the filter incorrectly indicating the presence of an element.</li><li>fpGrowth: A factor that determines how the false positive probability increases with each additional filter layer.</li></ul><p>The logic of the ScalableBloomFilter is designed to handle growing datasets while maintaining a manageable false positive rate. This is achieved by using a series of standard Bloom Filters, each acting as a &quot;layer&quot; within the scalable filter. Here&#39;s a breakdown of the logic and how it manages the dynamic nature of the data it stores</p><pre>// NewScalableBloomFilterWithHasher instantiates a new ScalableBloomFilter.<br>func NewScalableBloomFilterWithHasher(initialSize uint64, fpRate float64, fpGrowth float64, h Hasher) (*ScalableBloomFilter, error) {<br> if initialSize &lt;= 0 {<br>  return nil, errors.New(&quot;invalid initial size, must be greater than 0&quot;)<br> }<br> if fpRate &lt;= 0 || fpRate &gt;= 1 {<br>  return nil, fmt.Errorf(&quot;invalid false positive rate, must be between 0 and 1, got %f&quot;, fpRate)<br> }<br> if fpGrowth &lt;= 0 {<br>  return nil, fmt.Errorf(&quot;invalid false positive growth rate, must be greater than 0, got %f&quot;, fpGrowth)<br> }<br><br> bf, err := NewBloomFilterWithHasher(initialSize, fpRate, h)<br> if err != nil {<br>  return nil, err<br> }<br><br> // Return a new scalable Bloom filter struct with the initialized slice and parameters.<br> return &amp;ScalableBloomFilter{<br>  filters:  []*BloomFilter{bf}, // Start with one filter slice<br>  fpRate:   fpRate,             // Set the initial false positive rate<br>  fpGrowth: fpGrowth,           // Set the growth rate for false positives as the filter scales<br>  n:        0,                  // Initialize with zero elements added<br> }, nil<br>}</pre><p>NewScalableBloomFilter initializes a new scalable Bloom filter with estimated initial size, target false positive rate, and growth rate for the false positive probability with each new filter slice. It is essential to carefully select these values based on the use-case requirements to maintain system performance and desired accuracy as the dataset grows.</p><h4>Parameters:</h4><ul><li>initialSize (uint64): The estimated number of elements you expect to store in the bloom filter initially. This is not a hard limit but rather a guideline for preparing the initial Bloom filter layer. A size that’s too small could lead to a rapid addition of new slices, increasing memory usage, and a size that’s too large could waste memory upfront.</li><li>fpRate (float64): The desired false positive probability for the first Bloom filter slice. It determines how likely it is that a ‘Test’ operation falsely indicates the presence of an element. A smaller fpRate will increase the number of bits used in the initial filter (decreasing the chance of false positives), but also increase consumption of memory. Typical values are between 0.01 and 0.001 (1% — 0.1%).</li><li>fpGrowth (float64): The growth rate of the false positive probability with the addition of each subsequent filter slice. If set to a value greater than 1, each new filter layer tolerates a higher false positive rate, which can be useful to postpone the addition of new layers and control memory usage. Commonly, this parameter should be set to a value close to 1 (e.g. 1.5–2), as a higher growth rate could lead to a rapidly deteriorating false positive rate.</li></ul><h4>Best Practices</h4><ul><li>Conduct a pre-analysis based on expected data growth to estimate initialSize and what fpGrowth rate could be appropriate. Consider both current and future memory availability and access patterns of the data.</li><li>Create logging or metrics around the scalable Bloom filter behavior, monitoring the number of slices created and memory consumption over time, which can provide insights into usage patterns and need for adjustment.</li><li>Avoid using very small fpGrowth values, as these will lead to frequent addition of filter slices, compounding memory usage and potentially causing performance bottlenecks as the ‘Test’ function needs to check more slices.</li><li>In distributed systems where consistency across nodes is necessary, ensure that all instances are initialized with the same parameters and handle the updates to filter slices identically.</li><li>As with any Bloom filter, using multiple, independent hash functions improves the spread of elements across the bit set. In a Scalable Bloom Filter, these hash functions need to maintain their properties as additional layers are added.</li></ul><h4>Adding and Testing values</h4><pre>// Add inserts the given item into the scalable Bloom filter.<br>// If the current filter slice exceeds its capacity based on the growth rate, a new slice is added.<br>func (sbf *ScalableBloomFilter) Add(data []byte) {<br> // Add the item to all existing filter slices.<br> for _, filter := range sbf.filters {<br>  for i := uint64(0); i &lt; filter.k; i++ {<br>   filter.Add(data)<br>  }<br> }<br><br> // Increment the total number of items added across all filter slices.<br> sbf.n++<br><br> // Check the last filter&#39;s capacity, and if needed, add a new filter slice.<br> // The current implementation may be only adding new filters and not previous ones.<br> // We need to base the condition on the properties of the last filter and the current count of added items.<br> currentFilter := sbf.filters[len(sbf.filters)-1] // Get the most recent filter<br><br> // Use the correct threshold to decide when to add a new filter.<br> // This threshold should be defined by how full the current filter is.<br> currentCapacity := float64(currentFilter.m) * math.Log(sbf.fpGrowth) / math.Log(2)<br> // If the number of items exceeds the current capacity of the filter:<br> if float64(sbf.n) &gt; currentCapacity {<br>  newFpRate := sbf.fpRate * math.Pow(sbf.fpGrowth, float64(len(sbf.filters)))<br>  // Create and append the new filter slice.<br>  nbf, _ := NewBloomFilter(sbf.n, newFpRate)<br>  sbf.filters = append(sbf.filters, nbf)<br> }<br>}<br><br>func (sbf *ScalableBloomFilter) Test(data []byte) bool {<br> // Check the item against all filter slices from the oldest to the newest.<br> for _, filter := range sbf.filters {<br>  // Assume the item is in the filter until proven otherwise.<br>  isPresent := true<br><br>  // Use the same hash functions that were used to add the item.<br>  for i := uint64(0); i &lt; filter.k; i++ {<br>   // If any of the bits corresponding to the item&#39;s hash values are not set, it&#39;s definitely not present in this filter.<br>   if !filter.Test(data) {<br>    isPresent = false<br>    // We break out of the hash function loop as soon as we find a bit that is not set.<br>    break<br>   }<br>  }<br><br>  // If all the bits for this filter are set, then the item is potentially present (with some false positive rate).<br>  if isPresent {<br>   return true<br>  }<br>  // Otherwise, continue checking the next filter to see if the item may be present there.<br> }<br><br> // If none of the filters had all bits set, the item is definitely not in the set.<br> return false<br>}</pre><h4>Adding Elements (Add Function)</h4><ul><li>Distribution Across Layers: When an element is added, it is added to all existing Bloom Filter layers. This ensures that the element is tracked no matter how much the scalable filter grows.</li><li>Item Counting: The filter maintains a count of all elements added (n), which is crucial for knowing when to add a new layer.</li><li>Capacity Check and Expansion:</li><li>The filter checks if the current (most recent) layer has reached its capacity to maintain the desired false positive rate.</li><li>If the current layer is deemed full (based on currentCapacity, which considers the fpGrowth), a new Bloom Filter layer is created and added to the filter.</li><li>This new layer has a higher false positive rate, calculated based on the original fpRate and the number of existing layers. The fpGrowth factor determines how much the false positive rate increases with each new layer.</li></ul><h4>Testing for Elements (Test Function)</h4><ul><li>Layer-by-Layer Testing: To check if an element is in the scalable filter, the Test function checks the element against each Bloom Filter layer, starting from the oldest.</li><li>Early Exit on Positive Find: If any layer reports that the element might be present (all bits corresponding to the element’s hash are set), the function immediately returns true.</li><li>Definitive Absence: If none of the layers indicate the element’s presence, the function returns false, confirming that the element is definitely not in the filter.</li></ul><h4>Managing False Positives</h4><ul><li>Growth Factor: The fpGrowth parameter controls how the false positive rate increases with each new layer. As more layers are added, the likelihood of false positives increases, but this is a trade-off for handling more elements without significantly increasing memory usage per layer.</li><li>Balancing Act: This approach is a balancing act between maintaining a low false positive rate and accommodating an increasing number of elements. The filter allows for the dataset to grow while ensuring that the probability of false positives remains within an acceptable range.</li></ul><h3>Some use cases</h3><ol><li><strong>Network Security:</strong> In cybersecurity, Bloom Filters can quickly check if an IP address or domain is part of a blacklist, efficiently filtering out malicious traffic.</li><li><strong>Web Caching Mechanisms:</strong> Web browsers or proxy servers can use Bloom Filters to determine if a web resource is cached, reducing unnecessary network requests and speeding up web browsing.</li><li><strong>Database Query Optimization:</strong> Databases can employ Bloom Filters to check if a query result is stored in a cache, reducing the load on the database server by avoiding redundant queries.</li><li><strong>Spell Checking:</strong> Word processors and text editing software can use Bloom Filters to quickly check if a word is in a dictionary, offering a fast way to identify misspelled words.</li><li><strong>Sybil Attack Prevention in P2P Networks: </strong>In peer-to-peer (P2P) networks, Bloom Filters can help detect and prevent Sybil attacks by efficiently checking if a node is already part of the network.</li><li><strong>Blockchain and Cryptocurrency:</strong> Blockchain applications, like Bitcoin, use Bloom Filters to efficiently synchronize and verify transactions without downloading the entire blockchain.</li><li><strong>Anti-Fraud Systems:</strong> Financial institutions can use Bloom Filters to screen transactions or account access requests against known fraud patterns or compromised account lists.</li><li><strong>Email Spam and Phishing Protection:</strong> Email servers can utilize Bloom Filters to filter out spam or phishing emails by checking incoming messages against a list of known spam characteristics.</li><li><strong>Bioinformatics:</strong> In genome sequencing, Bloom Filters assist in quickly querying massive DNA databases, helping identify gene sequences without the need for exhaustive database searches.</li><li><strong>Distributed Systems and Big Data:</strong> In large-scale distributed systems, Bloom Filters can efficiently manage data distribution and replication, ensuring that operations like data sharding and load balancing are conducted optimally.</li></ol><h3>Conclusion</h3><p>We’ve delved into the intricacies of implementing both simple and scalable Bloom Filter in Go. These filters, while conceptually straightforward, embody a powerful approach to managing data efficiently and effectively in scenarios where traditional data structures might falter due to size constraints or performance requirements.</p><p>I invite readers to not only understand the theory but also to experiment with these implementations. The world of data is vast and ever-changing, and tools like Bloom Filters are key to navigating this landscape with confidence and skill.</p><p><strong>Happy coding!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d5ac62557de" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/bloom-filters-and-go-1d5ac62557de">Bloom Filters and Go</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consume OpenAI (ChatGPT) With Golang]]></title>
            <link>https://medium.com/better-programming/chatgpt-golang-a7bd524e7563?source=rss-d9e16079e054------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7bd524e7563</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[api]]></category>
            <dc:creator><![CDATA[Francisco Escher]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 03:11:55 GMT</pubDate>
            <atom:updated>2023-09-11T13:46:38.182Z</atom:updated>
            <content:encoded><![CDATA[<h4>My Go client library for the OpenAI API</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KMhsP_5K6dR-pVofxY1OkA.png" /><figcaption>OpenAI and Go logos</figcaption></figure><h3>Introduction</h3><p>Open AI exposes an API that is quite simple to use. It requires its clients to register an API Key with an account on their website and send the API Key in a bearer authentication header.</p><p>The documentation for the API can be found <a href="https://platform.openai.com/docs/introduction">at this link</a>.</p><p>With that in mind, I’ve built a package to implement its methods. I will describe the implementation here, but the full package can be found at <a href="https://github.com/franciscoescher/goopenai">this GitHub link</a>.</p><p>Please, feel free to contribute and use it.</p><h3>Development</h3><p>The calls require an API key, and you might want to specify an organization id as well, so first, we need to instantiate the client:</p><pre>type Client struct {<br> apiKey       string<br> Organization string<br>}<br><br>// NewClient creates a new client<br>func NewClient(apiKey string, organization string) *Client {<br> return &amp;Client{<br>  apiKey:       apiKey,<br>  Organization: organization,<br> }<br>}<br><br>// Post makes a post request<br>func (c *Client) Post(url string, input any) (response []byte, err error) {<br> response = make([]byte, 0)<br><br> rJson, err := json.Marshal(input)<br> if err != nil {<br>  return response, err<br> }<br><br> resp, err := c.Call(http.MethodPost, url, bytes.NewReader(rJson))<br> if err != nil {<br>  return response, err<br> }<br> defer resp.Body.Close()<br><br> response, err = io.ReadAll(resp.Body)<br> return response, err<br>}<br><br>// Get makes a get request<br>func (c *Client) Get(url string, input any) (response []byte, err error) {<br> if input != nil {<br>  vals, _ := query.Values(input)<br>  query := vals.Encode()<br><br>  if query != &quot;&quot; {<br>   url += &quot;?&quot; + query<br>  }<br> }<br><br> resp, err := c.Call(http.MethodGet, url, nil)<br> if err != nil {<br>  return response, err<br> }<br> defer resp.Body.Close()<br><br> response, err = io.ReadAll(resp.Body)<br> return response, err<br>}<br><br>// Call makes a request<br>func (c *Client) Call(method string, url string, body io.Reader) (response *http.Response, err error) {<br> req, err := http.NewRequest(method, url, body)<br> if err != nil {<br>  return response, err<br> }<br><br> req.Header.Add(&quot;Authorization&quot;, &quot;Bearer &quot;+c.apiKey)<br> req.Header.Add(&quot;Content-Type&quot;, &quot;application/json&quot;)<br> if c.Organization != &quot;&quot; {<br>  req.Header.Add(&quot;OpenAI-Organization&quot;, c.Organization)<br> }<br><br> client := &amp;http.Client{}<br> resp, err := client.Do(req)<br> return resp, err<br>}</pre><p>After that, I will create wrappers for the specific endpoints of the API. I will show here the create completions method, as it is the first to be shown in the documentation as well:</p><pre>import (<br> &quot;encoding/json&quot;<br>)<br><br>const COMPLETIONS_URL = &quot;https://api.openai.com/v1/chat/completions&quot;<br><br>type CreateCompletionsRequest struct {<br> Model            string            `json:&quot;model,omitempty&quot;`<br> Messages         []Message         `json:&quot;messages,omitempty&quot;`<br> Prompt           StrArray          `json:&quot;prompt,omitempty&quot;`<br> Suffix           string            `json:&quot;suffix,omitempty&quot;`<br> MaxTokens        int               `json:&quot;max_tokens,omitempty&quot;`<br> Temperature      float64           `json:&quot;temperature,omitempty&quot;`<br> TopP             float64           `json:&quot;top_p,omitempty&quot;`<br> N                int               `json:&quot;n,omitempty&quot;`<br> Stream           bool              `json:&quot;stream,omitempty&quot;`<br> LogProbs         int               `json:&quot;logprobs,omitempty&quot;`<br> Echo             bool              `json:&quot;echo,omitempty&quot;`<br> Stop             StrArray          `json:&quot;stop,omitempty&quot;`<br> PresencePenalty  float64           `json:&quot;presence_penalty,omitempty&quot;`<br> FrequencyPenalty float64           `json:&quot;frequency_penalty,omitempty&quot;`<br> BestOf           int               `json:&quot;best_of,omitempty&quot;`<br> LogitBias        map[string]string `json:&quot;logit_bias,omitempty&quot;`<br> User             string            `json:&quot;user,omitempty&quot;`<br>}<br><br>func (c *Client) CreateCompletionsRaw(r CreateCompletionsRequest) ([]byte, error) {<br> return c.Post(COMPLETIONS_URL, r)<br>}<br><br>func (c *Client) CreateCompletions(r CreateCompletionsRequest) (response CreateCompletionsResponse, err error) {<br> raw, err := c.CreateCompletionsRaw(r)<br> if err != nil {<br>  return response, err<br> }<br><br> err = json.Unmarshal(raw, &amp;response)<br> return response, err<br>}<br><br>type CreateCompletionsResponse struct {<br> ID      string `json:&quot;id,omitempty&quot;`<br> Object  string `json:&quot;object,omitempty&quot;`<br> Created int    `json:&quot;created,omitempty&quot;`<br> Model   string `json:&quot;model,omitempty&quot;`<br> Choices []struct {<br>  Message struct {<br>    Role    string `json:&quot;role,omitempty&quot;`<br>    Content string `json:&quot;content,omitempty&quot;`<br>  } `json:&quot;message&quot;`<br>  Text         string      `json:&quot;text,omitempty&quot;`<br>  Index        int         `json:&quot;index,omitempty&quot;`<br>  Logprobs     interface{} `json:&quot;logprobs,omitempty&quot;`<br>  FinishReason string      `json:&quot;finish_reason,omitempty&quot;`<br> } `json:&quot;choices,omitempty&quot;`<br> Usage struct {<br>  PromptTokens     int `json:&quot;prompt_tokens,omitempty&quot;`<br>  CompletionTokens int `json:&quot;completion_tokens,omitempty&quot;`<br>  TotalTokens      int `json:&quot;total_tokens,omitempty&quot;`<br> } `json:&quot;usage,omitempty&quot;`<br><br> Error *Error `json:&quot;error,omitempty&quot;`<br>}<br><br>// Error is the error standard response from the API<br>type Error struct {<br> Message string      `json:&quot;message,omitempty&quot;`<br> Type    string      `json:&quot;type,omitempty&quot;`<br> Param   interface{} `json:&quot;param,omitempty&quot;`<br> Code    interface{} `json:&quot;code,omitempty&quot;`<br>}<br><br>func (e *Error) Error() string {<br>  return fmt.Sprintf(&quot;%s: %s&quot;, e.Code, e.Message)<br>}<br><br>type Message struct {<br> Role    string `json:&quot;role,omitempty&quot;`<br> Content string `json:&quot;content,omitempty&quot;`<br>}</pre><p>With that, you should be ready to go!</p><p>Below is an example of how you call this method. (Other methods are implemented at the package link in the introduction).</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;os&quot;<br><br> &quot;github.com/franciscoescher/goopenai&quot;<br>)<br><br>func main() {<br> apiKey := os.Getenv(&quot;API_KEY&quot;)<br> organization := os.Getenv(&quot;API_ORG&quot;)<br><br> client := goopenai.NewClient(apiKey, organization)<br><br> r := goopenai.CreateCompletionsRequest{<br>  Model: &quot;gpt-3.5-turbo&quot;,<br>  Messages: []goopenai.Message{<br>   {<br>    Role:    &quot;user&quot;,<br>    Content: &quot;Say this is a test!&quot;,<br>   },<br>  },<br>  Temperature: 0.7,<br> }<br><br> completions, err := client.CreateCompletions(r)<br> if err != nil {<br>  panic(err)<br> }<br><br> fmt.Println(completions)<br> /* Response should be like this<br>  {<br>    &quot;id&quot;: &quot;chatcmpl-xxx&quot;,<br>    &quot;object&quot;: &quot;chat.completion&quot;,<br>    &quot;created&quot;: 1678667132,<br>    &quot;model&quot;: &quot;gpt-3.5-turbo-0301&quot;,<br>    &quot;usage&quot;: {<br>      &quot;prompt_tokens&quot;: 13,<br>      &quot;completion_tokens&quot;: 7,<br>      &quot;total_tokens&quot;: 20<br>    },<br>    &quot;choices&quot;: [<br>      {<br>        &quot;message&quot;: {<br>          &quot;role&quot;: &quot;assistant&quot;,<br>          &quot;content&quot;: &quot;\n\nThis is a test!&quot;<br>        },<br>        &quot;finish_reason&quot;: &quot;stop&quot;,<br>        &quot;index&quot;: 0<br>      }<br>    ]<br>  }<br> */<br>}</pre><p>Run the code using the following:</p><pre>API_KEY=&lt;your-api-key&gt; API_ORG=&lt;your-org-id&gt; go run .</pre><p>Thank you for reading! This is a simple usage of the Open AI API using Golang.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7bd524e7563" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/chatgpt-golang-a7bd524e7563">Consume OpenAI (ChatGPT) With Golang</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consume LinkedIn API with Golang]]></title>
            <link>https://francisco-escher.medium.com/consume-linkedin-api-with-golang-ffe7f7cc201e?source=rss-d9e16079e054------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffe7f7cc201e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[oauth2]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Francisco Escher]]></dc:creator>
            <pubDate>Tue, 21 Feb 2023 23:37:22 GMT</pubDate>
            <atom:updated>2023-03-18T17:36:25.144Z</atom:updated>
            <content:encoded><![CDATA[<h4>Or how to build an OAuth2 client with go</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1kHtixLreqqRpGiEDgZdQ.png" /></figure><h3>TLDR</h3><p>This story presents the implementation of a golang package for calling the LinkedIn API v2 using OAuth2. It can be extrapolated to other OAuth2 APIs.</p><p>The full code can be found at this <a href="https://github.com/franciscoescher/golinkedin">this Github link</a>.</p><p>Currently, it only implements the endpoints /me, /clientAwareMemberHandles, /people, and /emailAddress.</p><h3>Introduction</h3><p>LinkedIn API v2 uses OAuth2 so a user can grant permission over their data to external applications.</p><p>LinkedIn does not provide a native golang package to use this API, but luckily golang has a nice OAuth2 package (golang.org/x/oauth2) to help us make use of it. I will show my implementation here.</p><p>To use this package you will need a LinkedIn App registered (You can do so at <a href="https://developer.linkedin.com/">this link</a>). In that same page, you can get API credentials (client id and secret) for this app, and also access to the OAuth2 scopes you want to use. If you are not sure what are those scopes, take a look at <a href="https://oauth.net/2/scope/">this link</a>.</p><p>You will also need a redirect route so that LinkedIn will redirect the user who grants permission for the application. <strong>This route must be registered as an “Authorized redirect URLs for your app” on your app page, or you will get an error while trying to use it.</strong></p><h3>Development</h3><p>The steps for using the API are the following:</p><h4>1. Decide the type of data you want to query</h4><p>Depending on this type, you will need to use different scopes and endpoints. Check <a href="https://learn.microsoft.com/en-us/linkedin/">this link</a> to know more about them.</p><h4>2. Create an authorization URL</h4><p>Using the OAuth2 go package:</p><pre>conf := &amp;oauth2.Config{<br>  ClientID:     &quot;&lt;client_id&gt;&quot;,<br>  ClientSecret: &quot;&lt;client_secret&quot;,<br>  Scopes:       []string{&quot;r_liteprofile&quot;, &quot;r_emailaddress&quot;},<br>  Endpoint:     oauth2.Endpoint{<br>    AuthURL:   &quot;https://www.linkedin.com/oauth/v2/authorization&quot;,<br>    TokenURL:  &quot;https://www.linkedin.com/oauth/v2/accessToken&quot;,<br>    AuthStyle: oauth2.AuthStyleInParams,<br>   },<br>  RedirectURL:  &quot;https://localhost:8080/auth/linkedin/callback&quot;,<br> }</pre><p>which should result in something like:</p><blockquote><em>https://www.linkedin.com/oauth/v2/authorization?access_type=offline&amp;client_id=%3Cclient_id%3E&amp;redirect_uri=http%3A%2F%2Flocalhost%3A8080%2Fauth%2Flinkedin%2Fcallback&amp;response_type=code&amp;scope=r_liteprofile+r_emailaddress&amp;state=%3Cstate</em></blockquote><p>The user must sign in to LinkedIn and accept the use of their data using this link. He will be redirected to the address in the RedirectURL field.</p><h4>Create an HTTP client using the given code</h4><p>The user will be redirected to the address in the RedirectURL field, with the URL query parameters state and code.</p><p>The state is the same value you passed to generate the auth URL and should be used by you as a CSRF token (not in scope for us here).</p><p>First, retrieve a token using the OAuth2 configuration (with the same parameters you used before). Then, create an HTTP client with that same configuration and token (the first parameter of those methods is a context, you can pass any you prefer, something like context.Background() should work).</p><pre>// code is the code returned by the linkedin callback as a query parameter<br>code := r.URL.Query().Get(&quot;code&quot;)<br>// conf shoud be build with the same parameters as before<br>token, err := conf.Exchange(ctx, code)<br>if err != nil {<br>  return nil, err<br>}<br>client := conf.Client(ctx, token)</pre><p>Note: in case there is any error with the authentication, your callback will be called with the query parameters error and error_description instead.</p><p>Note 2: this use case us a generic use of OAuth2, and some other APIs using it could possibly be implemented using this same approach.</p><h4>Call the desired endpoint using the HTTP client</h4><p>There is not much secret in this step. In the example below I’ve called the emailAddress endpoint and decoded the response into a struct that follows the API format.</p><pre>const EndpointEmailAddress = &quot;https://api.linkedin.com/v2/emailAddress?q=members&amp;projection=(elements*(handle~))&quot;<br><br><br>// response structure<br>type EmailAddress struct {<br> ErrorResponse<br> Elements []struct {<br>  Handle      string `json:&quot;handle&quot;`<br>  HandleTilde struct {<br>   EmailAddress string `json:&quot;emailAddress&quot;`<br>  } `json:&quot;handle~&quot;`<br> } `json:&quot;elements&quot;`<br>}<br><br>// possible error response<br>type ErrorResponse struct {<br> ServiceErrorCode int    `json:&quot;serviceErrorCode&quot;`<br> Message          string `json:&quot;message&quot;`<br> Status           int    `json:&quot;status&quot;`<br>}<br><br>// calls the client<br>resp, err := client.Get(EndpointAddress)<br>if err != nil {<br>  log.Fatal(err)<br>}<br>defer resp.Body.Close()<br><br>// parses the response<br>var r EmailAddress<br>err = json.NewDecoder(resp.Body).Decode(&amp;r)<br>if err != nil {<br>  log.Fatal(err)<br>}<br><br>// prints parsed result<br>fmt.Println(r)</pre><h3>Wrapping up into a package</h3><p>I found it was a cleaner solution to wrap up everything into 2 structs, a Builder and a Client.</p><p>The builder is responsible for creating the auth URL and the Client, as it shows below:</p><pre>package golinkedin<br><br>import (<br> &quot;context&quot;<br><br> &quot;golang.org/x/oauth2&quot;<br>)<br><br>var (<br> oauth2endpoint = oauth2.Endpoint{<br>  AuthURL:   &quot;https://www.linkedin.com/oauth/v2/authorization&quot;,<br>  TokenURL:  &quot;https://www.linkedin.com/oauth/v2/accessToken&quot;,<br>  AuthStyle: oauth2.AuthStyleInParams,<br> }<br>)<br><br>// NewBuilder returns a new linkedin client, not yet authenticated.<br>func NewBuilder(clientID string, clientSecret string, scopes []string, redirectURL string) *Builder {<br> return &amp;Builder{<br>  clientID:     clientID,<br>  clientSecret: clientSecret,<br>  scopes:       scopes,<br>  redirectURL:  redirectURL,<br> }<br>}<br><br>type Builder struct {<br> // ClientID is the api key client&#39;s ID.<br> clientID string<br> // ClientSecret is the api key client&#39;s secret.<br> clientSecret string<br> // Scopes is the list of scopes that the client will request.<br> scopes []string<br> // redirectURL is the URL that the user will be redirected to after<br> // authenticating with linkedin in the GetAuthURL url response.<br> redirectURL string<br>}<br><br>// getOAuth2Config returns the oauth2 config<br>func (c *Builder) getOAuth2Config() *oauth2.Config {<br> conf := &amp;oauth2.Config{<br>  ClientID:     c.clientID,<br>  ClientSecret: c.clientSecret,<br>  Scopes:       c.scopes,<br>  Endpoint:     oauth2endpoint,<br>  RedirectURL:  c.redirectURL,<br> }<br> return conf<br>}<br><br>// GetAuthURL returns the URL to the linkedin login page<br>// The state is a string that will be returned to the redirect URL<br>// so it can be used to prevent CSRF attacks<br>func (c *Builder) GetAuthURL(state string) string {<br> oa2config := c.getOAuth2Config()<br> url := oa2config.AuthCodeURL(state, oauth2.AccessTypeOffline)<br> return url<br>}<br><br>// GetClient will exchange the code for an access token and<br>// use it to create a new client with an authorized http client<br>func (c *Builder) GetClient(ctx context.Context, code string) (*Client, error) {<br> oa2config := c.getOAuth2Config()<br> token, err := oa2config.Exchange(ctx, code)<br> if err != nil {<br>  return nil, err<br> }<br> client := oa2config.Client(ctx, token)<br> new := &amp;Client{*client}<br> return new, nil<br>}</pre><p>The client still should be used to call the API endpoints, but now has the API methods integrated into it, as below:</p><pre>package golinkedin<br><br>import (<br> &quot;net/http&quot;<br> &quot;encoding/json&quot;<br>)<br><br>type Client struct {<br> http.Client<br>}<br><br>// Common struct for error response in linkedin API<br>type ErrorResponse struct {<br> ServiceErrorCode int    `json:&quot;serviceErrorCode&quot;`<br> Message          string `json:&quot;message&quot;`<br> Status           int    `json:&quot;status&quot;`<br>}<br><br>const EndpointEmailAddress = &quot;https://api.linkedin.com/v2/emailAddress?q=members&amp;projection=(elements*(handle~))&quot;<br><br>// EmailAddressRequest calls emailAddress api.<br>// Please note that email address is only available with the scope r_emailaddress.<br>func (c *Client) EmailAddressRequest() (resp *http.Response, err error) {<br> return c.Get(EndpointEmailAddress)<br>}<br><br>// Same as EmailAddressRequest but parses the response.<br>func (c *Client) GetEmailAddress() (r EmailAddress, err error) {<br> resp, err := c.EmailAddressRequest()<br> if err != nil {<br>  return r, err<br> }<br> defer resp.Body.Close()<br> err = json.NewDecoder(resp.Body).Decode(&amp;r)<br> return r, err<br>}<br><br>type EmailAddress struct {<br> ErrorResponse<br> Elements []struct {<br>  Handle      string `json:&quot;handle&quot;`<br>  HandleTilde struct {<br>   EmailAddress string `json:&quot;emailAddress&quot;`<br>  } `json:&quot;handle~&quot;`<br> } `json:&quot;elements&quot;`<br>}</pre><p>And now we can use the package just as simple as below:</p><pre>package main<br><br>import (<br> &quot;context&quot;<br>  &quot;fmt&quot;<br><br>  &quot;github.com/franciscoescher/golinkedin&quot;<br>)<br><br>func main() {<br>  // first, you need to create a builder with your api key credentials<br>  scopes := []string{&quot;r_liteprofile&quot;, &quot;r_emailaddress&quot;}<br>  builder := golinkedin.NewBuilder(&quot;CLIENT_ID&quot;, &quot;CLIENT_SECRET&quot;, scopes, &quot;REDIRECT_URL&quot;)<br><br>  // then, you need to get the auth url to redirect the user to the linkedin login page<br>  url := builder.GetAuthURL(&quot;state&quot;)<br>  fmt.Println(url)<br><br>  // after user accepts, linkedin will redirect to REDIRECT_URL with a code parameter<br>  // you need to use this code to authenticate and get the client<br>  code := &quot;CODE&quot;<br>  client, err := builder.GetClient(context.Background(), code)<br>  if err != nil {<br>    log.Fatal(err)<br>  }<br><br>  // now you can use the client to make requests<br>  response, err := client.GetEmailAddress()<br>  if err != nil {<br>    log.Fatal(err)<br>  }<br>  fmt.Println(response)<br>}</pre><h3>Conclusion</h3><p>Overall, golang provides a quite simple implementation of the OAuth2 protocol. It was quite simple to call the LinkedIn API using it. although the API itself can be quite annoying to understand (especially because of its documentation). But in the end, the package created was quite clean and easy to use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffe7f7cc201e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implement Golang RESTful API Out of the Box]]></title>
            <link>https://medium.com/better-programming/golang-restful-api-out-of-the-box-6875bc19adbe?source=rss-d9e16079e054------2</link>
            <guid isPermaLink="false">https://medium.com/p/6875bc19adbe</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[rest]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Francisco Escher]]></dc:creator>
            <pubDate>Thu, 16 Feb 2023 04:08:40 GMT</pubDate>
            <atom:updated>2023-04-05T17:33:47.616Z</atom:updated>
            <content:encoded><![CDATA[<h4>A quicker way to only describe your data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*GIAhltM0SydS46zJETnqwQ.jpeg" /><figcaption>Go mascot in a box</figcaption></figure><p>This article presents a way to implement Golang’s RESTful API out of the box, and it will only require you to create the configuration files/structs that describe your data.</p><p>This article was created using the database/SQL connection for the database, logrus for logging, and go-playground/validation/v10 for input validation, but changing is just a matter of implementing interfaces on your current choices.</p><p>The full implementation (examples included) can be found at <a href="https://github.com/franciscoescher/gosimplerest">this GitHub link</a>.</p><h3>Introduction</h3><p>I’ve been developing Golang backends for web applications for a while now, and once you get an idea of your application, a lot of the work seems to be pretty standard.</p><p>It usually goes like that for me:</p><ol><li>Understand the requirements of your feature(s) and how it is translated to your data structures. This step is more of a design step rather than manual code development.</li><li>Create a struct representing your resources (your data formats, such as tables, files, streams, or whatever). Be sure to choose the field types properly to match the database and to add tags to them to represent JSON mapping and validation.</li><li>Create a model layer that implements database operations (insert, modify, query, or delete data).</li><li>Choose a router or framework for your server.</li><li>Create handlers that your server will use to handle requests to the application (this includes parameter validation and database operations).</li><li>Create middleware to act on non-core functions of your handlers (such as auth, logging, CSRF check, CORS configuration, etc.).</li><li>Register the endpoints that will be mapped to the handlers (those are your routes) and serve them.</li></ol><p>For RESTful APIs, more specifically, many of the structure and details of your routes follow the REST standards, and many times you won’t need much more than simply a simple CRUD (Create, Retrieve, Update, and Delete).</p><p>With that in mind, I thought it might be useful to remove some of the repetitive steps above for the sake of rapid development.</p><h3>First, the Results</h3><p>The package created with this idea produced the desired routes with only the following steps:</p><ol><li>Understand the requirements of your feature(s) and how it is translated to your data structures. (we cannot avoid this step).</li><li>Create a struct that represents your resources.</li><li>Call a function that registers the routes for you and then run your server with it.</li></ol><p>The package created for this experiment can be found at <a href="https://github.com/franciscoescher/gosimplerest">this GitHub link</a>.</p><p>I’ve had to choose some packages to handle logging, validation, and, most importantly, the database connection. I’ve chosen logrus, go-playground/validator/10, and database/SQL.</p><p>The handlers I’ve created followed the http.HandlerFunc interface, so most routers and frameworks can accept them. The code below shows the implementation for Gin, but I also included one for Gorilla Mux, Chi, Fiber, and Echo.</p><pre>package main<br><br>import (<br>  &quot;database/sql&quot;<br>  &quot;fmt&quot;<br>  &quot;os&quot;<br>  <br>  &quot;github.com/franciscoescher/gosimplerest&quot;<br>  &quot;github.com/gin-gonic/gin&quot;<br>  &quot;github.com/go-playground/validator/v10&quot;<br>  <br>  &quot;github.com/go-sql-driver/mysql&quot;<br>  &quot;github.com/sirupsen/logrus&quot;<br>  &quot;gopkg.in/guregu/null.v3&quot;<br>)<br><br>// Mysql Schema<br>/*<br>DROP TABLE IF EXISTS `users`;<br><br>CREATE TABLE `users` (<br> `uuid` varchar(191) NOT NULL,<br> `created_at` datetime(3) DEFAULT NULL,<br> `deleted_at` datetime(3) DEFAULT NULL,<br> `first_name` varchar(255) DEFAULT NULL,<br> `phone` varchar(255) DEFAULT NULL,<br> PRIMARY KEY (`uuid`)<br>);<br>*/<br><br>var UserResource = gosimplerest.Resource{<br>  Name:      &quot;users&quot;,<br>  PrimaryKey: &quot;uuid&quot;,<br>  Fields: map[string]gosimplerest.Field{<br>    &quot;uuid&quot;:       {},<br>    &quot;first_name&quot;: {},<br>    &quot;phone&quot;:      {},<br>    &quot;created_at&quot;: {},<br>    &quot;deleted_at&quot;: {},<br>  },<br>  SoftDeleteField: null.NewString(&quot;deleted_at&quot;, true),<br>}<br><br>func main() {<br>  logger := logrus.New()<br>  <br>  logger.Info(&quot;starting application&quot;)<br>  <br>  db := getDB()<br>  defer db.Close()<br>  <br>  // create routes for rest api<br>  r := gin.Default()<br>  resources := []gosimplerest.Resource{UserResource}<br>  params := gosimplerest.AddHandlersBaseParams{Logger: logger, Resources: resources, Respository: mysqlRepo.NewRepository(db)}<br>  gosimplerest.AddGinHandlers(r, params)<br>  <br>  logrus.Fatal(r.Run(&quot;:3333&quot;))<br>}<br><br>func getDB() *sql.DB {<br>  c := mysql.Config{<br>    User:                 os.Getenv(&quot;DB_USER&quot;),<br>    Passwd:               os.Getenv(&quot;DB_PASSWORD&quot;),<br>    Net:                  &quot;tcp&quot;,<br>    Addr:                 os.Getenv(&quot;DB_HOSTNAME&quot;) + &quot;:&quot; + os.Getenv(&quot;DB_PORT&quot;),<br>    DBName:               os.Getenv(&quot;DB_SCHEMA&quot;),<br>    ParseTime:            true,<br>    AllowNativePasswords: true,<br>  }<br>  <br>  db, err := sql.Open(&quot;mysql&quot;, c.FormatDSN())<br>    if err != nil {<br>    panic(err)<br>  }<br>  <br>  return db<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_A_31liZKcXmsnnHlAbCng.png" /><figcaption>The server running the automatically created routes</figcaption></figure><h3>Data Abstraction</h3><p>From every resource we want to expose, we need some basic information that will allow us to interact with and serve it. Those are:</p><ul><li>The name of the resource (the database table, for instance)</li><li>The fields of the data</li><li>Which field is the primary key (id)</li></ul><p>I’ve also worked with the possible existence of some standard date-time-related fields:</p><ul><li>date/time of creation</li><li>date/time of update</li><li>date/time of deletion — for soft deletes</li></ul><p>This is the resulting structure:</p><pre>// gosimplerest.Resource represents a data resource, such as<br>// a database table, a file in a storage system, etc.<br>type Resource struct {<br>  // Name of the resource<br>  Name string `json:&quot;name&quot;`<br>  // OverwriteTableName is the name of the table in case it is not the same as the resource name<br>  OverwriteTableName null.String `json:&quot;overwrite_table_name&quot;`<br>  // Fields is a list of fields in the table<br>  Fields map[string]Field `json:&quot;fields&quot;`<br>  // PrimaryKey is the name of field that is the primary key<br>  PrimaryKey string `json:&quot;primary_key&quot;`<br>  // AutoIncrementalPK is a flag that indicates if the primary key is autoincremental<br>  // and will not use it when inserting a new row<br>  AutoIncrementalPK bool `json:&quot;incremental_pk&quot;`<br>  // GeneratePrimaryKeyFunc is a function that generates a new primary key<br>  // if null, the defaultGeneratePrimaryKeyFunc is used<br>  GeneratePrimaryKeyFunc GeneratePrimaryKeyFunc `json:&quot;-&quot;`<br>  // SoftDeleteField is the name of the field that is used for soft deletes<br>  // if null, no soft deletes are used<br>  SoftDeleteField null.String `json:&quot;soft_delete_field&quot;`<br>  // CreatedAtField is the name of the field that is used as createion timestamp<br>  // if null, no creation timestamp is generated<br>  CreatedAtField null.String `json:&quot;created_at_field&quot;`<br>  // UpdatedAtField is the name of the field that is used as update timestamp<br>  // if null, no update timestamp is generated<br>  UpdatedAtField null.String `json:&quot;updated_at_field&quot;`<br>  // Ommmit&lt;Route Type&gt;Route are flags that omit the generation of the specific route from the router<br>  OmitCreateRoute        bool `json:&quot;omit_create_route&quot;`<br>  OmitRetrieveRoute      bool `json:&quot;omit_retrieve_route&quot;`<br>  OmitUpdateRoute        bool `json:&quot;omit_update_route&quot;`<br>  OmitPartialUpdateRoute bool `json:&quot;omit_partial_update_route&quot;`<br>  OmitDeleteRoute        bool `json:&quot;omit_delete_route&quot;`<br>  OmitSearchRoute        bool `json:&quot;omit_search_route&quot;`<br>  OmitHeadRoutes         bool `json:&quot;omit_head_routes&quot;`<br>}<br><br>type GeneratePrimaryKeyFunc func() any<br><br>type Field struct {<br>  // Validator is the validation rules for the field<br>  Validator string `json:&quot;validator&quot;`<br>  // Unsearchable is a flag that indicates that a field can not be used<br>  // as query parameter in the search route<br>  Unsearchable bool `json:&quot;unsearchable&quot;`<br>  // Immutable is a flag that indicates that a field can not be updated<br>  Immutable bool `json:&quot;immutable&quot;`<br>}</pre><p>You will notice I am also adding flags to control which route I want to omit in the API for the sake of a bit of customization.</p><p>This relation is used since it’s pretty common for APIs to expose routes such as /user/{id}/cars, indicating that we want to retrieve the cars of a user, instead of implementing the equivalent query /cars?user_id={id}.</p><h3>Building the Handlers</h3><p>The handlers use the http.HandlerFunc interface, so they will not present much news to the regular backend Golang developer, except for some minor tweaks:</p><ul><li>There is no implementation for reading parameters from the URL (like in /users/{id} (for the get and delete endpoints), so I am passing them to the request context with the context.WithValue function.</li></ul><pre>type GinContextKey int<br><br>const (<br>  ContextKeyParams GinContextKey = iota<br>)<br><br>// GetRequestWithParams returns a new request with the given params in the request context<br>// The params are stored with the key defined by ContextKeyParams<br>func GetRequestWithParams(r *http.Request, params map[string]string) *http.Request {<br>  if r.Context() == nil {<br>    r = r.WithContext(context.Background())<br>  }<br>  return r.WithContext(context.WithValue(r.Context(), ContextKeyParams, params))<br>}<br><br>// ReadParams returns the value of the given params previously written in<br>// the request context with the key defined by ContextKeyParams<br>func ReadParams(r *http.Request, s string) string {<br>  params := r.Context().Value(ContextKeyParams)<br>  if params == nil {<br>   return &quot;&quot;<br>  }<br>  values, ok := r.Context().Value(ContextKeyParams).(map[string]string)<br>  if !ok {<br>   return &quot;&quot;<br>  }<br>  val, ok := values[s]<br>  if !ok {<br>   return &quot;&quot;<br>  }<br>  return val<br>}</pre><ul><li>The normal “go way” of reading the body of the requests is to define a struct with the proper fields and tags and decode the JSON body into it. In our case, we want to be data format agnostic so that we will be representing them as maps map[string]interface{} instead.</li></ul><pre>// unmarshalBody converts the body of the request to a map where<br>// the keys are the field names and the values are the field values<br>func unmarshalBody(r *http.Request) (map[string]any, error) {<br>  b := new(bytes.Buffer)<br>  _, err := b.ReadFrom(r.Body)<br>  if err != nil {<br>    return nil, err<br>  }<br>  var objmap map[string]any<br>  err = json.Unmarshal(b.Bytes(), &amp;objmap)<br>  return objmap, err<br>}</pre><p>The following code shows the implementation of one of the handlers:</p><pre>package handlers<br><br>import (<br>  &quot;net/http&quot;<br>)<br><br>// RetrieveHandler returns a handler for the GET method<br>func RetrieveHandler(params *GetHandlerFuncParams) http.HandlerFunc {<br>  return func(w http.ResponseWriter, r *http.Request) {<br>    id := ReadParams(r, &quot;id&quot;)<br><br>    // validates id<br>    err := params.Resource.ValidateField(params.Validate, params.Resource.PrimaryKey, id)<br>    if err != nil {<br>      params.Logger.Error(err)<br>      w.WriteHeader(http.StatusBadRequest)<br>      return<br>    }<br><br>    result, err := params.Repository.Find(params.Resource, id)<br>    if err != nil {<br>      params.Logger.Error(err)<br>      w.WriteHeader(http.StatusInternalServerError)<br>      return<br>    }<br><br>    if len(result) == 0 {<br>      w.WriteHeader(http.StatusNotFound)<br>      err = encodeJsonError(w, r, &quot;not found&quot;)<br>      if err != nil {<br>        params.Logger.Error(err)<br>        w.WriteHeader(http.StatusInternalServerError)<br>        return<br>      }<br>      return<br>    }<br><br>    err = encodeJson(w, r, result)<br>    if err != nil {<br>      params.Logger.Error(err)<br>      w.WriteHeader(http.StatusInternalServerError)<br>      return<br>    }<br>  }<br>}</pre><h3>The Data Layer</h3><p>Here we face two dilemmas: we don’t want to depend on the database type or the data format to implement generic functions that interact with data.</p><p>For the first one, we implement a repository layer with an interface as shown below:</p><pre>package repository<br><br>import &quot;github.com/franciscoescher/gosimplerest/resource&quot;<br><br>type RepositoryInterface interface {<br>  // Delete deletes a row with the given primary key from the database<br>  Delete(b *resource.Resource, id any) error<br>  // Find returns a single row from the database, search by the primary key<br>  // return 0 rows if not found, but no error<br>  Find(b *resource.Resource, id any) (map[string]any, error)<br>  // Insert inserts a new row into the database<br>  // returns pk only if auto incremental<br>  Insert(b *resource.Resource, data map[string]any) (int64, error)<br>  // Search searches for rows in the database using the query parameters<br>  // returns 0 rows if not found, but no error<br>  // query is a map of field names and values<br>  // multiple values for the same field are ORed<br>  Search(b *resource.Resource, query map[string][]string) ([]map[string]any, error)<br>  // Update updates a row in the database<br>  // One of the fields must be the primary key or it will return an error<br>  // Returns true if the a row was updated, false if not found<br>  Update(b *resource.Resource, data map[string]any) (bool, error)<br>}</pre><p>The second one is solved by implementing a struct for such an interface. Luckily, our resource struct contains all the necessary information to construct our queries, and we can work with maps and interfaces to store the incoming data.</p><pre>type Repository struct {<br>  db *sql.DB<br>}<br><br>func (r Repository) Find(b *resource.Resource, id any) (map[string]any, error) {<br>  fields := b.GetFieldNames()<br><br>  sqlStatement := concatStr(`SELECT `, strings.Join(fields, &quot;,&quot;), ` FROM `, b.Table(), ` WHERE `, b.PrimaryKey, ` = ? LIMIT 1`)<br>  response := r.db.QueryRow(sqlStatement, id)<br><br>  values := make([]any, len(b.Fields))<br>  scanArgs := make([]any, len(b.Fields))<br>  for i := range values {<br>    scanArgs[i] = &amp;values[i]<br>  }<br><br>  err := response.Scan(scanArgs...)<br>  if err != nil {<br>    if err == sql.ErrNoRows {<br>      return make(map[string]any, 0), nil<br>    }<br>    return make(map[string]any, 0), err<br>  }<br><br>  return r.parseRow(b, values)<br>}</pre><p>The challenge lies in parsing the rows and casting them to the correct types, which is done by the following function (this might need to be expanded if new types are required):</p><pre>// parseRow parses a row from the database, returning a map with<br>// the field names as keys and the values as values<br>func (r Repository) parseRow(b *resource.Resource, values []any) (map[string]any, error) {<br>  fields := b.GetFieldNames()<br>  result := make(map[string]any, len(b.Fields))<br>  for i, v := range values {<br>    casted, err := r.castVal(v)<br>    if err != nil {<br>      return result, fmt.Errorf(&quot;failed on if for type %T of %v&quot;, v, v)<br>    }<br>    result[fields[i]] = casted<br>  }<br>  return result, nil<br>}<br><br>// castVal casts the value incomming from the database to a valid type<br>func (r Repository) castVal(v any) (any, error) {<br>  // if nil, set to nil<br>  if v == nil {<br>    return nil, nil<br>  }<br><br>  n3, ok := v.(int64)<br>  if ok {<br>    logrus.Info(n3)<br>    return n3, nil<br>  }<br><br>  n2, ok := v.(float64)<br>  if ok {<br>    logrus.Info(n2)<br>    return n2, nil<br>  }<br><br>  // bool or string<br>  x, ok := v.([]byte)<br>  if ok {<br>    if p, ok := strconv.ParseBool(string(x)); ok == nil {<br>      return p, nil<br>    } else {<br>      return string(x), nil<br>    }<br>  }<br><br>  t, ok := v.(time.Time)<br>  if ok {<br>    return t, nil<br>  }<br><br>  return nil, fmt.Errorf(&quot;failed on if for type %T of %v&quot;, v, v)<br>}</pre><h3>Adding the Handlers to the Router</h3><p>After that, there is no big mystery: we only need to add the handlers to our chosen router.</p><p>The function below will do this implementation:</p><pre>// AddHandlers adds the routes to the router<br>func AddHandlers(params AddHandlersParams) {<br>  if params.Validator == nil {<br>    params.Validator = &amp;validator.BlankValidator{}<br>  }<br>  if params.Logger == nil {<br>    params.Logger = &amp;logger.BlankLogger{}<br>  }<br>  for i := range params.Resources {<br>    p := &amp;handlers.GetHandlerFuncParams{<br>      Logger:     params.Logger,<br>      Validate:   params.Validator,<br>      Resource:   &amp;params.Resources[i],<br>      Repository: params.Respository,<br>    }<br>    var sb strings.Builder<br>    sb.WriteString(&quot;/&quot;)<br>    sb.WriteString(strcase.KebabCase(params.Resources[i].Table()))<br>    name := sb.String()<br>    nameID := params.AddParamFunc(name, &quot;id&quot;)<br><br>    h := params.AddRouteFunctions<br>    if !params.Resources[i].OmitCreateRoute {<br>      h.Post(name, handlers.CreateHandler(p))<br>    }<br>    if !params.Resources[i].OmitRetrieveRoute {<br>      h.Get(nameID, handlers.RetrieveHandler(p))<br>    }<br>    if !params.Resources[i].OmitUpdateRoute {<br>      h.Put(name, handlers.UpdateHandler(p))<br>    }<br>    if !params.Resources[i].OmitPartialUpdateRoute {<br>      h.Patch(name, handlers.UpdateHandler(p))<br>    }<br>    if !params.Resources[i].OmitDeleteRoute {<br>      h.Delete(nameID, handlers.DeleteHandler(p))<br>    }<br>    if !params.Resources[i].OmitSearchRoute {<br>      h.Get(name, handlers.SearchHandler(p))<br>    }<br>    if !params.Resources[i].OmitHeadRoutes {<br>      h.Head(nameID, handlers.RetrieveHandler(p))<br>      h.Head(name, handlers.SearchHandler(p))<br>    }<br>  }<br>}<br><br>// AddRouteFunc is type of a function that adds a route<br>// to a router, with a given name and handler.<br>// It should also add the params to the handler.<br>// Each router/framework should implement this function.<br>type AddRouteFunc func(name string, h http.HandlerFunc)<br><br>// AddParamFunc is a function that adds a param to a url/endpoint.<br>// It should return the route with the param added.<br>// Example: /users + id -&gt; /users/:id<br>// Each router/framework should implement this function.<br>type AddParamFunc func(name string, param string) string<br><br>// AddRouteFunctions is a struct that contains the functions<br>// to add routes to a router, one for each request method.<br>// Each router/framework should implement this struct.<br>type AddRouteFunctions struct {<br>  Post   AddRouteFunc<br>  Get    AddRouteFunc<br>  Put    AddRouteFunc<br>  Patch  AddRouteFunc<br>  Delete AddRouteFunc<br>  Head   AddRouteFunc<br>}<br><br>type AddHandlersBaseParams struct {<br>  Resources   []resource.Resource<br>  Respository repository.RepositoryInterface<br>  Validator   validator.Validator<br>  Logger      logger.Logger<br>}<br><br>type AddHandlersParams struct {<br>  AddHandlersBaseParams<br>  AddRouteFunctions AddRouteFunctions<br>  AddParamFunc      AddParamFunc<br>}</pre><p>Note that it requires a parameter that specifies your resource, repository, logger, and validator (the last two are not described in this story but are interfaces that will perform logging and input validation for your handlers).</p><p>It also requires two more parameters: AddParamFunc, which is a function that adds parameters to a route, and AddRouteFunctions, which is a function that adds routes to a router engine. Those depend on the framework you use, so we need an implementation for each framework we want to adapt.</p><p>For instance, below there is an implementation for the Gin framework (in the GitHub repo, I’ve added other frameworks that are popular as well):</p><pre>func AddGinHandlers(r *gin.Engine, base AddHandlersBaseParams) *gin.Engine {<br>  params := AddHandlersParams{<br>    AddHandlersBaseParams: base,<br>    AddRouteFunctions: AddRouteFunctions{<br>      Post:   GinAddRouteFunc(r.POST),<br>      Get:    GinAddRouteFunc(r.GET),<br>      Put:    GinAddRouteFunc(r.PUT),<br>      Patch:  GinAddRouteFunc(r.PATCH),<br>      Delete: GinAddRouteFunc(r.DELETE),<br>      Head:   GinAddRouteFunc(r.HEAD),<br>    },<br>    AddParamFunc: func(name string, param string) string {<br>      var sb strings.Builder<br>      sb.WriteString(name)<br>      sb.WriteString(&quot;/:&quot;)<br>      sb.WriteString(param)<br>      return sb.String()<br>    },<br>  }<br>  AddHandlers(params)<br>  return r<br>}<br><br>// GinAddRouteType is the type of the function that gin.Engine uses to add routes to the router.<br>// Example: r.POST, r.GET...<br>type GinAddRouteType func(relativePath string, handlers ...gin.HandlerFunc) gin.IRoutes<br><br>// GinAddRouteFunc uses the f function to add a route to the router,<br>// wrapping the handler to add params to request context.<br>func GinAddRouteFunc(f GinAddRouteType) AddRouteFunc {<br>  return func(name string, h http.HandlerFunc) {<br>    f(name, GinHandlerWrapper(h))<br>  }<br>}<br><br>// GinHandlerWrapper converts a http.HandlerFunc to a gin.HandlerFunc<br>// It adds params to request context.<br>func GinHandlerWrapper(h http.HandlerFunc) gin.HandlerFunc {<br>  return func(c *gin.Context) {<br>    params := make(map[string]string, 0)<br>    for _, param := range c.Params {<br>      params[param.Key] = param.Value<br>    }<br>    handlers.AddParamsToHandlerFunc(h, params)(c.Writer, c.Request)<br>  }<br>}</pre><p>In this case, AddParamFunc build the endpoint with a parameter like /users/:id, which is how Gin expects it to be, and AddRouteFunctions will call Gin’s native handler functions.</p><h3>Making It More “Go-Friendly”</h3><p>So you tell me: “Hey, Francisco, I am used to writing Golang using structs and tags. What can I do?” OK, let’s implement that concept as well.</p><pre>user := resource.Resource{}<br>type Users struct {<br>  UUID      string    `json:&quot;uuid&quot; validate:&quot;required&quot; pk:&quot;true&quot;`<br>  FirstName string    `json:&quot;first_name&quot; validate:&quot;required&quot;`<br>  Phone     string    `json:&quot;phone&quot; validate:&quot;required&quot;`<br>  CreatedAt time.Time `json:&quot;created_at&quot; created_at:&quot;true&quot;`<br>  DeletedAt null.Time `json:&quot;deleted_at&quot; soft_delete:&quot;true&quot;`<br>}<br>err := user.FromStruct(Users{})<br>if err != nil {<br>  logrus.Fatal(err)<br>}</pre><p>The method FromStruct receives a struct and populates the resource instance fields with information from the struct name, fields, and tags.</p><pre>/*<br>FromStruct reads a struct and populates the resource<br><br>The struct should have the following tags:<br>  - db: used to get the field name<br>  - json: used to get the field name (if db is not present)<br>  - pk: used to get the primary key field name:<br>    set to autoincremental if the primary key is autoincremental<br>    set to true if the primary key is not autoincremental<br>  - soft_delete: used to get the soft delete field<br>  - created_at: used to get the created at field<br>  - updated_at: used to get the updated at field<br>  - validate: used to get the validation rules<br>  - unsearchable: used to get the unsearchable fields<br>  - pk: used to get the primary key<br><br>The omit route flags, OverwriteTableName and GeneratePrimaryKeyFunc are not populated by this function<br>*/<br>func (b *Resource) FromStruct(s any) error {<br>  // Table name<br>  t := reflect.TypeOf(s)<br>  if t.Kind() != reflect.Struct {<br>    return fmt.Errorf(&quot;not struct type: %s&quot;, t.Kind())<br>  }<br>  b.Name = strcase.SnakeCase(t.Name())<br>  b.OverwriteTableName = null.NewString(&quot;&quot;, false)<br><br>  // Fields<br>  // iterate over fields<br>  fields := make(map[string]Field, t.NumField())<br>  for i := 0; i &lt; t.NumField(); i++ {<br>    field := t.Field(i)<br>    // get the field name<br>    name := field.Tag.Get(&quot;json&quot;)<br>    db := field.Tag.Get(&quot;db&quot;)<br>    if db != &quot;-&quot; &amp;&amp; db != &quot;&quot; {<br>      name = db<br>    } else if db == &quot;&quot; {<br>      json := field.Tag.Get(&quot;json&quot;)<br>      if json != &quot;-&quot; &amp;&amp; json != &quot;&quot; {<br>        name = json<br>      } else if json == &quot;&quot; {<br>        name = strcase.SnakeCase(field.Name)<br>      }<br>    }<br>    presentOrTrue := func(tag string) bool {<br>      val, ok := field.Tag.Lookup(tag)<br>      return ok &amp;&amp; (val == &quot;&quot; || val == &quot;true&quot;)<br>    }<br>    // get the field struct<br>    fields[name] = Field{<br>      Validator:    field.Tag.Get(&quot;validate&quot;),<br>      Immutable:    presentOrTrue(&quot;immutable&quot;),<br>      Unsearchable: presentOrTrue(&quot;unsearchable&quot;),<br>    }<br>    // get the primary key<br>    if presentOrTrue(&quot;pk&quot;) {<br>      b.PrimaryKey = name<br>    } else if field.Tag.Get(&quot;pk&quot;) == &quot;autoincremental&quot; {<br>      b.PrimaryKey = name<br>      b.AutoIncrementalPK = true<br>    }<br>    // get the soft delete field<br>    if presentOrTrue(&quot;soft_delete&quot;) {<br>      b.SoftDeleteField = null.StringFrom(name)<br>    }<br>    // get the created at field<br>    if presentOrTrue(&quot;created_at&quot;) {<br>      b.CreatedAtField = null.StringFrom(name)<br>    }<br>    // get the updated at field<br>    if presentOrTrue(&quot;updated_at&quot;) {<br>      b.UpdatedAtField = null.StringFrom(name)<br>    }<br>  }<br>  b.Fields = fields<br><br>  return nil<br>}</pre><p>If we want to do it similarly using JSON files instead of hardcoded structs, it’s even easier:</p><pre>// FromJSON reads a JSON file and populates the model<br>func (b *Resource) FromJSON(filename string) error {<br>  file, err := os.ReadFile(filename)<br>  if err != nil {<br>    return err<br>  }<br>  err = json.Unmarshal([]byte(file), &amp;b)<br>  if err != nil {<br>    return err<br>  }<br>  return nil<br>}</pre><pre>{<br>  &quot;name&quot;: &quot;users&quot;,<br>  &quot;primary_key&quot;: &quot;uuid&quot;,<br>  &quot;fields&quot;: {<br>    &quot;uuid&quot;: {&quot;validator&quot;: &quot;uuid&quot;},<br>    &quot;first_name&quot;: {},<br>    &quot;phone&quot;: {},<br>    &quot;created_at&quot;: {},<br>    &quot;deleted_at&quot;: {}<br>  },<br> &quot;soft_delete_field&quot;: &quot;deleted_at&quot;,<br> &quot;omit_search_route&quot;: true<br>}</pre><h3>Conclusion</h3><p>This experiment showed it’s possible to cut some steps in developing a Golang API, implementing a generic REST CRUD application that serves data from a SQL database.</p><p>While this code hasn’t been used in production, it might be a good choice when implementing microservices with low customization requirements.</p><p>This code still has a lot of space for expansion (accepting other data sources, for instance).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6875bc19adbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/golang-restful-api-out-of-the-box-6875bc19adbe">Implement Golang RESTful API Out of the Box</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>