<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Danny Hawkins on Medium]]></title>
        <description><![CDATA[Stories by Danny Hawkins on Medium]]></description>
        <link>https://medium.com/@dannyhawkins?source=rss-87e1572c08b8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4gET-SP43-9SVec29c2O0g@2x.jpeg</url>
            <title>Stories by Danny Hawkins on Medium</title>
            <link>https://medium.com/@dannyhawkins?source=rss-87e1572c08b8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:12:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dannyhawkins/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Monolith, SOA, Microservices: When to Use What]]></title>
            <link>https://blog.stackademic.com/monolith-soa-microservices-when-to-use-what-afdac137b169?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/afdac137b169</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Tue, 05 Sep 2023 03:55:50 GMT</pubDate>
            <atom:updated>2023-09-05T09:41:58.194Z</atom:updated>
            <content:encoded><![CDATA[<p>There has been a lot of hype around microservice architecture in the last decade. More often than not, discussions focus on its numerous advantages. However, diving into a microservices architecture without careful consideration and awareness of its complexities could set you up for a nasty surprise.</p><p>In this article I want to discuss the pros and cons of 4 different architectural approaches, the <strong>tightly coupled monolith</strong>, the <strong>loosly coupled monolith</strong>, <strong>service oriented architecture (SOA)</strong> and <strong>microservices</strong>, as well as my recommendations on when to use what and tooling / approaches that can help</p><h3>Tightly Coupled Monolith</h3><p>A tightly coupled monolith refers to a software architecture where different components or functionalities are deeply interwoven and dependent on each other within a single codebase. Changes or failures in one component can affect others, making the system less modular and flexible. This architecture contrasts with more decoupled or modular systems where components interact through well-defined interfaces and can often be developed, scaled, or replaced independently.</p><h3>Pros</h3><ul><li><strong>Simplicity</strong>: Monolithic applications are often easier to develop initially since everything is in one place, without the need for complex communication between services.</li><li><strong>Single Deployment</strong>: You deploy a single application, which can simplify the deployment process and avoid complexities of managing multiple services.</li><li><strong>Atomic Changes</strong>: Changes can be made atomically, meaning everything can be updated at once. This can reduce inconsistencies and version mismatch issues.</li><li><strong>Tight Integration</strong>: All components are naturally integrated, so there’s less chance of incompatibilities between modules.</li><li><strong>Efficiency</strong>: There’s no network overhead of calling between different services, which can boost performance.</li><li><strong>Simplified Testing</strong>: Testing can be more straightforward since you’re only dealing with a single application.</li><li><strong>Transactional Simplicity</strong>: Transactions across modules are simpler to manage without having to handle distributed transactions.</li><li><strong>No Service Coordination</strong>: There’s no need to manage coordination or communication between multiple services.</li><li><strong>Immediate Consistency</strong>: Data consistency is often easier to ensure with a monolithic architecture.</li><li><strong>Simplified Scaling</strong>: Initially, it’s often simpler to scale a monolithic application by just replicating the entire application.</li></ul><h3>Cons</h3><ul><li><strong>Limited Scalability</strong>: Monolithic applications can become a bottleneck, especially as they grow, making it harder to scale specific parts of the app.</li><li><strong>Lengthy Builds</strong>: As the codebase grows, build and deployment times can become significantly longer.</li><li><strong>High Impact of Failures</strong>: If one part of the system fails, the entire system could go down.</li><li><strong>Inflexibility</strong>: Making changes or adding features can be cumbersome as the codebase grows.</li><li><strong>Barrier to New Technologies</strong>: It’s challenging to integrate new technologies or frameworks into a large monolithic codebase.</li><li><strong>Tough Parallel Development</strong>: When many developers work on the same codebase, it can lead to merge conflicts and integration issues.</li><li><strong>Hard to Understand</strong>: A large codebase can be overwhelming and challenging to understand for new developers.</li><li><strong>Slower Iteration</strong>: Larger monolithic systems can slow down the development cycle and make continuous delivery or continuous integration harder.</li><li><strong>Technical Debt</strong>: Over time, the monolith can accumulate more technical debt, leading to more complexities and making refactoring challenging.</li><li><strong>Vendor Lock-in</strong>: Monolithic architectures might lead to being locked into specific technology choices made early in the project.</li></ul><h3>Loosley Coupled Monolith</h3><p>A loosely coupled, modular monolith refers to a software architecture where the application is organized into distinct and independent modules within a single codebase. These modules interact through well-defined interfaces, ensuring minimal dependencies between them. While still a monolithic application in terms of deployment, its design emphasizes modularity and separation of concerns, allowing for easier maintenance, testing, and potential scaling of individual modules without affecting others.</p><h3>Pros</h3><ul><li><strong>Modularity</strong>: Encourages cleaner code organization based on functionality or domain, which improves maintainability.</li><li><strong>Isolation</strong>: Failures or bugs in one module are less likely to affect other parts of the application.</li><li><strong>Easier Scaling</strong>: Specific modules that require more resources can be scaled independently, even within the confines of a monolithic deployment.</li><li><strong>Flexibility</strong>: Modules can be developed, replaced, or refactored without affecting other parts of the system significantly.</li><li><strong>Easier Parallel Development</strong>: Different teams can work on different modules with minimal interference.</li><li><strong>Reusability</strong>: Modules can be reused across different parts of the application or even in other projects.</li><li><strong>Optimized Performance</strong>: Individual modules can be optimized for their specific tasks without compromising the performance of other modules.</li><li><strong>Incremental Refactoring</strong>: Transitioning to a different architecture, like microservices, can be done incrementally, one module at a time.</li><li><strong>Better Testability</strong>: Individual modules can be tested in isolation, which can improve test coverage and reliability.</li><li><strong>Tech Stack Flexibility</strong>: Different modules can potentially be written in different technologies or frameworks, provided they expose consistent interfaces.</li></ul><h3>Cons</h3><ul><li><strong>Complexity</strong>: While more modular than tightly coupled monoliths, they can still grow complex over time.</li><li><strong>Interface Management</strong>: The interfaces between modules must be carefully managed to ensure compatibility.</li><li><strong>Deployment Overhead</strong>: While it’s simpler than deploying many microservices, it might be more complex than deploying a tightly coupled monolith.</li><li><strong>Potential for Hidden Coupling</strong>: Poor discipline can lead to hidden dependencies and unintentional coupling.</li><li><strong>Integration Testing</strong>: Ensuring that all modules work harmoniously can require extensive integration testing.</li><li><strong>Performance Overhead</strong>: Inter-module communication, even within a single codebase, might introduce some performance overhead.</li><li><strong>Learning Curve</strong>: Developers might need to adapt to the modularity and the best practices required to maintain loose coupling.</li><li><strong>Possible Inconsistencies</strong>: If not managed well, different modules might implement slightly different behaviors or data models.</li><li><strong>Deployment Consistency</strong>: Ensuring all modules are compatible during deployment can introduce challenges, especially when modules are being updated at different rates.</li><li><strong>Transition Challenges</strong>: Moving from a tightly coupled monolith to a modular one can be a time-consuming process.</li></ul><h3>Service Oriented Architecture (SOA)</h3><p>Service-Oriented Architecture (SOA) is a software design pattern where components provide services to other components through well-defined interfaces and protocols, typically over a network. It emphasizes modularity, interoperability, and reusability, allowing different services to communicate seamlessly and facilitating easier integration of diverse systems.</p><h3>Pros</h3><ul><li><strong>Scalability</strong>: Services can be independently scaled based on demand, allowing for more efficient use of resources.</li><li><strong>Flexibility</strong>: New services or changes to existing ones can be introduced without disrupting other services or clients.</li><li><strong>Reusability</strong>: Services are designed to be reused across different applications or parts of the same application.</li><li><strong>Interoperability</strong>: SOA often employs standard communication protocols, allowing for integration with a wide range of services and platforms.</li><li><strong>Modularity</strong>: SOA promotes modularity in design, which can lead to better organized and more maintainable systems.</li><li><strong>Improved Fault Isolation</strong>: If one service fails, it does not necessarily mean the entire system will fail.</li><li><strong>Parallel Development</strong>: Different teams can work on separate services simultaneously, speeding up development.</li><li><strong>Location Independence</strong>: Services can be located anywhere, facilitating distributed development and deployment.</li><li><strong>Optimized Maintenance</strong>: Fixing or updating a specific service can be done without taking down the entire system.</li><li><strong>Easier Integration</strong>: Integrating with third-party systems or legacy applications can be more straightforward with SOA, especially when standards like SOAP or REST are used.</li></ul><h3>Cons</h3><ul><li><strong>Complexity</strong>: Introducing services, especially if overdone, can add a level of complexity in terms of setup, maintenance, and monitoring.</li><li><strong>Performance Overhead</strong>: The constant communication between services, especially if over a network, can introduce latency.</li><li><strong>Increased Network Dependency</strong>: Relying heavily on network communication can introduce potential points of failure.</li><li><strong>Security Concerns</strong>: More communication endpoints mean more potential entry points for malicious actors.</li><li><strong>Data Consistency</strong>: Managing data consistency across services can be challenging, especially in distributed systems.</li><li><strong>Versioning Issues</strong>: As services evolve, ensuring that different versions of services interact correctly can become problematic.</li><li><strong>Potential for Tight Coupling</strong>: Poorly designed services can become tightly coupled, defeating one of the main purposes of SOA.</li><li><strong>Testing Challenges</strong>: Testing in an SOA environment can be more complicated, especially when considering integration testing across services.</li><li><strong>Deployment and Monitoring</strong>: Deploying and monitoring multiple services can require more sophisticated tools and processes.</li><li><strong>Initial Setup Time</strong>: Setting up an SOA, especially in environments not used to it, can require a significant initial time investment.</li></ul><h3>Microservices</h3><p>Microservices is an architectural style that structures an application as a collection of small, independent services, each focused on a specific business capability. These services are loosely coupled, can be developed, deployed, and scaled independently, and they communicate with each other using lightweight protocols, typically over HTTP. This approach aims to enhance scalability, flexibility, and maintainability compared to traditional monolithic architectures.</p><h3>Pros</h3><ul><li><strong>Scalability</strong>: Individual microservices can be scaled independently based on their specific demand.</li><li><strong>Flexibility in Technology Stack</strong>: Different microservices can use different technologies, frameworks, and data stores.</li><li><strong>Resilience</strong>: A failure in one service doesn’t necessarily bring down the entire system.</li><li><strong>Easier Deployment and Rollback</strong>: Changes or fixes can be deployed for a single service without affecting the entire system. If issues arise, rollbacks affect only the updated service.</li><li><strong>Organizational Alignment</strong>: Microservices can align well with small development teams, each taking ownership of specific services.</li><li><strong>Granularity</strong>: Focused on single business capabilities, which can lead to more manageable and understandable codebases.</li><li><strong>Independent Development and Release Cycles</strong>: Each service can be developed, deployed, and versioned independently.</li><li><strong>Optimized for Continuous Deployment/Delivery</strong>: Fits well with modern CI/CD pipelines, allowing frequent releases.</li><li><strong>Improved Fault Isolation</strong>: A problematic service can be isolated and dealt with without affecting others.</li><li><strong>Potential for Reusability</strong>: Some services might be reusable across different projects or parts of the system.</li></ul><h3>Cons</h3><ul><li><strong>Complexity</strong>: Introduces complexities in terms of service coordination, data consistency, and network reliance.</li><li><strong>Network Latency</strong>: Inter-service communication over a network can introduce latency.</li><li><strong>Data Consistency Challenges</strong>: With each service managing its own database, ensuring data consistency can be complex, especially without distributed transactions.</li><li><strong>Operational Overhead</strong>: Requires sophisticated monitoring, logging, and alerting to keep track of numerous services.</li><li><strong>Service Discovery</strong>: As services scale or change, mechanisms are needed to track and locate them.</li><li><strong>Integration Testing</strong>: Testing the interactions between multiple services can be challenging.</li><li><strong>Deployment Complexity</strong>: Requires advanced deployment orchestration tools and practices.</li><li><strong>Potential for Overengineering</strong>: Not every application needs a microservices approach, and using it unnecessarily can introduce unneeded complexity.</li><li><strong>Versioning</strong>: Managing different versions of services and ensuring compatibility can be tricky.</li><li><strong>Security Concerns</strong>: More services mean more potential points of attack. Ensuring secure communication and isolation can be demanding.</li></ul><h3>Wait, so whats the difference between SOA and Microservices</h3><p>Service-Oriented Architecture (SOA) and microservices are both architectural patterns that emphasize building applications as collections of services. However, they have different design philosophies, characteristics, and implementations. Here’s a comparison to highlight the differences:</p><p>Scope and Granularity:</p><ul><li><strong>SOA:</strong> SOA often revolves around enterprise-level services, which might lead to larger, more encompassing services often referred to as “enterprise services”. These services might group multiple functionalities.</li><li><strong>Microservices:</strong> As the name implies, microservices focus on breaking down the application into small, single-functionality services. Each service typically corresponds to a specific business capability.</li></ul><p>Communication:</p><ul><li><strong>SOA:</strong> Traditionally uses Enterprise Service Bus (ESB) for communication which can handle routing, transformation, and applying business rules. This could sometimes introduce a centralized bottleneck.</li><li><strong>Microservices:</strong> Tends to use simpler communication mechanisms like HTTP/REST or lightweight messaging queues. The idea is to keep inter-service communication decoupled and direct.</li></ul><p>Data Storage:</p><ul><li><strong>SOA:</strong> Services in SOA might share a centralized database.</li><li><strong>Microservices:</strong> Each microservice typically manages its own database to ensure decoupling and maintain its bounded context.</li></ul><p>Size and Complexity:</p><ul><li><strong>SOA:</strong> Components in SOA might be larger and can be heterogeneous in nature.</li><li><strong>Microservices:</strong> Components are smaller, focused, and are typically homogeneous in terms of technology and design.</li></ul><p>Service Independence:</p><ul><li><strong>SOA:</strong> Though services are modular, they might still have dependencies on shared components like the ESB.</li><li><strong>Microservices:</strong> Services are designed to be fully independent, capable of being developed, deployed, and scaled individually.</li></ul><p>Development and Deployment:</p><ul><li><strong>SOA:</strong> Might have a more centralized governance and often requires coordinated deployments.</li><li><strong>Microservices:</strong> Advocates for decentralized governance, allowing individual teams to develop, deploy, and scale their services independently.</li></ul><p>Failure Isolation:</p><ul><li><strong>SOA:</strong> A failure in the ESB or a core service can impact multiple consumers.</li><li><strong>Microservices:</strong> Services are isolated, so the failure of one microservice (in theory) shouldn’t impact others.</li></ul><p>Technology Stack:</p><ul><li><strong>SOA:</strong> Might be more prescriptive in terms of technology due to centralized governance.</li><li><strong>Microservices:</strong> Provides flexibility for individual services to choose the technology stack that fits best for their requirements.</li></ul><h3>When to use what</h3><p>There are many factors to consider when deciding which of these approaches to adopt. While some factors are more evident than others, let’s delve into specific profiles:</p><h3>Startup</h3><p>When starting a new company, it’s best to opt for the simplest approach. This will facilitate rapid development, from MVP to your fully realized initial product. Given that startups often have small development teams, it’s unlikely you’d want separate product teams for every aspect of your product or service.</p><p>A loosely-coupled modular monolith is advisable here. But why not a tightly-coupled monolith? In my view, there’s seldom a reason to develop a tightly-coupled monolith. It’s essential to conceptualize distinct modules from the outset, even if they aren’t perfect initially, refactoring is always an option. Overlooking boundaries at the beginning with the notion of revisiting them later can cause substantial issues down the line.</p><p>With a loosely-coupled monolith, you benefit from simpler development and deployment processes, all while avoiding challenges like networking, versioning, and transactional consistency. Plus, the modular approach lays the groundwork for transitioning into SOA or microservices as your business scales.</p><h3>Re-architecture</h3><p>If you’re contemplating transitioning from a monolith to a more distributed system, I’d advocate for SOA. Employ domain-driven design (DDD) to pinpoint your business’s domains. A heuristic I favor is to identify areas of your business that could standalone with a unique value proposition. For instance, at Quiqup, we designated “fleet” as a domain, covering all facets concerning driver fleet management — a domain that could very well be an independent business.</p><p>When transitioning to SOA or microservices, it’s crucial to establish contracts between services. This could entail agreed-upon JSON/Avro schemas for integration events or well-defined OpenAPI specs. Proper contracts accelerate development and facilitate testing at the entry and exit points of individual systems.</p><h3>Many teams / domains</h3><p>If you’re in the advantageous position of overseeing multiple teams and domains, your choice should oscillate between SOA and microservices. These architectures empower teams with autonomy and the authority over their software, granting them the liberty to decide how to best deliver business value. However, managing numerous teams or services necessitates robust platform architecture. A software catalog can be invaluable at this juncture.</p><p>Proficient platform engineering and DevOps practices are vital when working with microservices, and even SOA depending on your service count. If you lack effective mechanisms to handle:</p><ul><li>New service creation</li><li>Service Versioning</li><li>Contract / API management</li><li>Dependency versioning</li><li>Network failures</li></ul><p>You may encounter challenges with SOA or microservices.</p><h3>Tooling</h3><p>Here are some intriguing projects I’ve stumbled upon recently, which could potentially mitigate the pain points of microservices:</p><h3>Service Weaver</h3><p><em>Note: Golang only!</em></p><p><a href="https://serviceweaver.dev/docs.html">Service Weaver</a> courtesy of Google, is a framework for designing, deploying, and managing distributed applications. It facilitates local application testing and debugging and offers effortless cloud deployment.</p><p>The platform allows for single-repo code development that can be deployed as separate services. Thus, you enjoy the convenience of a monolith during development and the advantages of microservices during deployment. Moreover, it supports comprehensive local testing spanning all concerns.</p><h3>Temporal.io</h3><p><a href="https://temporal.io/">temporal.io</a> is an open-source programming paradigm designed to streamline your code, bolster application reliability, and expedite feature delivery.</p><p>I’m currently navigating through the Temporal 101 and 102 courses, but my initial impressions are positive. Temporal accommodates multiple programming languages and oversees task coordination, albeit all code execution remains local. More details are available on their <a href="https://temporal.io/how-temporal-works">website</a>.</p><p><em>Originally published at </em><a href="https://blog.dannyhawkins.me/posts/monolith-soa-microservices/"><em>https://blog.dannyhawkins.me/posts/monolith-soa-microservices/</em></a><em>.</em></p><p><em>Thank you for reading until the end. Please consider following the writer and this publication. Visit </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><em> to find out more about how we are democratizing free programming education around the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afdac137b169" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/monolith-soa-microservices-when-to-use-what-afdac137b169">Monolith, SOA, Microservices: When to Use What</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang Testing Using Docker Services via dockertest]]></title>
            <link>https://blog.stackademic.com/golang-testing-using-docker-services-via-dockertest-c70fc9dcba57?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c70fc9dcba57</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[integration-testing]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Sun, 03 Sep 2023 01:05:58 GMT</pubDate>
            <atom:updated>2023-09-04T00:50:10.706Z</atom:updated>
            <content:encoded><![CDATA[<p>During my path learning go so far I have come across some amazing libraries and utilites, one of my favourite for integration testing is <a href="https://github.com/ory/dockertest">dockertest</a>.</p><p>Whenever I am using a service backed by postgres, mongo, mysql or other services that are not part of my codebase, I generally create a <a href="https://docs.docker.com/compose/">docker-compose</a> file so that my development environments are isolated. Then when I’m working in a particular project, all I have to do is docker-compose up -d to get going, and docker-compose down when I&#39;m finished for the day.</p><p>For example if I just need a postgres database I would usually have something like</p><pre>services:<br>  postgres:<br>    image: postgres:15<br>    ports:<br>      - 5432:5432<br>    volumes:<br>      - pg-db:/var/lib/postgresql/data<br>    environment:<br>      POSTGRES_HOST_AUTH_METHOD: trust<br>      POSTGRES_DB: example<br><br>volumes:<br>  pg-data:</pre><p>This is great for development, but when running integration tests, if I want a clean database is means I need some additional steps. I can either:</p><ul><li>Manually drop and create the database</li><li>Drop and create the database as part of the integration test</li><li>Many other options</li></ul><p>Instead of creating some automation to drop and create databases or tables, dockertest allows us to create a completley clean isolated instance of the service, so we know every time we are starting from nothing. This follows the <a href="https://geektechstuff.com/2021/06/24/devops-what-does-cattle-not-pets-mean/">cattle not pets</a> idealology from devops.</p><p>We are just discussing postgres here, but some other docker services might require a LOT manual steps to get into a good condition for test runs.</p><h3>Example</h3><p>I created an example project <a href="https://github.com/danhawkins/go-dockertest-example">here</a> to show everything in action, but I will work through each step I took.</p><p>The project I created is very simple and the main work is in the following code:</p><pre>package database<br><br>import (<br>	&quot;fmt&quot;<br>	&quot;log&quot;<br>	&quot;os&quot;<br><br>	&quot;gorm.io/driver/postgres&quot;<br>	&quot;gorm.io/gorm&quot;<br>)<br><br>type (<br>	Person struct {<br>		gorm.Model<br>		Name string<br>		Age  uint<br>	}<br>)<br><br>var db *gorm.DB<br><br>func Connect() {<br>	log.Println(&quot;Setting up the database&quot;)<br><br>	pgUrl := fmt.Sprintf(&quot;postgresql://postgres@127.0.0.1:%s/example&quot;, os.Getenv(&quot;POSTGRES_PORT&quot;))<br>	log.Printf(&quot;Connecting to %s\n&quot;, pgUrl)<br>	var err error<br><br>	db, err = gorm.Open(postgres.Open(pgUrl), &amp;gorm.Config{})<br><br>	if err != nil {<br>		panic(&quot;failed to connect database&quot;)<br>	}<br><br>	// Migrate the schema<br>	db.AutoMigrate(&amp;Person{})<br>}<br><br>func CreatePerson() {<br>	log.Println(&quot;Creating a new person in the database&quot;)<br>	person := Person{Name: &quot;Danny&quot;, Age: 42}<br>	db.Create(&amp;person)<br><br>	log.Println(&quot;Trying to write a new person to the database&quot;)<br>}<br><br>func CountPeople() int {<br>	var count int64<br>	db.Model(&amp;Person{}).Count(&amp;count)<br>	return int(count)<br>}</pre><p>So we have a method to connect, a method to create a new person record and a method to count the records. These are all called from the main entry point</p><pre>func main() {<br>	database.Connect()<br><br>	database.CreatePerson()<br><br>	count := database.CountPeople()<br><br>	log.Printf(&quot;Database has %d people&quot;, count)<br>}</pre><p>If I run this from the command line (once docker compose is up) the result increments the count on every run (as it should)</p><pre>go-dockertest-example λ git main → go run main.go<br>2023/09/03 13:41:06 Setting up the database<br>2023/09/03 13:41:06 Connecting to postgresql://postgres@127.0.0.1:5432/example<br>2023/09/03 13:41:06 Creating a new person in the database<br>2023/09/03 13:41:06 Trying to write a new person to the database<br>2023/09/03 13:41:06 Database has 3 people<br><br>go-dockertest-example λ git main → go run main.go<br>2023/09/03 13:41:07 Setting up the database<br>2023/09/03 13:41:07 Connecting to postgresql://postgres@127.0.0.1:5432/example<br>2023/09/03 13:41:08 Creating a new person in the database<br>2023/09/03 13:41:08 Trying to write a new person to the database<br>2023/09/03 13:41:08 Database has 4 people</pre><h3>The Test</h3><p>Now lets look at the test. First there is the setup of dockertest:</p><pre>func TestMain(m *testing.M) {<br>	// Start a new docker pool<br>	pool, err := dockertest.NewPool(&quot;&quot;)<br>	if err != nil {<br>		log.Fatalf(&quot;Could not construct pool: %s&quot;, err)<br>	}<br><br>	// Uses pool to try to connect to Docker<br>	err = pool.Client.Ping()<br>	if err != nil {<br>		log.Fatalf(&quot;Could not connect to Docker: %s&quot;, err)<br>	}<br><br>	pg, err := pool.RunWithOptions(&amp;dockertest.RunOptions{<br>		Repository: &quot;postgres&quot;,<br>		Tag:        &quot;15&quot;,<br>		Env: []string{<br>			&quot;POSTGRES_DB=example&quot;,<br>			&quot;POSTGRES_HOST_AUTH_METHOD=trust&quot;,<br>			&quot;listen_addresses = &#39;*&#39;&quot;,<br>		},<br>	}, func(config *docker.HostConfig) {<br>		// set AutoRemove to true so that stopped container goes away by itself<br>		config.AutoRemove = true<br>		config.RestartPolicy = docker.RestartPolicy{<br>			Name: &quot;no&quot;,<br>		}<br>	})<br><br>	if err != nil {<br>		log.Fatalf(&quot;Could not start resource: %s&quot;, err)<br>	}<br><br>	pg.Expire(10)<br><br>	// Set this so our app can use it<br>	postgresPort := pg.GetPort(&quot;5432/tcp&quot;)<br>	os.Setenv(&quot;POSTGRES_PORT&quot;, postgresPort)<br><br>	// Wait for the Postgres to be ready<br>	if err := pool.Retry(func() error {<br>		_, connErr := gorm.Open(postgres.Open(fmt.Sprintf(&quot;postgresql://postgres@localhost:%s/example&quot;, postgresPort)), &amp;gorm.Config{})<br>		if connErr != nil {<br>			return connErr<br>		}<br><br>		return nil<br>	}); err != nil {<br>		panic(&quot;Could not connect to postgres: &quot; + err.Error())<br>	}<br><br>	code := m.Run()<br><br>	os.Exit(code)<br>}</pre><p>So first we create a new docker pool, we make sure the pool responds to a ping then start a postgres instance. When you start a new instance it will grab a random port, so you need a way to pass this to the service, this is why we have the POSTGRES_PORT env var. At the very end of the module setup we set that env so that test will use it.</p><pre>// Set this so our app can use it<br>	postgresPort := pg.GetPort(&quot;5432/tcp&quot;)<br>	os.Setenv(&quot;POSTGRES_PORT&quot;, postgresPort)</pre><p>There is a section for waiting for postgres to be ready, you can use this in different ways, but basically you are using some condition to test the instance is ready. This could be anything from checking a port is addressable, to calling a http healthcheck.</p><pre>// Wait for the Postgres to be ready<br>	if err := pool.Retry(func() error {<br>		_, connErr := gorm.Open(postgres.Open(fmt.Sprintf(&quot;postgresql://postgres@localhost:%s/example&quot;, postgresPort)), &amp;gorm.Config{})<br>		if connErr != nil {<br>			return connErr<br>		}<br><br>		return nil<br>	}); err != nil {<br>		panic(&quot;Could not connect to postgres: &quot; + err.Error())<br>	}</pre><p>Then there is:</p><pre> // Make sure we expire the instance after 10 seconds<br> postgres.Expire(10)// Make sure we expire the instance after 10 seconds</pre><p>this ensures if cleanup does not succeed we will remove the docker instance after 10 seconds regardless.</p><p>Finally we have the actual test</p><pre>func TestCreatePerson(t *testing.T) {<br>	// Connect to the database<br>	database.Connect()<br><br>	// Create a person in the database<br>	database.CreatePerson()<br><br>	// Check that the person was created<br>	count := database.CountPeople()<br><br>	if count != 1 {<br>		t.Errorf(&quot;Expected 1 person to be in the database, got %d&quot;, count)<br>	}<br>}</pre><p>Each time the test runs there will only be one record in the database.</p><p><em>Originally published at </em><a href="https://blog.dannyhawkins.me/posts/go-dockertest-example/"><em>https://blog.dannyhawkins.me</em></a><em> on September 3, 2023.</em></p><p><em>Thank you for reading until the end. Please consider following the writer and this publication. Visit </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><em> to find out more about how we are democratizing free programming education around the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c70fc9dcba57" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/golang-testing-using-docker-services-via-dockertest-c70fc9dcba57">Golang Testing Using Docker Services via dockertest</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A simple theme switcher in React for Tailwind CSS]]></title>
            <link>https://blog.stackademic.com/a-simple-theme-switcher-in-react-for-tailwind-css-5667373089d3?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5667373089d3</guid>
            <category><![CDATA[tailwind-css]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[css]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Mon, 21 Aug 2023 01:06:14 GMT</pubDate>
            <atom:updated>2023-08-22T08:49:39.155Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cQytVRc72kROQ1Nx" /><figcaption>Photo by <a href="https://unsplash.com/@pankajpatel?utm_source=medium&amp;utm_medium=referral">Pankaj Patel</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve been building a lot of projects using <a href="https://react.dev/">React</a> and <a href="https://tailwindcss.com/">TailwindCSS</a> recently, being a big fan of dark mode interfaces, I wanted a simple light / dark switcher for by project. When I went searching I couldn’t seem to find anything simple. So I tried something myself and am fairly happy with the result.</p><p>All code for this example can be found on my <a href="https://github.com/danhawkins/react-tailwind-themeswitch">github</a></p><h3>Create the app</h3><p>Create a react app with vite:</p><pre>yarn create vite <br><br>✔ Project name: ... react-tailwind-themeswitch <br>✔ Select a framework: › React <br>✔ Select a variant: › TypeScript + SWC</pre><h3>Install tailwindcss</h3><p>Follow the <a href="https://tailwindcss.com/docs/guides/vite">guide</a> from tailwind site (adjusted for yarn)</p><pre>yarn add -D tailwindcss postcss autoprefixer yarn tailwindcss init -p</pre><h3>Configure Tailwind</h3><p>Use the config provided from the tailwind site for vite, plus add the darkMode option</p><pre>/** @type {import(&#39;tailwindcss&#39;).Config} */<br>export default {<br>  darkMode: &#39;class&#39;,<br>  content: [&#39;./index.html&#39;, &#39;./src/**/*.{js,ts,jsx,tsx}&#39;],<br>  theme: {<br>    extend: {},<br>  },<br>  plugins: [],<br>}</pre><p>Replace index.css</p><pre>@tailwind base;<br>@tailwind components;<br>@tailwind utilities;</pre><h3>Create a basic layout</h3><p>We need a basic layout that supports dark mode, remove the contents of App.css as we won&#39;t be needing them, and replace App.tsx with the following</p><pre>// App.tsx<br>import React from &#39;react&#39;<br><br>function App() {<br>  return (<br>    &lt;div className=&#39;app bg-slate-200 dark:bg-slate-900 h-screen w-full p-6 transition-colors duration-500 ease-in-out&#39;&gt;<br>      &lt;div className=&#39;flex flex-col items-center h-full justify-between&#39;&gt;<br>        &lt;h1 className=&#39;text-slate-800 dark:text-slate-200&#39;&gt;Hello World&lt;/h1&gt;<br>        &lt;div className=&#39;self-end&#39;&gt;This will be the theme switch&lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  )<br>}<br><br>export default App</pre><p>What you should be looking at now</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MJZhgW-Mde0nIprA.png" /></figure><h3>The theme switch component</h3><p>Add a new component in src/ThemeSwitch.tsx, our theme switch adds a couple of depedencies so lets install them also</p><pre>yarn add @headlessui/react @heroicons/react usehooks-ts</pre><p>These dependencies provide unstyled accessible components from <a href="https://headlessui.com/">headless ui</a>, icons from <a href="https://heroicons.com/">heroicons</a> and <a href="https://usehooks-ts.com/">common hooks</a> with typescript support.</p><p>This switch component itself was from <a href="https://tailwindui.com/">tailwindui.com</a></p><pre>// ThemeSwitch.tsx<br>import { useEffect, useState } from &#39;react&#39;<br>import { Switch } from &#39;@headlessui/react&#39;<br>import { SunIcon } from &#39;@heroicons/react/24/solid&#39;<br>import { useLocalStorage } from &#39;usehooks-ts&#39;<br><br>function classNames(...classes: string[]) {<br>  return classes.filter(Boolean).join(&#39; &#39;)<br>}<br><br>function ThemeSwitch() {<br>  const [theme, setTheme] = useLocalStorage(&#39;theme&#39;, &#39;light&#39;)<br><br>  useEffect(() =&gt; {<br>    document.body.classList.remove(&#39;light&#39;, &#39;dark&#39;)<br>    document.body.classList.add(theme)<br>  }, [theme])<br><br>  const [enabled, setEnabled] = useState(theme == &#39;light&#39;)<br><br>  const handleThemeChange = (enabled: boolean) =&gt; {<br>    setTheme(enabled ? &#39;light&#39; : &#39;dark&#39;)<br>    setEnabled(enabled)<br>  }<br><br>  return (<br>    &lt;Switch<br>      checked={enabled}<br>      onChange={handleThemeChange}<br>      className={classNames(<br>        enabled ? &#39;bg-gray-400&#39; : &#39;bg-yellow-600&#39;,<br>        &#39;relative inline-flex h-6 w-11 flex-shrink-0 cursor-pointer rounded-full border-2 border-transparent transition-colors duration-200 ease-in-out&#39;<br>      )}<br>    &gt;<br>      &lt;span className=&#39;sr-only&#39;&gt;Use setting&lt;/span&gt;<br>      &lt;span<br>        className={classNames(<br>          enabled ? &#39;translate-x-5&#39; : &#39;translate-x-0&#39;,<br>          &#39;pointer-events-none relative inline-block h-5 w-5 transform rounded-full bg-white shadow ring-0 transition duration-200 ease-in-out&#39;<br>        )}<br>      &gt;<br>        &lt;span<br>          className={classNames(<br>            enabled<br>              ? &#39;opacity-0 duration-100 ease-out&#39;<br>              : &#39;opacity-100 duration-200 ease-in&#39;,<br>            &#39;absolute inset-0 flex h-full w-full items-center justify-center transition-opacity&#39;<br>          )}<br>          aria-hidden=&#39;true&#39;<br>        &gt;<br>          &lt;SunIcon className=&#39;h-3 w-3 text-gray-400&#39; /&gt;<br>        &lt;/span&gt;<br>        &lt;span<br>          className={classNames(<br>            enabled<br>              ? &#39;opacity-100 duration-200 ease-in&#39;<br>              : &#39;opacity-0 duration-100 ease-out&#39;,<br>            &#39;absolute inset-0 flex h-full w-full items-center justify-center transition-opacity&#39;<br>          )}<br>          aria-hidden=&#39;true&#39;<br>        &gt;<br>          &lt;SunIcon className=&#39;h-3 w-3 text-yellow-600&#39; /&gt;<br>        &lt;/span&gt;<br>      &lt;/span&gt;<br>    &lt;/Switch&gt;<br>  )<br>}<br><br>export default ThemeSwitch</pre><p>Lets walk through whats happening</p><p>We are using local storage to persist the setting for light or dark mode</p><pre>const [theme, setTheme] = useLocalStorage(&#39;theme&#39;, &#39;light&#39;)</pre><p>We’re setting the body class to dark or light based on the setting</p><pre>useEffect(() =&gt; {<br>    document.body.classList.remove(&#39;light&#39;, &#39;dark&#39;)<br>    document.body.classList.add(theme)<br>  }, [theme])</pre><p>We are keeping the state of the button toggle being on or off seperatly to the light dark</p><pre>const [enabled, setEnabled] = useState(theme == &#39;light&#39;)<br><br>  const handleThemeChange = (enabled: boolean) =&gt; {<br>    setTheme(enabled ? &#39;light&#39; : &#39;dark&#39;)<br>    setEnabled(enabled)<br>  }</pre><h3>Integrate the component into the view</h3><p>Finally we just need to update App.tsx</p><pre>import React from &#39;react&#39;<br>import ThemeSwitch from &#39;./ThemeSwitch&#39;<br><br>function App() {<br>  return (<br>    &lt;div className=&#39;app bg-slate-200 dark:bg-slate-900 h-screen w-full p-6 transition-colors duration-500 ease-in-out&#39;&gt;<br>      &lt;div className=&#39;flex flex-col items-center h-full justify-between&#39;&gt;<br>        &lt;h1 className=&#39;text-slate-800 dark:text-slate-200&#39;&gt;Hello World&lt;/h1&gt;<br>        &lt;div className=&#39;self-end&#39;&gt;<br>          &lt;ThemeSwitch /&gt;<br>        &lt;/div&gt;<br>      &lt;/div&gt;<br>    &lt;/div&gt;<br>  )<br>}<br><br>export default App</pre><p>And thats it, we now have a simple theme switcher</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/0*eVVRbc3OYWlo48ur.gif" /></figure><p><em>Originally published at </em><a href="https://blog.dannyhawkins.me/posts/theme-toggle-react-tailwind/"><em>https://blog.dannyhawkins.me</em></a><em> on August 21, 2023.</em></p><p><em>Thank you for reading until the end. Please consider following the writer and this publication. Visit </em><a href="https://stackademic.com/"><em>Stackademic</em></a><em> to find out more about how we are democratizing free programming education around the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5667373089d3" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/a-simple-theme-switcher-in-react-for-tailwind-css-5667373089d3">A simple theme switcher in React for Tailwind CSS</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embed Vite React in Golang binary, with live reload!]]></title>
            <link>https://blog.stackademic.com/golang-with-vite-react-with-live-reload-9e929099752d?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e929099752d</guid>
            <category><![CDATA[full-stack]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[vitejs]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Sun, 13 Aug 2023 03:42:04 GMT</pubDate>
            <atom:updated>2024-11-23T13:44:53.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*97yx13MLNR0ouyvIVXlPdw.png" /></figure><p>I recently followed <a href="https://dev.to/pacholoamit/one-of-the-coolest-features-of-go-embed-reactjs-into-a-go-binary-41e9">this excellent article</a> from <a href="https://dev.to/pacholoamit">Pacholo Amit</a>, he documents a method where you can embed the static assets built from Vite using the Echo framework in go, into a single binary.</p><p>Whilst his solution was great, it was missing something still for me, I like to be able to make changes in code / css and have it instantly updated in the browser, being able to see the frontend change with live reload is a must have</p><p>I created an <a href="https://github.com/danhawkins/go-vite-react-example/tree/b3998c3f5824d4d79f8963331cfb8c8d20b970cc">example project of my own</a> that starts of using his method but then adds a live reloading method in the first PR <a href="https://github.com/danhawkins/go-vite-react-example">https://github.com/danhawkins/go-vite-react-example</a></p><h3>The Solution</h3><p>I solved it in <a href="https://github.com/danhawkins/go-vite-react-example/pull/1">this pull request</a> where we run the vite dev server alongside a proxy mode of the echo server in golang, meaning that we proxy all requests that are relevant directly to the vite server, but we only do this in dev mode.</p><p>In our case dev mode is defined by the environment variable <strong>ENV</strong> being “dev”.</p><p>The first step is to create a .env file</p><pre>ENV=dev</pre><p>Then we update the frontend.go file likes so</p><pre>package frontend<br><br>import (<br> &quot;embed&quot;<br> &quot;log&quot;<br> &quot;net/url&quot;<br> &quot;os&quot;<br><br> _ &quot;github.com/joho/godotenv/autoload&quot;<br> &quot;github.com/labstack/echo/v4&quot;<br> &quot;github.com/labstack/echo/v4/middleware&quot;<br>)<br><br>var (<br> //go:embed dist/*<br> dist embed.FS<br><br> //go:embed dist/index.html<br> indexHTML embed.FS<br><br> distDirFS     = echo.MustSubFS(dist, &quot;dist&quot;)<br> distIndexHTML = echo.MustSubFS(indexHTML, &quot;dist&quot;)<br>)<br><br>func RegisterHandlers(e *echo.Echo) {<br> if os.Getenv(&quot;ENV&quot;) == &quot;dev&quot; {<br>  log.Println(&quot;Running in dev mode&quot;)<br>  setupDevProxy(e)<br>  return<br> }<br> // Use the static assets from the dist directory<br> e.FileFS(&quot;/&quot;, &quot;index.html&quot;, distIndexHTML)<br> e.StaticFS(&quot;/&quot;, distDirFS)<br> // This is needed to serve the index.html file for all routes that are not /api/*<br> // neede for SPA to work when loading a specific url directly<br> e.Use(middleware.StaticWithConfig(middleware.StaticConfig{<br>  Skipper: func(c echo.Context) bool {<br>   // Skip the proxy if the prefix is /api<br>   return len(c.Path()) &gt;= 4 &amp;&amp; c.Path()[:4] == &quot;/api&quot;<br>  },<br>  // Root directory from where the static content is served.<br>  Root: &quot;/&quot;,<br>  // Enable HTML5 mode by forwarding all not-found requests to root so that<br>  // SPA (single-page application) can handle the routing.<br>  HTML5:      true,<br>  Browse:     false,<br>  IgnoreBase: true,<br>  Filesystem: http.FS(distDirFS),<br> }))<br>}<br><br>func setupDevProxy(e *echo.Echo) {<br> url, err := url.Parse(&quot;http://localhost:5173&quot;)<br> if err != nil {<br>  log.Fatal(err)<br> }<br> // Setep a proxy to the vite dev server on localhost:5173<br> balancer := middleware.NewRoundRobinBalancer([]*middleware.ProxyTarget{<br>  {<br>   URL: url,<br>  },<br> })<br> e.Use(middleware.ProxyWithConfig(middleware.ProxyConfig{<br>  Balancer: balancer,<br>  Skipper: func(c echo.Context) bool {<br>   // Skip the proxy if the prefix is /api<br>   return len(c.Path()) &gt;= 4 &amp;&amp; c.Path()[:4] == &quot;/api&quot;<br>  },<br> }))<br>}</pre><p>We have a new condition in RegiserHandlers that checks if the env is dev and if so we call setupDevProxy.</p><p>All this does is create a balancer group with one member (the vite dev server) and use a skipper function if the path is prefixed with /api. This keeps our api routes isolated.</p><p>I found this to work really well so far, and the developer experience is the best I have had so far when working on fullstack web applications.</p><p>The full repo is <a href="https://github.com/danhawkins/go-vite-react-example/">https://github.com/danhawkins/go-vite-react-example/</a></p><p>Original post: <a href="https://blog.dannyhawkins.me/posts/go-vite-react-embed/">https://blog.dannyhawkins.me/posts/go-vite-react-embed/</a></p><p><em>Thank you for reading until the end. Please consider following the writer and this publication. Visit </em><a href="https://stackademic.com/"><em>Stackademic</em></a><em> to find out more about how we are democratizing free programming education around the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e929099752d" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/golang-with-vite-react-with-live-reload-9e929099752d">Embed Vite React in Golang binary, with live reload!</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connect Livebook to Elixir in Kubernetes]]></title>
            <link>https://medium.com/quiqup-engineering/connect-livebook-to-kubernetes-9d15d97e44db?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d15d97e44db</guid>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[phoenix]]></category>
            <category><![CDATA[livebook]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Tue, 19 Apr 2022 16:29:15 GMT</pubDate>
            <atom:updated>2023-08-16T03:24:05.200Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been playing around with <a href="https://livebook.dev/">Livebook</a> recently, it’s an amazing tool, even more so considering how new it is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H7JejMoY9ruEBI3_f3-2DA.png" /></figure><p>I had a specific use case where I needed to connect Livebook to a service running inside a private Kubernetes cluster where my QA environment is running. So that Livebook can be used to setup and teardown common QA Scenarios and make it a lot easier to document and execute.</p><p>Livebook has several ways to configure the connected runtime, we’re going to be using the Attached node mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FBowuoUgEs9ydJwoblpOmA.png" /></figure><p>we’re going to be focused on setting up a remote service to be ready to connect to, as well as making a script that allows us to boot livebook already connected to our remote service.</p><h3>Preparing your service</h3><p>We will assume that your Elixir service is <strong>not setup </strong>for remote connections / distributed erlang, but if it is you can probably <strong>skip this section</strong>.</p><h4>Update Deployment env vars</h4><p>First you need to edit the Kubernetes deployment spec, we need to add in some additional environment variables that will be used to configure the node name and address, we need to add add the additional values for:</p><p><strong>RELEASE_COOKIE</strong> this should be a secret that is basically the required value to connect to the node later</p><p><strong>NAMESPACE</strong> we get the pod namespace by using k8s internal valueFrom env vars</p><p><strong>POD_IP</strong> this will be used in our env script to figure out what to set the node name to, so we can ready the service node</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a79655ed785be91abdc4cf3d649fcd5/href">https://medium.com/media/4a79655ed785be91abdc4cf3d649fcd5/href</a></iframe><h4>Update for release and change env script</h4><p>Update mix.ex for releases (this is not 100% essential but keeps things clean and avoids building windows executables in my case):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/124dd21f554a0ea033d5fdfddd028b2d/href">https://medium.com/media/124dd21f554a0ea033d5fdfddd028b2d/href</a></iframe><p>Then run the release init script to create the necessary files</p><pre>mix release.init</pre><p>Edit the created <strong>env.sh.eex</strong> file, it will use the previously configured environment variables to set the release name. Re-deploy the service and you should be good to go.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8e1bb7c9654094b29ee9a95604aa4438/href">https://medium.com/media/8e1bb7c9654094b29ee9a95604aa4438/href</a></iframe><p>In my case the service name on the terminal now looks like something I can address via k8s dns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vADv0gG8ZYcCLyLX7T30Jg.png" /></figure><p>In my case my addressable node is <strong>test_svc@10–8–7–66.dev.pod.cluster.local</strong> this is using Kubernetes DNS. I use this strategy because it’s recommended for this kind of setup from <a href="https://hexdocs.pm/libcluster/Cluster.Strategy.Kubernetes.html#content">libcluster</a>:</p><blockquote>The benefit of using :dns over :ip is that you can establish a remote shell (as well as run observer) by using kubectl port-forward in combination with some entries in /etc/hosts.</blockquote><blockquote>Using :hostname is useful when deploying your app to K8S as a stateful set. In this case you can set your erlang name as the fully qualified domain name of the pod which would be something similar to my-app-0.my-service-name.my-namespace.svc.cluster.local</blockquote><h3>Livebook!!</h3><p>OK now for the fun part. I decided wrap everything in a simple script to be able to connect. It’s almost too easy!!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/347cc48aae8296a8d8fde83e5e5341ab/href">https://medium.com/media/347cc48aae8296a8d8fde83e5e5341ab/href</a></iframe><p>What we are doing here, is getting the POD_NAME, RELEASE_NODE and RELEASE_COOKIE using some lookups and execs with <strong>kubectl</strong>, then we are running a pod in the cluster preconfigured to access the running node.</p><p>Run the script to create the pod in the cluster.</p><p>In order to access it, we need find the token, we can do that by looking at the logs:</p><pre>kubectl logs danny-livebook</pre><pre># [Livebook] Application running at http://localhost:8080?token=&lt;tokenishere&gt;</pre><p>then we need to port forward using kubectl</p><pre>kubectl port-forward danny-livebook 8080:8080</pre><p>And there we have it, a connected Livebook inside Kubernetes, accessible at localhost:8080</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ansJEiWSxiwu1SIis-xdcQ.png" /></figure><h4>Clean Up</h4><p>When you are finished with your livebook, you can just close the port-foward. Then delete the pod</p><pre>kubectl delete po/danny-livebook</pre><h3>Bonus: Standalone Node</h3><p>What I’ve covered so far, is the ability to connect to a running node, but if you want to use the same approach to run a standalone node in Kubernetes, all you need is to adjust the Livebook script (and don’t need to run other configuration):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92d2798ba664de5cf1c16862d8baa18c/href">https://medium.com/media/92d2798ba664de5cf1c16862d8baa18c/href</a></iframe><h3>Another Bonus: Cloud Storage</h3><p>We can add a little bonus to our startup script, if you have an s3 bucket and the credentials you can setup Livebook to always be connected to remote storage when it starts</p><p>Just add this line to the startup command:</p><pre>--env LIVEBOOK_FILE_SYSTEM_1=&quot;s3 ${BUCKET} ${ACCESS_KEY} ${ACCESS_KEY_SECRET}&quot; \</pre><p>Then you will have access via the Liveview interface</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*velxA764clCCR_9qzMhbuw.png" /></figure><p><em>Original post: </em><a href="https://blog.dannyhawkins.me/posts/livebook-to-kubernetes/"><em>https://blog.dannyhawkins.me/posts/livebook-to-kubernetes/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d15d97e44db" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiqup-engineering/connect-livebook-to-kubernetes-9d15d97e44db">Connect Livebook to Elixir in Kubernetes</a> was originally published in <a href="https://medium.com/quiqup-engineering">Quiqup Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Display localised date / time information from Phoenix LiveView]]></title>
            <link>https://medium.com/quiqup-engineering/display-localised-date-time-information-from-phoenix-liveview-e7d45b832b58?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7d45b832b58</guid>
            <category><![CDATA[liveview]]></category>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[phoenix-liveview]]></category>
            <category><![CDATA[phoenix]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Wed, 13 Apr 2022 22:57:43 GMT</pubDate>
            <atom:updated>2022-04-20T18:39:54.787Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YzFkhyx9Viox0dQ_Z5CF6Q.jpeg" /></figure><p>I recently hit a snag moving away from JS heavy frontends to <a href="https://github.com/phoenixframework/phoenix_live_view">Phoenix LiveView</a>. There are a few things one might take for granted, and it’s only when you try to do that thing in a new technology that you appreciate the simple method you had before. In this case it was giving the user a time back in their local timezone (and preferred locale settings)</p><p>I’m a strong advocate that all date/time information in the database and the code base should stay in UTC until it needs to be returned / presented to the user.</p><p>I won’t go into the whys, but I found a <a href="https://kylekatarnls.medium.com/always-use-utc-dates-and-times-8a8200ca3164#:~:text=Even%20if%20you&#39;re%20100,export%20and%20import%20them%20elsewhere">good article</a> that summarises it pretty well.</p><p>When creating API’s I will always be returning UTC eg <strong>“2022–04–14T22:00:00Z”,</strong> typically when javascript frontend decide to display that date they will use something like:</p><pre>new Date(&#39;2022-04-13T12:00:00Z&#39;).toLocaleString()</pre><pre>// Output is specific to the user locale (I&#39;m UTC+4 in GST)<br>&#39;13/04/2022, 16:00:00&#39;</pre><p>This is super useful, because it means we are using a friendly time format for data, but giving the user something that makes sense to them.</p><h3>The Problem</h3><p>When using phoenix LiveView, there is no direct way to get back the locale of the user from the browser, and use it to format the time in the HEEX / HTML template (or if there is it eluded me, or was flaky).</p><p>Somehow we need to be able to leverage the local javascript functionality to be able to translate a UTC date into a locale friendly date.</p><h3>The Solution</h3><p>The way I found to solve this was using AlpineJS on the front end, and create a live helper / web component to help with the conversion. Assuming you have a recent phoenix live view application already setup, let’s go step by step.</p><h4>Step 1: Install Alpine</h4><p>First of all we need to add alpine, move into the assets folder and install the npm package</p><pre>cd /assets<br>npm install alpinejs</pre><p>Next update <strong>assets/js/app.js</strong> to include alpine</p><pre>import &#39;phoenix_html&#39;</pre><pre><strong>import Alpine from &#39;alpinejs&#39; // Import Alpine</strong></pre><pre>// Establish Phoenix Socket and LiveView configuration.import { Socket } from &#39;phoenix&#39;<br>import { LiveSocket } from &#39;phoenix_live_view&#39;<br>import topbar from &#39;../vendor/topbar&#39;</pre><pre><strong>// Add alpine to window and start<br>window.Alpine = Alpine<br>Alpine.start()</strong></pre><p>Also we want alpine values to survive a LV update, so update the LiveSocket creation, adding the whole section for the <strong>dom</strong> key</p><pre>let liveSocket = new LiveSocket(&#39;/live&#39;, Socket, {<br>  params: { _csrf_token: csrfToken },<br>  <strong>dom: {<br>    onBeforeElUpdated(from, to) {<br>      if (from.__x) {<br>        window.Alpine.clone(from.__x, to)<br>      }<br>    },<br>  },</strong><br>})</pre><p>This next change is not well documented, and had me scratching my head for a couple of hours, you need to make sure the the esbuild target is <strong>es2019 or later,</strong> in <strong>config.exs</strong></p><pre>config :esbuild,<br>  version: &quot;0.12.18&quot;,<br>  default: [<br>    args:<br>      ~w(js/app.js --bundle <strong>--target=es2019</strong> --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),<br>    cd: Path.expand(&quot;../assets&quot;, __DIR__),<br>    env: %{&quot;NODE_PATH&quot; =&gt; Path.expand(&quot;../deps&quot;, __DIR__)}<br>]</pre><p>OK, so thats alipine all good to go</p><h4>Step 2: Create the datetime helper function</h4><p>Create a new module (or update an existing one) to include the datetime helper. Mine looks like this</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc53035a06c6db09388b842e072f3a40/href">https://medium.com/media/fc53035a06c6db09388b842e072f3a40/href</a></iframe><p>Then make sure to include this helper in the <strong>view helpers block</strong> in your web module root (in my case QuiqupFleetWeb)</p><pre>defp view_helpers do<br>  quote do<br>    use Phoenix.HTML<br>    <br>    import Phoenix.LiveView.Helpers<br>    import Phoenix.View<br>    import QuiqupFleetWeb.Gettext</pre><pre><strong>    # Import the live helpers module here<br>    import QuiqupFleetWeb.LiveHelpers</strong></pre><pre>    import Phoenix.LiveView.Helpers<br>    alias QuiqupFleetWeb.Router.Helpers, as: Routes</pre><pre>  end<br>end</pre><p>And thats it, now in your view you are able to just call the web component, and it will automatically replace empty inner text with locale specific date time. For me using GST (Gulf Time) +4hrs UTC</p><pre>&lt;.datetime id=&quot;time&quot; value={~U[2022-04-14 18:00:00Z]} format=&quot;timeshort&quot; /&gt;</pre><pre>Outputs: <strong>22:00</strong></pre><pre>&lt;.datetime id=&quot;full-date&quot; value={~U[2022-04-14 18:00:00Z]} format=&quot;datetime&quot; /&gt;</pre><pre>Outputs: <strong>14/4/2022 22:00:00</strong></pre><h3>Breakdown</h3><p>The main thing we are using the helper for is to return a span, and on that span we are setting alpinejs <strong>x-data</strong> to a date in js, datetime being passed in via assigns as <strong>value</strong></p><pre>x-data={&quot;{date: new Date(&#39;#{datetime}&#39;)}&quot;}</pre><p>Then we are using alpines <strong>x-text</strong> to tell alpine what to replace the inner text of the element with</p><pre>x-text={&quot;date.#{locale_fn}&quot;}</pre><p>The locale_fn is just describing the function to call against date in javascript, for which I have made some predefined formats that I want</p><pre>case format do<br>  &quot;datetime&quot; -&gt; &quot;toLocaleString([],{dateStyle: &#39;short&#39;, timeStyle: &#39;short&#39;})&quot;<br>  &quot;date&quot; -&gt; &quot;toLocaleDateString()&quot;<br>  &quot;timeshort&quot; -&gt; &quot;toLocaleTimeString([],{timeStyle: &#39;short&#39;})&quot;<br>  &quot;time&quot; -&gt; &quot;toLocaleTimeString()&quot;<br>  _ -&gt; &quot;toLocaleString()&quot;<br>end</pre><p>and thats it!</p><h3>EDIT 18th April 2020</h3><p>My first version of this was without the phx-update=&quot;ignore&quot; attribute on the date span, turns out the LV and alpine don’t play super well right now, so if you don’t ignore the phx update the text can get wiped out. I’ll think about a more elegant solution, but for my current needs this gets over the issue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7d45b832b58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiqup-engineering/display-localised-date-time-information-from-phoenix-liveview-e7d45b832b58">Display localised date / time information from Phoenix LiveView</a> was originally published in <a href="https://medium.com/quiqup-engineering">Quiqup Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Effective Meetings]]></title>
            <link>https://medium.com/quiqup-engineering/running-effective-meetings-8ad70059d720?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ad70059d720</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[meetings]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Fri, 01 Apr 2022 10:18:00 GMT</pubDate>
            <atom:updated>2022-04-01T14:31:54.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAvBQi1iqB_PzREeJ-p1fQ.jpeg" /></figure><p>I wrote something a while ago about running effective meetings, for me this is the holy grail of how meeting should be organised, whilst it’s not all black and white and even establishing a rigid process like this can be counter productive, a lot of the ideas here are really just about respecting peoples time, and not being wasteful. So I thought I would share my thoughts. I’d love to hear about ways that you run effective meetings, or even if you think any of the ideas presented here are actually counter productive for you.</p><p>Meetings can bring the biggest negative impact on productivity to teams when they start to take up more of the week than focused work. When meetings are not executed in the right way, people can have a negative view on having to attend, and the value that the meeting should have given is lost.</p><h3>Before you meet</h3><h4>Do you really need to?</h4><p>Meetings can be costly, you are taking away people’s time and breaking their focus, and moving them away from whatever context they might be working in during that time.</p><p>The first question to ask yourself before scheduling a meeting is does this meeting really need to happen? Perhaps the desired outcome can be achieved using collaboration tools, a messaging platform or even good old email, consider using these options in the first instance, they are far less intrusive and allow people to respond in their own time.</p><p>There are also other benefits to taking this non meeting approach in that the topic and comments are all documented for you as a byproduct, without additional work.</p><h4>Who should attend?</h4><p>Decide who should attend the meeting, make sure that people involved in the meeting will get value to be able to contribute effectively. If it’s likely some member of the meeting will be “zoned out” during the whole session, consider booking separate sessions to cover relevant topics.</p><p>Too often the approach is to invite everyone that <em>might</em> be relevant without being sure, consider marking people as optional if you are not 100% sure they can add or take value from the session.</p><h4>Should I attend?</h4><p>Just because you were invited to a meeting, don’t just accept and attend if you are unsure of where you can add value or take value. In these cases you can speak to the organiser.</p><h3>Meeting Agenda</h3><p>Any meeting that has <strong>more than three attendees </strong>should have a clear agenda, it makes it much easier for people to decide if they are needed, or prepare materials.</p><h4>Set the style of the meeting</h4><p>What kind of meeting are you booking:</p><p><strong>If it’s a presentation</strong> perhaps you can share information to set the context of the presentation, for example if the presentation a proposed product change you are planning to make, perhaps you can provide information about the current state of the product, and/or reading materials that exist within the company. Identify in the agenda how interaction should be, is it OK to ask questions at any time, should there be breaks in the delivery of the presentation for questions, or should attendees save them for the end.</p><p><strong>If it’s a discussion</strong> around a problem area we are looking to find solutions for, make sure to clearly describe the problem, link to any materials that can help set the context, provide and solutions that have already been considered, this will help to avoid wasted time and allow people to come with ideas ready to discuss.</p><p><strong>If it’s a ceremonial</strong> <strong>meeting</strong>, make sure any information required is readily available, and encourage your attendees to prepare their notes in between the ceremonial meetings, for example when you run a product Retrospective, attendees should be gathering their notes throughout the week(s), so that are not just trying to think back. If you plan to do a sprint review, make sure the stories that need to be discussed are accessible.</p><p>It can also be good in the meeting to set the time expectations for any segments of the meeting, this is essential for good moderation.</p><h4>Who will chair, moderate and minute the meeting</h4><p>Identify the person chairing the meeting, also identify a moderator, the chair will be expected to introduce the meeting and confirm the agenda at the beginning.</p><p>The moderator will be expected to keep the meeting on track, according the style of meeting set in the agenda, for example if a discussion is going off track or losing the attention of the group, it should be documented and deferred for later. If people are asking questions at the wrong time, they should be noted for the appropriate time.</p><p>If the meeting should be minuted, also identify the minute maker.</p><h4>Provide Materials and Consider the time of booking</h4><p>When booking the meeting in the calendar provide materials early, if you have documentation, presentations, questions to consider, problem statements, or anything else that can help attendees to get in the right context and consider their contribution ahead of time provide them.</p><p>Consider how long your attendees need to be able to digest the information provided, don’t provide a meeting booking on a Monday night for a meeting on Tuesday with materials that take time to digest!</p><h4>Identify the outcome</h4><p>What are we expecting to gain from having this meeting, it could be potential solutions to a problem and next steps, a new way of approaching work, a decision need to be taken, an answer to a question or anything really, but make sure when booking you have considered this, and provided it on the agenda. It can help you also to qualify if the meeting is worthwhile.</p><h3>Be Prepared!!!</h3><p>All attendees of the meeting should be prepared prior to the meeting.</p><p>For the chairperson, presenter or otherwise leading the meeting, think about who is in the meeting, what level of context they might have. Try to remember that you likely have been thinking the subject matter through for some time, but your attendees might be new to it, try to think of the path you want to go through in the meeting so that you can try to keep on track.</p><p>If you have a presentation, use the presenter notes option on google slides or keynote to put your thoughts, if you are just reading the slides verbatim, then probably you don’t need a meeting for that!</p><p>For attendees, ensure that you have read the agenda and any materials that will help you get straight into the valuable parts of the meeting, if there are decisions to be made, try to form your ideas / opinions with what context you have, so you can contribute effectively</p><p>In fact be a little early, it’s a good policy to be at the meeting a couple of minutes early (online or in person).</p><p>Give yourself a short time to gather your notes / thoughts and try to get in the right frame of mind for the meeting, if you have something urgent you have to take care of, or there is something that will cause you to be distracted during the meeting, potentially apologise and don’t attend, moving a meeting close to the meeting time is extremely bad practise!</p><h3>During the meeting</h3><h4>Introduction</h4><p>The chairperson should recap on what the meeting is about, re-iterate the format and when to ask questions, as well as the desired outcome from the meeting.</p><h4>Moderation</h4><p>Moderation is a fine art, getting it right takes a bit of time and practise. Some basic however can go a long way.</p><p>The moderator should look for times during the meeting where the rules are not being followed (when to ask questions for example), where conversations are going off track (we have seen many times where two people end up dominating the time with a one on one discussion that does not add overall value in the meeting context) or when segments of the meeting are not falling with the allotted time.</p><p>When conversations are going off track, it’s best to request that conversation is deferred for later, take the point down as an action point or note that can be addressed either at the end of the meeting, outside of the meeting or at a later meeting.</p><p>If segments of the meeting are running close to the allotted time, the moderation can bring attention to that, and try to move on, ask if attendees can all stay in meeting a little bit longer, or potentially sacrifice a later segment of the meeting and book another meeting to continue. This really comes down to subject matter and common sense, again moderation is not just a set of rules that should be followed to the letter.</p><p>The minute maker should take down the key points discussed during the meeting and and conclusions from discussions.</p><p>During the meeting if any actions are taken for attendees (or non attendees) this should be captured clearly and assigned to the correct person</p><p>After the meeting, where relevant minutes should be sent out to all attendees, likewise for action points.</p><h3>Summary</h3><p>Use this simple checklist to structure your meeting:</p><ul><li>Does this meeting really need to happen?</li><li>What is the desired outcome?</li><li>Who should attend?</li><li>What is my role in this meeting?</li><li>What kind of meeting will it be?</li><li>Provide the relevant content before hand</li><li>Who will chair and moderate the minutes?</li><li>Be prepared</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ad70059d720" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiqup-engineering/running-effective-meetings-8ad70059d720">Running Effective Meetings</a> was originally published in <a href="https://medium.com/quiqup-engineering">Quiqup Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ranked data with Ecto window Queries]]></title>
            <link>https://medium.com/quiqup-engineering/ranked-data-with-ecto-window-queries-be4c169b9f86?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/be4c169b9f86</guid>
            <category><![CDATA[ecto]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[elixir]]></category>
            <category><![CDATA[phoenix]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Sun, 20 Mar 2022 00:30:12 GMT</pubDate>
            <atom:updated>2022-08-04T17:57:56.690Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uRv8JgoqXsI4L1Ft8Hhc7A.png" /></figure><p>I recently came across a problem that I would have usually thought to solve by adding additional field(s) to the database to track the rank of a row of data within its table. However having learned about window functions recently, I realised I could solve this in a much more elegant way using <a href="https://hexdocs.pm/ecto/Ecto.Query.WindowAPI.html">window functions</a> and <a href="https://hexdocs.pm/ecto/Ecto.SubQuery.html">subqueries</a> in Ecto</p><h3>The problem</h3><p>I have a set of data where there is a scoring element and a grouping element, for example:</p><pre>| Name       | Region | Score |<br>| ---------- | ------ | ----- |<br>| AE Group 1 | UAE    | 10    |<br>| AE Group 2 | UAE    | 30    |<br>| AE Group 3 | UAE    | 35    |<br>| AE Group 4 | UAE    | 50    |<br>| UK Group 1 | UK     | 8     |<br>| UK Group 2 | UK     | 30    |<br>| UK Group 3 | UK     | 38    |<br>| UK Group 4 | UK     | 52    |</pre><p>When presenting this data, I also want to be able to show a rank element, and depending on the filter I want to be able to show that position within the scope, for example when looking at all results, the rank should be within the scope of all records:</p><pre>| Rank | Name       | Region | Score |<br>| ---- | ---------- | ------ | ----- |<br>| <strong>1</strong>    | UK Group 4 | UK     | 52    |<br>| <strong>2</strong>    | AE Group 4 | UAE    | 50    |<br>| <strong>3</strong>    | UK Group 3 | UK     | 38    |<br>| <strong>4</strong>    | AE Group 3 | UAE    | 35    |<br>| <strong>5</strong>    | AE Group 2 | UAE    | 30    |<br>| <strong>6</strong>    | UK Group 2 | UK     | 30    |<br>| <strong>7</strong>    | AE Group 1 | UAE    | 10    |<br>| <strong>8</strong>    | UK Group 1 | UK     | 8     |</pre><p>When filtering for <strong>UK </strong>the rank should be within the scope of the UK<strong>:</strong></p><pre>| Rank | Name       | Region | Score |<br>| ---- | ---------- | ------ | ----- |<br>| <strong>1 </strong>   | UK Group 4 | UK     | 52    |<br>| <strong>2</strong>    | UK Group 3 | UK     | 38    |<br>| <strong>3</strong>    | UK Group 2 | UK     | 30    |<br>| <strong>4</strong>    | UK Group 1 | UK     | 8     |</pre><p>When filtering for <strong>UAE </strong>the rank should be in the scope of UAE<strong>:</strong></p><pre>| Rank | Name       | Region | Score |<br>| ---- | ---------- | ------ | ----- |<br>| <strong>1</strong>    | AE Group 4 | UAE    | 50    |<br>| <strong>2</strong>    | AE Group 3 | UAE    | 35    |<br>| <strong>3</strong>    | AE Group 2 | UAE    | 30    |<br>| <strong>4</strong>    | AE Group 1 | UAE    | 10    |</pre><p>Finally to make things a little more interesting we will allow for a search to be made against the dataset, which will filter data at the database level (using a <strong>WHERE</strong> condition) but <strong>not lose the ranking within the regional or global scope</strong>. This is the part that made this particular solution effective.</p><pre><strong>Filter: &quot;UAE&quot;, Search: &quot;Group 4&quot;</strong></pre><pre>| Rank | Name       | Region | Score |<br>| ---- | ---------- | ------ | ----- |<br>| <strong>1</strong>    | AE Group 4 | UAE    | 50    |</pre><pre><strong>Filter: &quot;UK&quot;, Search: &quot;UK Group 2&quot;</strong></pre><pre>| Rank | Name       | Region | Score |<br>| ---- | ---------- | ------ | ----- |<br>| <strong>2</strong>    | UK Group 3 | UK     | 38    |</pre><pre><strong>Filter: &quot;All&quot;, Search: &quot;Group 3&quot;</strong></pre><pre>| Rank | Name       | Region | Score |<br>| ---- | ---------- | ------ | ----- |<br>| <strong>3</strong>    | UK Group 3 | UK     | 38    |<br>| <strong>4</strong>    | AE Group 3 | UAE    | 35    |</pre><h3>Solution</h3><p>I’ll to go through the full solution here, but if you want to just head to the code, it’s on github here: <a href="https://github.com/quiqupltd/ecto_window_functions_example">https://github.com/quiqupltd/ecto_window_functions_example</a>, follow the Getting started section in the README.</p><p>First of all lets create our migration and schema, we’re creating and model and table for groups, that have a name, region and score.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99713540427a2707f21908c4de6c0267/href">https://medium.com/media/99713540427a2707f21908c4de6c0267/href</a></iframe><p>The main thing to note here is the virtual attribute <strong>group_rank, </strong>we’re going to use this later to hold a computed value for the rank</p><p>Next lets create a test suite that matches our brief:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fbe7e6d9ab31aac0ad013c1dff9d020/href">https://medium.com/media/4fbe7e6d9ab31aac0ad013c1dff9d020/href</a></iframe><p>Then create the code to make the tests pass, we want to break down the task into 3 main parts.</p><ol><li>Filter the region</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f449f7e908270be2a223a215b58964b/href">https://medium.com/media/1f449f7e908270be2a223a215b58964b/href</a></iframe><p>2. Rank the data using a window function</p><p>This is where we are using the virtual attribute from earlier <strong>group_rank</strong>, we populate it in the select using the window function <a href="https://hexdocs.pm/ecto/Ecto.Query.WindowAPI.html#row_number/0"><strong>row_number</strong></a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07d8f568d8561ede02a0feadd541a35f/href">https://medium.com/media/07d8f568d8561ede02a0feadd541a35f/href</a></iframe><p>3. Search the result set</p><p>Finally we want to run a search across the results if a search string is passed, the important part here is to put the query we have built so far into a subquery, so the the search is executed over the already ranked data, this way we don’t calculate the rank on the result after the search</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8506f7c3f37a74a8eebfb7e517b661b/href">https://medium.com/media/b8506f7c3f37a74a8eebfb7e517b661b/href</a></iframe><p>The final groups module looks looks like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4f75b0231eb0b12e1369786ffa21588/href">https://medium.com/media/a4f75b0231eb0b12e1369786ffa21588/href</a></iframe><h3>Summary</h3><p>Window functions and subquery options provide a great deal of power to work with data before it’s gets fetched directly into your runtime.</p><p>Typically developers who are less SQL savvy may solve this problem by adding additional columns to the database (efficient for queries), or pulling larger amounts of data from the database, the manipulating the results with code (over fetching data). I found this approach to be very pleasing and I’m already thinking of other areas that can be improved in code that I work on with better use of Ecto and SQL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be4c169b9f86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiqup-engineering/ranked-data-with-ecto-window-queries-be4c169b9f86">Ranked data with Ecto window Queries</a> was originally published in <a href="https://medium.com/quiqup-engineering">Quiqup Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choice of Technology at Quiqup]]></title>
            <link>https://medium.com/quiqup-engineering/choice-of-technology-at-quiqup-b2acc92ba4c7?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2acc92ba4c7</guid>
            <category><![CDATA[phoenix]]></category>
            <category><![CDATA[commanded]]></category>
            <category><![CDATA[event-sourcing]]></category>
            <category><![CDATA[elixir]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Fri, 18 Mar 2022 10:24:00 GMT</pubDate>
            <atom:updated>2022-03-21T08:38:39.633Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Throughout the lifetime of Quiqup we been evolving our ability and understanding of good engineering. We’ve made a fair few mistakes along the way, but all of it has led to where we are now, and making the mistakes we have only make use more confident in our current decisions. I wanted to write a brief summary of our journey so far.</p><p>In the early days of Quiqup we built our API with Ruby on Rails, at the time (and still now) it’s a very popular option for rapid development, using convention over configuration allowing developers to produce consistent code and leverage the framework elements included to work fast.</p><p>We were building many features one after another, in many different areas of the business, and Ruby on Rails served us well as an API platform.</p><p>For frontend we started with Ionic (web on mobile technology), followed by native IoS for our for customer facing product (at that time our proposition was consumer facing). Internally for our dispatch tool we were using Angular1.3</p><h3>Pivot to B2B</h3><p>When we realised how important our place was in the business to business world, providing logistics as a service , there was a big push to build out client facing tools for businesses. We started with a google form, to a react based form, to a fully fledged business focused offering called Quiqdash . This was all built out using React as we started to move away from Angular 1.3, primarily because React was considered to be much better for large apps composed of smaller components.</p><p>From the backend we were still with Ruby on Rails primarily, although we had our first Elixir service which was RealtimeEx, whose responsibility it was to receive updates from the CoreAPI via Redis pub/sub and push those updates out to connected web socket clients, primarily Quiqdash, and Dispatch. Elixir was selected for its erlang roots, erlang itself having a great reputation for web socket connections, distributed compute and fault tolerant, and Elixir being created to provide “something better” for Rubyists.</p><h3>Time for Change</h3><p>Working with the CoreAPI was starting to become very difficult, we did have parts of the CoreAPI that were written in a much cleaner way since we realised the default “rails” way had some approaches that we now consider to be bad practice including but not limited to, too much logic in models, using a lot of includes for code separation only, model callbacks, and no default “service” layer. But the majority of the code was very tightly coupled and had a lot of “magic” such as model callbacks that would send a realtime broadcast whenever a model changed.</p><p>Although we could have stayed with Rails, we had some exposure to both Typescript and Elixir and considered both as better options than to forward develop using Ruby on Rails, primarily because of the NestJS (Typescript) framework and the Phoenix (Elixir) framework.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*n6YO0shJBMSY2jn0" /></figure><p>We took the decision to start to adopt NestJS as a general use language in 2019, although there was a strong preference from the backend team to use Elixir / Phoenix, there were a few things we considered that made our final choice to be NestJS:</p><p>Elixir / Phoenix issues:</p><ul><li>Still relatively new</li><li>Devs will need to learn functional programming</li><li>Availability of talent</li></ul><p>NestJS benefits:</p><ul><li>Typescript is then used across the full stack</li><li>Modular approach with dependency injection from the start</li><li>Developer experience is amazing in vscode because of typescript support</li></ul><p>Once we started to use NestJS for a few projects, like our area restrictions gRPC service, and our Invoicer solution, we had really good results, although some parts of using Typescript / Node.JS can be a developer hell (like tracing an async exception). Increased productivity, concurrency, code reusability and modularity all meant that the overall NestJS had been a good choice.</p><h3>Event Sourcing</h3><p>For a long time we had been wanting to move to a <a href="https://martinfowler.com/eaaDev/EventSourcing.html">event sourcing</a> for a large amount of our business logic, it makes a lot of sense for Quiqup as we are dealing with real world events all the time, so mapping them 1-to-1 to software would make it easy to create a common language across business, product and technology.</p><p>Given some of the problems from our past, the idea of command query responsibility segregation (CQRS) and the flexibility of being able to create projections (read models) as and when needed was also very appealing.</p><p>We had been researching and learning about Event Sourcing for some time before we decided to bring it into a project. Whilst the benefits were clear, so were the complexities of adopting this approach. We had a proof of concept that was created to test a basic chain of custody system when we were working with one of our first big e-commerce clients. This was before we had adopted NestJS as general use. We built a PoC in Elixir using a framework called <a href="https://github.com/commanded/commanded">Commanded</a>. This framework had been built for event sourcing, and has some very elegant methods to solve some key problems. One such example is that it keeps aggregates in running processes on the erlang VM called GenServers. This meant that when you want to work with an aggregate, you do not need to always load the stream of events.</p><p>Although the PoC was not developed further as we stopped working with the client, the experience of working with Commanded and building our first real event sourced service only strengthened our resolve that event sourcing made sense for Quiqup.</p><h3>Courier Incentives Initiative</h3><p>Towards the end of 2020 we had started an initiative to break courier compensation into two parts, a flat hourly fee and an order based bonus scheme, the system would also be able to track penalties issued to drivers and would take over the financial responsibility of calculating courier / agency pay, and therefore costs.</p><p>This project was an ideal candidate to move to an event sourced system, it could be separate enough from the CoreAPI to stand by itself, it fit well into one of our identified domains (Fleet Management) and it was relatively small in scope. We did some initial technical qualification on using NestJS and NestJS CQRS module to build our system using our general use language / framework and the knowledge we had of event sourcing.</p><p>The project took some time to get off the ground as we had to bring the team into the world of event sourcing, we were also learning a lot how to design the system using event modelling. During development we hit a few problems that sucked a lot of dev time to solve, and when we were solving these problems (specifically around event sourcing) we found ourselves looking toward the Elixir Commanded framework to see how they solved it. At some point along this journey it became apparent we would need a larger team to “roll our own” equivalent of what already existed in Commanded, and we would then have to maintain and develop that framework, alongside future work. It was at this point that we considered if we should just be using Elixir, Phoenix and commanded.</p><p>One of our lead engineers created a PoC for one module of our courier incentive system using Elixir, the result was great, he was able to port a lot of logic very quickly because commands, events and projections are very portable by nature.</p><p>It became apparent that using Elixir for courier incentive we could end up with a stable solution much faster, so we re-created the core areas of the service over 2 weeks.</p><p>During this time two of our typescript engineers who had been working with NestJS also got involved with the conversation, and we were very surprised to find that they were able to pick up and start moving with Elixir very fast, with little guidance.</p><h3>Elixir, Phoenix and Commanded</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*PYxgv3jyVP6YwUy5" /></figure><p>Now that we are running multiple services in Elixir and we have explored options away from Elixir we are confident at this point in the evolution of Quiqup, <a href="https://elixir-lang.org/">Elixir</a>, <a href="http://www.phoenixframework.org/">Phoenix</a> and <a href="https://github.com/commanded/commanded">Commanded</a> make a lot of sense for our core services. We also learned that porting an event sourced system is less complicated than alternatives, largely due to the responsibility of code in a command, event, aggregate and projection are clearly defined.</p><p>Making a choice to use and language or framework becomes a lot easier when you know that porting to other options is simple.</p><p>The productivity and stability gains we have experienced have been very pleasing, and using the excellent ElixirLS VSCode extension has resulted in a developer experience compatible to that of typescript (with WAY better stack traces) it’s for that reason that we are focused on using Elixir for immediate future at least.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2acc92ba4c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiqup-engineering/choice-of-technology-at-quiqup-b2acc92ba4c7">Choice of Technology at Quiqup</a> was originally published in <a href="https://medium.com/quiqup-engineering">Quiqup Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create webhooks from Google sheets]]></title>
            <link>https://medium.com/quiqup-engineering/create-webhooks-from-google-sheets-cba629151909?source=rss-87e1572c08b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cba629151909</guid>
            <category><![CDATA[development]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[webhooks]]></category>
            <category><![CDATA[gsuite-integration]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[Danny Hawkins]]></dc:creator>
            <pubDate>Tue, 15 Oct 2019 14:53:51 GMT</pubDate>
            <atom:updated>2019-10-15T14:58:20.729Z</atom:updated>
            <content:encoded><![CDATA[<p>Google have an awesome and often overlooked capability in their Google scripts offering. I have always shied away from it with a kind of PTSD from the days of using VBscript in MSOffice. But recently I had a reason to create a custom function; the goal of which was to use Google vision API over an image fetched from a URL and put the results in a cell… but that’s a post for another time.</p><p>This requirement meant I started to explore Google scripts in more depth, I was honestly shocked at the capabilities that had been hiding from me for so long (<a href="https://developers.google.com/apps-script/guides/sheets/functions">https://developers.google.com/apps-script/guides/sheets/functions</a>) so I figured given all these scripts, can I make a row by row webhook from a Google script, the answer is YES, and it’s super easy!</p><h3>Webapp first</h3><p>I didn’t go straight to the webhook, I first wanted to try out something I read about making the Google sheet into a <a href="https://blog.runscope.com/posts/tutorial-capturing-webhooks-with-google-sheets">web app and receiving webhooks</a>, that was pretty simple:</p><h4>Create a new sheet with some data</h4><p>The sample one used below is public and available at <a href="https://docs.google.com/spreadsheets/d/1nWjQWnem8buUJKTVV0etcsVUzq6RU5k2wu9Scqha8Hg">here</a> (make a copy to use it yourself)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8AtU6HQDDGq3PaJd12FLvg.png" /></figure><h4>Add some scripts</h4><p>Open up the script editor from Tools -&gt; Script Editor, and add a <strong>doGet</strong> and <strong>doPost</strong> function:</p><pre>function doGet(e) {<br> var ss = SpreadsheetApp.getActive();<br> var rng = ss.getActiveSheet().getRange(2, 1, 10, 3)<br> var vals = rng.getValues()<br> Logger.log(vals)<br> return ContentService.createTextOutput(JSON.stringify(vals)).setMimeType(ContentService.MimeType.JSON);<br>}</pre><pre>function doPost(e) {<br> return HtmlService.createHtmlOutput(“post request received”);<br>}</pre><p>The example above will just return row 2 -10 as json</p><h4>Publish as Webapp</h4><p>Settings should be customised for you here, but I just set it to open for anyone), If you wanted to have this open but secure it yourself you could check the request header for some token</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*Gez_HxxpgJlfGMcOdXnuZg.png" /></figure><h4>Go to the <a href="https://script.googleusercontent.com/macros/echo?user_content_key=3fe97HN7uvRzxlsdBnj8Nu-L7hfmthHMIpuX-UjJ8h-F4Lcjg4VjmwXATv_4BCFxvBZeflGdyEfbLYrIrdBcWSMcOwrSJb4Km5_BxDlH2jW0nuo2oDemN9CCS2h10ox_1xSncGQajx_ryfhECjZEnCPjxDBR0y7SlxGwzvMDYRP1S8s-kYm6nsMPV3mFh7DpR-MP1f4894dqgSnajmO6-X257tIUFgy6&amp;lib=MkRyGNKK3zAISSqriD1Ggd5cHvhhfPBn6">published url</a> to get the JSON data</h4><p>One downside here as there appears to be a redirect, so if using this from curl or wget you might need to mess with it a bit</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCRNXBm0FQd2zNyM2Atc_w.png" /></figure><p>OK thats neat, but what about…</p><h3>The Webhook</h3><p>To make the webhook you create a custom function for the gsheet, a custom function is basically like =LOOKUP, =IF except that you define the functionality.</p><p>Custom functions are fired whenever the input data is changed, this is the mechanism I am exploiting to get our webhook.</p><h4>Create the custom function</h4><p>Back in the script editor add the following code (this will post to a request bin the input from the row parameter):</p><pre>function currentTime() {<br> var d = new Date();<br> var currentTime = d.toLocaleTimeString()<br> return currentTime;<br>}</pre><pre>function CELL_CHANGED(row) {<br> var options = {<br> &#39;method&#39; : &#39;post&#39;,<br> &#39;payload’ : JSON.stringify(row)<br> };<br> <br> UrlFetchApp.fetch(&#39;<a href="http://requestbin.net/r/1hqonni1&#39;">http://requestbin.net/r/1hqonni1&#39;</a>, options);<br> return &quot;UpdatedAt: &quot;+ currentTime()<br>}</pre><h4>Call the function in the sheet</h4><p>Now you update the sheet to have a call to the function that takes the entire row</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0dJJVoJ8erGXTk5rfGZd2A.png" /></figure><p>The row will update to say something like “UpdatedAt: 17:08:28 EEST”</p><p>When you see a successful message from the function return, the webhook has been sent with the row data as an array.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POCtNSikaboTAzHSwdmDsw.png" /></figure><p>This has another awesome advantage, if you were to bake validation into your script you can return to the user a useful error message why some input data is invalid, preventing bad data from ever getting to your webhook.</p><p>The best part is, I was able to achieve all of this in &lt; 30 mins!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cba629151909" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiqup-engineering/create-webhooks-from-google-sheets-cba629151909">Create webhooks from Google sheets</a> was originally published in <a href="https://medium.com/quiqup-engineering">Quiqup Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>