<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mohanish Patel on Medium]]></title>
        <description><![CDATA[Stories by Mohanish Patel on Medium]]></description>
        <link>https://medium.com/@mohanishp1?source=rss-e9e313a7551e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tIO5tM0PrQOV9GfzA06u_Q.jpeg</url>
            <title>Stories by Mohanish Patel on Medium</title>
            <link>https://medium.com/@mohanishp1?source=rss-e9e313a7551e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:00:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mohanishp1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing Syqnest!]]></title>
            <link>https://medium.com/@mohanishp1/introducing-syqnest-a542cd4e0f70?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a542cd4e0f70</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[promotion]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Sat, 18 Nov 2023 09:51:47 GMT</pubDate>
            <atom:updated>2023-11-18T09:51:47.891Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kV_T0zH_MSqYHDiQzlbduQ.png" /></figure><p><em>Note: This is a promotional post to my app, you can skip.</em></p><p>👋 Hey <a href="https://www.reddit.com/r/synqnest/">t</a>here, thanks for checking out.</p><p>I’m excited to introduce you to Synqnest. A mobile app and Chrome extension which lets you view all your Android mobile notifications on your desktop, and interact with it, you can also copy and paste between devices.</p><p>Key Features:</p><ol><li><strong>Manage notifications</strong>: Remove, and interact with notifications. Select specific apps to send in notifications, and select dnd time. The cheery on the cake is you can copy OTP directly from a PC.</li><li><strong>Send text between devices</strong>: You can quickly send selected text to Android’s clipboard and vice versa from Android to PC.</li><li><strong>Encryption</strong>: The app/extension encrypts anything going out from the device and only decrypts on the app/extension side.</li><li><strong>Self notes (Upcoming):</strong> Add notes, images, and files and store them on your device. Chats are stored in your mobile device and synced with extension as and when required rather than on the cloud.</li></ol><p>Join the <a href="https://www.reddit.com/r/synqnest/comments/17y2vkn/introducing_synqnest/">Synqnest Community on reddit</a>!</p><p>I will like you all to use the app and share your thoughts, feedback.<br>Thanks for giveing your time 🤝.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a542cd4e0f70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Real-Time Trading App with Golang, Kafka & Websockets: Charts & Websockets in Flutter]]></title>
            <link>https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-charts-websockets-in-flutter-9e1ac36ff767?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e1ac36ff767</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[flutter]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Wed, 27 Sep 2023 10:13:31 GMT</pubDate>
            <atom:updated>2023-09-27T10:13:31.547Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*nRUaFaDNEWLO73qi.png" /></figure><p>As an engineer, we constantly wonder how different systems work behind the hood. Since I invest a bit of my earnings on a monthly basis, I have had a fair share of experiences with various trading apps. Trading View, Binance, and Zerodha Kite are some of them.</p><p>We will be dividing this blog into 4 Parts:<br><strong>1. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-introduction-setup-ac2ae3a780da"><strong>Introduction &amp; Setup</strong></a><strong><br>2. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-setting-up-kafka-with-golang-5a47e9b1bff8"><strong>Setting up Kafka with Golang</strong></a><strong>.<br>3. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-consumer-service-websockets-57b0a9f12e35"><strong>Consumer Service &amp; Websockets</strong></a><strong>.<br>4. Frontend</strong></p><h4><strong>A Little Preview</strong></h4><p>This blog is the final article in this series. To all those who are coming first time, please visit the previous articles if you want to understand the backend and part where we attach Golang &amp; Kafka.</p><p>You can also choose to skip the previous articles if you only want to understand making charts with WebSockets.</p><h4>Datasource</h4><p>If you want to run go services as a data source, I have updated the <a href="https://github.com/mohanish2504/RealTimeTradingApp/tree/main/backend">GitHub repo</a> and added a setup guide to run the required components.</p><p>If you looking to connect external data sources, here is a link to Binance’s open-source websocket <a href="https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md">api</a>.</p><p><em>Note: I will be using localhost API running on my machine from the above project.</em></p><h3>Flutter Setup</h3><p>Before we proceed further, let’s add some dependenceis. We are going to use syncfusion charts, bloc, dio and web_socket_channel.</p><pre>dio: ^5.3.2<br>flutter_bloc: ^8.1.3<br>syncfusion_flutter_charts: any<br>equatable: any<br>web_socket_channel: ^2.4.0</pre><h4>Folders &amp; Files.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HUnEQSlEf2wTm4UXBlzdaQ.png" /></figure><p>I will be explaining on parts that we require to make justice to this article, mainly web socket and article. To all those who are not familiar with the Bloc pattern, it&#39;s a state management library we are going to use. You don’t have to know it but having experience with it with be good.</p><p><strong>models/ticker.dart</strong></p><pre>  final String? name;<br>  final double? price;<br>  final double? quantity;<br>  final DateTime? timestamp;</pre><p><strong>bloc/cubit/ticker_listener_cubit.dart</strong></p><pre>import &#39;dart:convert&#39;;<br><br>import &#39;package:bloc/bloc.dart&#39;;<br>import &#39;package:equatable/equatable.dart&#39;;<br>import &#39;package:flutter/material.dart&#39;;<br>import &#39;package:frontend/models/ticker.dart&#39;;<br>import &#39;package:frontend/urls.dart&#39;;<br>import &#39;package:meta/meta.dart&#39;;<br>import &#39;package:web_socket_channel/web_socket_channel.dart&#39;;<br><br>part &#39;ticker_listener_state.dart&#39;;<br><br>class TickerListenerCubit extends Cubit&lt;TickerListenerState&gt; {<br>  final String ticker;<br>  WebSocketChannel? _tickerChannel;<br>  TickerListenerCubit(this.ticker) : super(TickerListenerInitial()) {<br>    _listen();<br>  }<br><br>  void _listen() {<br>    _tickerChannel = WebSocketChannel.connect(<br>        Uri.parse(&quot;&lt;websocket url&gt;&quot;);<br>    _tickerChannel?.stream.listen((event) {<br>      // debugPrint(event.runtimeType.toString());<br>      Ticker? ticker = Ticker.fromJson(jsonDecode(event));<br>      if (ticker != null) {<br>        emit(NewTicker(ticker));<br>      }<br>    });<br>  }<br><br>  void closeConnection() {<br>    _tickerChannel?.sink.close();<br>  }<br><br>  @override<br>  Future&lt;void&gt; close() {<br>    // TODO: implement close<br>    closeConnection();<br>    return super.close();<br>  }<br>}</pre><p><strong>_listen() </strong>is where we get connected to the data source which emits the data points to widgets. Let’s have a look over that side.</p><p><strong>app/trading_view.dart</strong></p><pre>import &#39;package:flutter/material.dart&#39;;<br>import &#39;package:frontend/app/chart.dart&#39;;<br>import &#39;package:frontend/models/ticker.dart&#39;;<br>import &#39;package:frontend/repository/trading_repository.dart&#39;;<br><br>class TradingView extends StatefulWidget {<br>  const TradingView({super.key});<br><br>  @override<br>  State&lt;TradingView&gt; createState() =&gt; _TradingViewState();<br>}<br><br>class _TradingViewState extends State&lt;TradingView&gt;<br>    with TickerProviderStateMixin {<br>  List&lt;String&gt; currentTickers = [];<br>  TickerRepository _tickerRepository = TickerRepository();<br><br>  @override<br>  void initState() {<br>    // TODO: implement initState<br><br>    super.initState();<br>  }<br><br><br>  @override<br>  Widget build(BuildContext context) {<br>    return Scaffold(<br>      body: FutureBuilder&lt;List&lt;String&gt;?&gt;(<br>          future: _tickerRepository.fetchTickers(),<br>          builder: (context, snapshot) {<br>            if (!snapshot.hasData) {<br>              return Center(<br>                child: Text(&quot;Sorry! no data&quot;),<br>              );<br>            }<br>            if (snapshot.connectionState == ConnectionState.waiting) {<br>              return Center(<br>                child: CircularProgressIndicator(),<br>              );<br>            }<br><br>            if (snapshot.hasData &amp;&amp; snapshot.data != null) {<br>              currentTickers = snapshot.data!;<br>              currentTickers.sort();<br>            }<br><br>            return Padding(<br>                padding: const EdgeInsets.all(8.0),<br>                child: GridView.builder(<br>                    itemCount: currentTickers.length,<br>                    gridDelegate: SliverGridDelegateWithFixedCrossAxisCount(<br>                        crossAxisSpacing: 4,<br>                        mainAxisSpacing: 4,<br>                        crossAxisCount: 2,<br>                        childAspectRatio: 4 / 2),<br>                    itemBuilder: (ctx, index) {<br>                      String ticker = currentTickers[index];<br>                      return Container(<br>                          margin: EdgeInsets.all(16),<br>                          child: TickerChart(ticker: ticker));<br>                    }));<br>          }),<br>    );<br>  }<br>}</pre><pre>future: _tickerRepository.fetchTickers(),</pre><p>Here I have connected to my backend project which is made in Go for that refer to previous articles or the repository, if you are using any other source please modify accordingly.</p><p>Jumping to the <strong>TickerChart</strong> widget, we are passing the name of the ticker we want to subscribe to.</p><p><strong>app/chart.dart</strong></p><pre>import &#39;package:flutter/material.dart&#39;;<br>import &#39;package:flutter_bloc/flutter_bloc.dart&#39;;<br>import &#39;package:frontend/bloc/cubit/ticker_listener_cubit.dart&#39;;<br>import &#39;package:syncfusion_flutter_charts/charts.dart&#39;;<br>import &#39;dart:math&#39; as math;<br>import &#39;../models/ticker.dart&#39;;<br><br>class TickerChart extends StatefulWidget {<br>  final String ticker;<br>  const TickerChart({super.key, required this.ticker});<br><br>  @override<br>  State&lt;TickerChart&gt; createState() =&gt; _TickerChartState();<br>}<br><br>class _TickerChartState extends State&lt;TickerChart&gt; {<br>  TickerListenerCubit? tickerListenerCubit;<br>  List&lt;Ticker&gt; currTickers = [];<br>  ChartSeriesController? chartSeriesController;<br>  static final int maxLength = 100;<br>  @override<br>  void initState() {<br>    // TODO: implement initState<br>    tickerListenerCubit = TickerListenerCubit(widget.ticker);<br><br>    super.initState();<br>  }<br><br>  double currMax = 1;<br>  @override<br>  Widget build(BuildContext context) {<br>    Color color =<br>        Color((math.Random().nextDouble() * 0xFFFFFF).toInt()).withOpacity(1.0);<br>    return BlocBuilder&lt;TickerListenerCubit, TickerListenerState&gt;(<br>      bloc: tickerListenerCubit,<br>      buildWhen: ((previous, current) {<br>        debugPrint(current.toString() + &quot; &quot; + chartSeriesController.toString());<br>        if (current is NewTicker) {<br>          if (chartSeriesController != null) {<br>            currTickers.add(current.ticker);<br>            //currMax = math.max(currMax, current.ticker.price ?? 0);<br><br>            if (currTickers.length &gt;= maxLength) {<br>              currTickers.removeAt(0);<br>            }<br>          }<br>        }<br>        return true;<br>      }),<br>      builder: (context, state) {<br>        return SfCartesianChart(<br>          //zoomPanBehavior: ZoomPanBehavior(),<br>          legend: Legend(isVisible: true, position: LegendPosition.bottom),<br>          primaryXAxis:<br>              DateTimeAxis(intervalType: DateTimeIntervalType.minutes),<br>          primaryYAxis: NumericAxis(),<br>          enableAxisAnimation: false,<br><br>          series: [<br>            FastLineSeries&lt;Ticker, DateTime&gt;(<br>                color: color,<br>                markerSettings: const MarkerSettings(isVisible: true),<br>                onRendererCreated: (c) =&gt; chartSeriesController = c,<br>                name: widget.ticker.toUpperCase(),<br>                dataSource: currTickers,<br>                animationDuration: 0,<br>                xValueMapper: (t, index) =&gt; t.timestamp,<br>                yValueMapper: (t, index) =&gt; t.price)<br>          ],<br>        );<br>      },<br>    );<br>  }<br>}</pre><p>Have a look a this part.</p><pre>return BlocBuilder&lt;TickerListenerCubit, TickerListenerState&gt;(<br>      bloc: tickerListenerCubit,<br>      buildWhen: ((previous, current) {<br>        debugPrint(current.toString() + &quot; &quot; + chartSeriesController.toString());<br>        if (current is NewTicker) {<br>          if (chartSeriesController != null) {<br>            currTickers.add(current.ticker);<br>            //currMax = math.max(currMax, current.ticker.price ?? 0);<br><br>            if (currTickers.length &gt;= maxLength) {<br>              currTickers.removeAt(0);<br>            }<br>          }<br>        }<br>        return true;<br>      }),<br>      builder: (context, state) {<br>        return SfCartesianChart(<br>          //zoomPanBehavior: ZoomPanBehavior(),<br>          legend: Legend(isVisible: true, position: LegendPosition.bottom),<br>          primaryXAxis:<br>              DateTimeAxis(intervalType: DateTimeIntervalType.minutes),<br>          primaryYAxis: NumericAxis(),<br>          enableAxisAnimation: false,<br><br>          series: [<br>            FastLineSeries&lt;Ticker, DateTime&gt;(<br>                color: color,<br>                markerSettings: const MarkerSettings(isVisible: true),<br>                onRendererCreated: (c) =&gt; chartSeriesController = c,<br>                name: widget.ticker.toUpperCase(),<br>                dataSource: currTickers,<br>                animationDuration: 0,<br>                xValueMapper: (t, index) =&gt; t.timestamp,<br>                yValueMapper: (t, index) =&gt; t.price)<br>          ],<br>        );<br>      },<br>    );<br></pre><p>Here we listen for the state to be emitted, from the Ticker cubit. As shown earlier, each cubit listens for a particular ticker and as soon as it is emitted we pass the data source to Synfussion’s SF chart.</p><p>We are using FastLineSeries here as data points emitted can be pretty fast in real-world scenarios. You can read more on the type of chart and options <a href="https://help.syncfusion.com/flutter/cartesian-charts/getting-started">here</a>.</p><p>That’s it then, now all we have to do is to compile everything together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zAGbVdx67fyQiw-p7VSdKQ.png" /></figure><p>This is how it looks after running. We are done here, you can optimize the UI as you want.</p><p>Now I know I haven’t explained all the files. I have attached all the source code please have a look and comment down any questions you have.</p><p>Link =&gt; <a href="https://github.com/mohanish2504/RealTimeTradingApp">mohanish2504/RealTimeTradingApp (github.com)</a></p><p><strong>Thanks for reading.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e1ac36ff767" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Real-Time Trading App with Golang, Kafka & Websockets: Consumer Service & Websockets]]></title>
            <link>https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-consumer-service-websockets-57b0a9f12e35?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/57b0a9f12e35</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Sat, 23 Sep 2023 08:12:55 GMT</pubDate>
            <atom:updated>2023-09-27T11:02:46.695Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*bSzDQ6T7gtj6iEbp.png" /></figure><p>As an engineer, we constantly wonder how different systems work behind the hood. Since I invest a bit of my earnings on a monthly basis, I have had a fair share of experiences with various trading apps. Trading View, Binance, and Zerodha Kite are some of them.</p><p>We will be dividing this blog into 4 Parts:<br><strong>1. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-introduction-setup-ac2ae3a780da"><strong>Introduction &amp; Setup</strong></a><strong><br>2. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-setting-up-kafka-with-golang-5a47e9b1bff8"><strong>Setting up Kafka with Golang</strong></a><strong>.<br>3. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-consumer-service-websockets-57b0a9f12e35"><strong>Consumer Service &amp; Websockets</strong></a><strong>.<br>4. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-charts-websockets-in-flutter-9e1ac36ff767"><strong>Frontend.</strong></a></p><h4>Creating Consumer Service &amp; Websockets</h4><p>In this series, we have so far covered the part where we went through some basic introduction to tech stack, and high-level architecture and implemented a producer service with Kafka and Golang.</p><p>So far so good. We will be now going through listening for kafka events and pushing ticker data to the front end via web sockets.</p><h4>Why websockets and not kafka on frontend?</h4><p>A quick revision to the introductory post. I have mentioned that there is next to no support for Kafka on browsers.</p><p><strong>Some of the reasons</strong>:</p><ol><li>By default Kafka uses TCP and browsers keep TCP connections open for a very short time.</li><li>Kafka’s policy is to distribute messages across consumers. For each tab, our app opened will be treated as a separate consumer. Thus, each tab won’t receive all the data points but rather will be distributed across not only tabs but across all other devices consuming it.</li><li>Kafka consumers use a decent amount of resources to keep track of offset, message reads, and overall states. Compared to this, WebSockets work is lightweight.</li></ol><p>Let’s not go too deep<strong>.</strong></p><p>I think I have given a good enough justification for using WS to deliver data on the front end. Now we should start with some implementation on the consumer part.</p><p><strong>Folder Structure.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*PEEZgTaNiUitrmuFY8AxHQ.png" /></figure><p>As we did in the last post, we will create folders &amp; files from scratch. The consumer service app we are creating not only will consume the Kafka stream but will also host the API to a list of tickers available and a connection to subscribe to it. You can give whatever name you want.</p><p>Run the below command inside the consumer project folder.</p><pre>go mod init &lt;consumer_app_name&gt;</pre><p><strong>Dependencies for this app</strong>.</p><pre>module app<br><br>go 1.20<br><br>require (<br> github.com/gofiber/contrib/websocket v1.2.1<br> github.com/gofiber/fiber/v2 v2.49.1<br> github.com/joho/godotenv v1.5.1<br> github.com/segmentio/kafka-go v0.4.42<br>)<br><br>require (<br> github.com/andybalholm/brotli v1.0.5 // indirect<br> github.com/fasthttp/websocket v1.5.4 // indirect<br> github.com/google/uuid v1.3.1 // indirect<br> github.com/klauspost/compress v1.16.7 // indirect<br> github.com/mattn/go-colorable v0.1.13 // indirect<br> github.com/mattn/go-isatty v0.0.19 // indirect<br> github.com/mattn/go-runewidth v0.0.15 // indirect<br> github.com/pierrec/lz4/v4 v4.1.15 // indirect<br> github.com/rivo/uniseg v0.4.4 // indirect<br> github.com/savsgio/gotils v0.0.0-20230208104028-c358bd845dee // indirect<br> github.com/valyala/bytebufferpool v1.0.0 // indirect<br> github.com/valyala/fasthttp v1.49.0 // indirect<br> github.com/valyala/tcplisten v1.0.0 // indirect<br> golang.org/x/sys v0.12.0 // indirect<br>)</pre><p>Now you don’t need to know about all dependencies, we’re using fiber just to serve a basic http endpoint. Other dependencies are carried forward from the previous <a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-setting-up-kafka-with-golang-5a47e9b1bff8">producer</a> article.</p><p>Let’s start with files one by one. First, let’s create the core structure of our app.</p><p><strong>.env</strong></p><pre>KAFKA_HOST=127.0.0.1<br>KAFKA_PORT=9092<br><br><br>#Zookeeper<br>ALLOW_ANONYMOUS_LOGIN=yes<br>ZOO_PORT_NUMBER=2181<br><br><br>#Kafka<br>KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181<br>ALLOW_PLAINTEXT_LISTENER=yes<br>KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092<br>TICKERS=&quot;btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt&quot;</pre><p><strong>core/settings.go</strong></p><pre>package core<br><br>import (<br> &quot;log&quot;<br> &quot;os&quot;<br> &quot;strings&quot;<br><br> &quot;github.com/joho/godotenv&quot;<br>)<br><br>var TICKERS []string<br>var KAFKA_HOST string<br>var KAFKA_PORT string<br><br>func Load() {<br><br> err := godotenv.Load()<br> if err != nil {<br>  log.Fatal(&quot;Failed to load environment file&quot;)<br> }<br> t := os.Getenv(&quot;TICKERS&quot;)<br> TICKERS := strings.Split(t, &quot;,&quot;)<br> LoadTikers(TICKERS)<br><br> KAFKA_HOST = os.Getenv(&quot;KAFKA_HOST&quot;)<br> KAFKA_PORT = os.Getenv(&quot;KAFKA_PORT&quot;)<br>}</pre><p><strong>core/ticker.go</strong></p><pre>package core<br><br>import &quot;strings&quot;<br><br><br>var tickerSet map[string]struct{}<br><br>func GetAllTickers() []string{<br> tickerList := []string {};<br> for key := range tickerSet {<br>  tickerList = append(tickerList,key);<br> }<br> return tickerList<br>}<br><br>func IsTickerAllowed(ticker string) bool {<br> _, ok := tickerSet[strings.ToLower(ticker)]<br> return ok<br>}<br><br>func LoadTikers(tickers []string) {<br> if tickerSet == nil {<br>  tickerSet = make(map[string]struct{})<br> }<br> for _, t := range tickers {<br>  tickerSet[strings.ToLower(strings.Trim(strings.Trim(t,&quot;\\&quot;),&quot;\&quot;&quot;))] = struct{}{};<br> }<br>}</pre><p>settings.go is nothing but loading all the env variables, and ticker.go has all the tickers and functions that we will be using later.</p><p>Let’s configure this part in our main.go</p><pre>package main<br><br>import (<br> &quot;app/core&quot;<br>)<br><br>func main() { <br> core.Load()<br>}</pre><p>nothing complicated, just simple core.Load(). Now our env file is loaded.<br>to check you can print tickers</p><pre>fmt.PrintLn(core.GetAllTickers())</pre><p>now only part that remains to build is the API and websocket connection.</p><p><strong>api/ticker.go</strong></p><pre>package api<br><br>import (<br> &quot;context&quot;<br> &quot;fmt&quot;<br> &quot;log&quot;<br> &quot;strings&quot;<br><br> &quot;github.com/gofiber/contrib/websocket&quot;<br> &quot;github.com/gofiber/fiber/v2&quot;<br> &quot;github.com/segmentio/kafka-go&quot;<br><br> &quot;app/core&quot;<br>)<br><br>func GetAllTickers(c *fiber.Ctx) error {<br> c.Status(200).JSON(core.GetAllTickers())<br> return nil<br>}<br><br>func ListenTicker(conn *websocket.Conn) {<br> currTicker := conn.Params(&quot;ticker&quot;)<br> log.Println(&quot;Current ticker: &quot;, currTicker)<br><br> if !core.IsTickerAllowed(currTicker) {<br>  conn.WriteMessage(websocket.CloseUnsupportedData, []byte(&quot;Ticker is not allowed&quot;))<br>  log.Println(&quot;Ticker not allowed ticker: &quot;, currTicker)<br>  conn.Close()<br>  return<br> }<br> topic := &quot;trades-&quot; + strings.ToLower(currTicker)<br> reader := kafka.NewReader(kafka.ReaderConfig{<br>  Brokers: []string{core.KAFKA_HOST + &quot;:&quot; + core.KAFKA_PORT},<br>  Topic:   topic,<br> })<br> reader.SetOffset(-1)<br><br> conn.SetCloseHandler(func(code int, text string) error {<br>  reader.Close()<br>  log.Printf(&quot;Received connection close request. Closing connection .....&quot;)<br>  return nil<br> })<br><br> defer reader.Close()<br> defer conn.Close()<br><br> go func() {<br>  code, wsMessage, err := conn.NextReader()<br>  if err != nil {<br>   log.Println(&quot;Error reading last message from WS connection. Exiting ...&quot;)<br>   conn.Close()<br>   return<br>  }<br>  fmt.Println(&quot; CODE : %d MESSAGE : %s&quot;, code, wsMessage)<br> }()<br><br> for {<br>  message, err := reader.ReadMessage(context.Background())<br>  if err != nil {<br>   log.Println(&quot;Error: &quot;, err)<br>   return<br>  }<br>  fmt.Println(&quot;Reading..... &quot;, string(message.Value))<br><br>  conn.WriteMessage(websocket.TextMessage, message.Value)<br> }<br>}</pre><p>I will be explaining the <strong>ListenTicker</strong> function in detail. Let’s break it down into parts. Have a look at the comments.</p><pre>func ListenTicker(conn *websocket.Conn) {<br>//this is the part where we get ticker value to subscribe and setup initial connenction<br>//example ws://127.0.0.1:8000/ws/trades/btcusdt<br>//this is how out path will look like<br> currTicker := conn.Params(&quot;ticker&quot;)<br> log.Println(&quot;Current ticker: &quot;, currTicker)<br> if !core.IsTickerAllowed(currTicker) {<br>  conn.WriteMessage(websocket.CloseUnsupportedData, []byte(&quot;Ticker is not allowed&quot;))<br>  log.Println(&quot;Ticker not allowed ticker: &quot;, currTicker)<br>  conn.Close() // we close the connection with the message if not dound<br>  return<br> }<br> topic := &quot;trades-&quot; + strings.ToLower(currTicker) // topic<br> reader := kafka.NewReader(kafka.ReaderConfig{<br>  Brokers: []string{core.KAFKA_HOST + &quot;:&quot; + core.KAFKA_PORT},<br>  Topic:   topic,<br> }) //establishing connection to KAFKA<br> reader.SetOffset(-1) // we set reader offset to latest.<br>}</pre><p>once the connection part is done we do some error handling.</p><pre>func ListenTicker(conn *websocket.Conn) {<br>  ... // previous code<br><br>conn.SetCloseHandler(func(code int, text string) error {<br>  reader.Close()<br>  log.Printf(&quot;Received connection close request. Closing connection .....&quot;)<br>  return nil<br> }) // this code will close kafka reader as soon as our ws conn is closed due to any reason.<br><br> defer reader.Close() <br> defer conn.Close() // please check previous article to know why we use defer.<br><br><br>// continuosly listening to ws connection in separate thread so that <br>// if it closes due to some error we close kafka reader too.<br> go func() {<br>  code, wsMessage, err := conn.NextReader()<br>  if err != nil {<br>   log.Println(&quot;Error reading last message from WS connection. Exiting ...&quot;)<br>   conn.Close()<br>   return<br>  }<br>  fmt.Println(&quot; CODE : %d MESSAGE : %s&quot;, code, wsMessage)<br> }() <br><br>// we read from kafka and continously send data to frotend.<br> for {<br>  message, err := reader.ReadMessage(context.Background())<br>  if err != nil {<br>   log.Println(&quot;Error: &quot;, err)<br>   return<br>  }<br>  fmt.Println(&quot;Reading..... &quot;, string(message.Value))<br>  conn.WriteMessage(websocket.TextMessage, message.Value)<br> }<br>}</pre><p>Now let’s link all this and expose ws and api connections.<br>routing.go</p><pre>package api<br><br>import (<br> &quot;github.com/gofiber/fiber/v2&quot;<br> &quot;github.com/gofiber/contrib/websocket&quot;<br>)<br><br>func AddRoutes(app *fiber.App) {<br> app.Use(&quot;/ws&quot;, func(c *fiber.Ctx) error {<br>        if websocket.IsWebSocketUpgrade(c) {<br>            c.Locals(&quot;allowed&quot;, true)<br>            return c.Next()<br>        }<br>        return fiber.ErrUpgradeRequired<br>    })<br> app.Get(&quot;ws/trades/:ticker&quot;,websocket.New(ListenTicker))<br> app.Route(&quot;api/v1&quot;, func(router fiber.Router){<br>  router.Get(&quot;/tickers&quot;,GetAllTickers)<br> })<br>}<br><br></pre><p><strong>main.go</strong></p><pre>package main<br><br>import (<br> &quot;app/api&quot;<br> &quot;app/core&quot;<br> &quot;github.com/gofiber/fiber/v2&quot;<br> &quot;github.com/gofiber/fiber/v2/middleware/cors&quot;<br>)<br><br>func main() {<br> <br> core.Load()<br> app := fiber.New()<br> app.Use(cors.New(cors.Config{<br>        AllowHeaders:     &quot;Origin,Content-Type,Accept,Content-Length,Accept-Language,Accept-Encoding,Connection,Access-Control-Allow-Origin&quot;,<br>        AllowOrigins:     &quot;*&quot;,<br>        AllowCredentials: true,<br>        AllowMethods:     &quot;GET,POST,HEAD,PUT,DELETE,PATCH,OPTIONS&quot;,<br>    }))<br><br> api.AddRoutes(app)<br> app.Listen(&quot;:8000&quot;) <br>}</pre><pre>go build . &amp;&amp; ./app </pre><p>open browser and go to URL <a href="http://127.0.0.1:8000/api/v1/tickers">127.0.0.1:8000/api/v1/tickers</a>. You will get a response something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*3ZgVOjFBhbzSRRuLon1sPQ.png" /></figure><p>Before moving forward make sure you build the producer service and make sure it is running in order to push to Kafka&#39;s current data.</p><p><em>Don’t forget to change </em><strong><em>127.0.0.1</em></strong><em> to </em><strong><em>kafka</em></strong><em> in the </em><strong><em>.env</em></strong><em> file if you are running with docker. Ignore if running on local.</em></p><p>You can quickly do it with this command.</p><pre>go build . &amp;&amp; ./&lt;producer_app_name&gt; #inside producer folder.  <br><br>// or<br><br>sudo docker-compose up -d </pre><p>Now let’s test the web socket stream.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ATF9thDDJO8QLR5sK1uqBg.png" /></figure><p>If you also get the data, then your backend is ready which links to Kafka.</p><h4><strong>Moving towards the end.</strong></h4><p>We are almost done now, we have the data coming from the backend to the frontend. Now in the next article, we will represent the data in charts and will see how it works.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57b0a9f12e35" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Real-Time Trading App with Golang, Kafka & Websockets: Setting up Kafka with Golang]]></title>
            <link>https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-setting-up-kafka-with-golang-5a47e9b1bff8?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a47e9b1bff8</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Fri, 15 Sep 2023 15:24:09 GMT</pubDate>
            <atom:updated>2023-09-27T11:01:23.881Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*pg6WdDPz-71XMOo_.png" /></figure><p>As an engineer, we constantly wonder how different systems work behind the hood. Since I invest a bit of my earnings on a monthly basis, I have had a fair share of experiences with various trading apps. Trading View, Binance, and Zerodha Kite are some of them.</p><p>We will be dividing this blog into 4 Parts:<br><strong>1. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-introduction-setup-ac2ae3a780da"><strong>Introduction &amp; Setup</strong></a><strong><br>2. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-setting-up-kafka-with-golang-5a47e9b1bff8"><strong>Setting up Kafka with Golang</strong></a><strong>.<br>3. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-consumer-service-websockets-57b0a9f12e35"><strong>Consumer Service &amp; Websockets</strong></a><strong>.<br>4. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-charts-websockets-in-flutter-9e1ac36ff767"><strong>Frontend.</strong></a></p><h4><strong>Setting up Kafka with Golang</strong></h4><p>In our previous article, we went through some introductions and clarified the tech stack we will be using to make a full-stack real-time trading app.<br>In that post, I have also mentioned details to set up Kafka with docker and also adding in zookeeper. Let’s revise it again here again quickly.</p><p>docker-compose.yaml</p><pre>version: &#39;3.7&#39;<br>services:<br>  zookeeper:<br>    env_file:<br>    - ./.env<br>    image: bitnami/zookeeper<br>    expose:<br>    - &quot;2181&quot;<br>    ports:<br>    - &quot;2181:2181&quot;<br>  <br>  kafka:<br>    image: bitnami/kafka<br>    env_file:<br>    - ./.env<br>    depends_on:<br>    - zookeeper<br>    ports:<br>    - &#39;9092:9092&#39;</pre><p>.env</p><pre>KAFKA_HOST=127.0.0.1<br>KAFKA_PORT=9092<br><br>#Zookeeper<br>ALLOW_ANONYMOUS_LOGIN=yes<br>ZOO_PORT_NUMBER=2181<br><br><br>#Kafka<br>KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181<br>ALLOW_PLAINTEXT_LISTENER=yes<br>KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092<br><br>TICKERS=&quot;btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt&quot;</pre><p>If you look closely I have changed the hostname to 127.0.0.1 from Kafka, because we are not going to deploy yet. Docker resolves the host name internally not while trying to use it from outside.</p><p>Let’s pull and start the services.</p><pre>sudo docker-compose up -d</pre><h4>Creating the Golang Producer App</h4><p>First and foremost let’s look at how I am setting up the project structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*aMz4-KE0bShJgwqi2Kt_3Q.png" /></figure><p>There are 2 main folders app and producer. The app is the consumer app we will be covering in the next article. For now, we will be covering up producer service. I have merged both apps in a single folder, but kept them as separate services while configuring docker. You can choose to keep both separately.</p><p>To create producer service from scratch. In your root folder run the following folder.</p><pre>mkdir &lt;app_name&gt; &amp;&amp; cd &lt;app_name&gt;<br>go mod init &lt;compayn_name/your_name&gt;/&lt;app_name&gt;</pre><p>A file will be created with the name <strong>go.mod</strong>. That’s the file to keep track of the dependencies we will be using in this project.</p><p>Here are dependencies</p><pre>require (<br> github.com/gorilla/websocket v1.5.0 # to fetch from binance websocket api<br> github.com/joho/godotenv v1.5.1 # to get tickers from env file<br> github.com/segmentio/kafka-go v0.4.42 #sdk to connect with kafka <br>)</pre><p>after adding dependencies you can download it with</p><pre>go mod download</pre><h4>Implementation</h4><p>We are done with basic implementation. Let’s start by creating our first file <strong>main.go</strong></p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;os&quot;<br> &quot;strings&quot;<br> &quot;github.com/joho/godotenv&quot;<br>)<br><br>func main() {<br> err := godotenv.Load(&quot;../.env&quot;)<br> if err != nil {<br>  fmt.Print(&quot;Failed to load environment&quot;)<br> }<br> t := os.Getenv(&quot;TICKERS&quot;)<br> topics := strings.Split(t, &quot;,&quot;)<br> for i,topic := range topics {<br>  topics[i] = strings.Trim(strings.Trim(topic,&quot;\\&quot;),&quot;\&quot;&quot;) <br> }<br>}</pre><p>The above code will read the TICKERS/Crypto we will be listening to from the envfile.</p><h4>Connecting to the data source.</h4><p>As mentioned earlier we will be using Binance’s <a href="https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md">websocket API</a> to get the data source. In your app folder create folder trades and files name <strong>listener.go</strong>, <strong>publish.go</strong> and <strong>ticker.go</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*MnDdJ07n1nHkvg-eMFJ3eA.png" /></figure><p>Let’s go through them one by one.</p><p><strong>ticker.go </strong>contains a model for the ticker data we will be getting from binance and we will be using the same model to publish the data</p><pre>package trades<br><br>type Ticker struct {<br> Symbol string `json:&quot;s&quot;`<br> Price string `json:&quot;p&quot;`<br> Quantity string `json:&quot;q&quot;` <br> Time int64 `json:&quot;T&quot;` // timestamp<br>}</pre><p><strong>listener.go </strong>as the name suggests has the actual code where we listen for particular tickers through web socket. Let’s go deep into it.</p><p>We will setup webscoket connection first</p><pre>package trades<br><br>import (<br> &quot;encoding/json&quot;<br> &quot;log&quot;<br> &quot;github.com/gorilla/websocket&quot;<br>)<br><br>var conn *websocket.Conn<br>const (<br> subscribeId   = 1<br> unSubscribeId = 2<br>)<br><br>func getConnection() (*websocket.Conn, error) {<br> if conn != nil {<br>  return conn, nil<br> }<br> u := url.URL{Scheme: &quot;wss&quot;, Host: &quot;stream.binance.com:443&quot;, Path: &quot;/ws&quot;}<br> log.Printf(&quot;connecting to %s&quot;, u.String())<br> c, resp, err := websocket.DefaultDialer.Dial(u.String(), nil)<br> if err != nil {<br>  log.Printf(&quot;handshake failed with status %d&quot;, resp.StatusCode)<br>  log.Fatal(&quot;dial:&quot;, err)<br> }<br> return c, nil<br>}<br><br>func unsubscirbeOnClose(conn *websocket.Conn, tradeTopics []string) error {<br> message := struct {<br>  Id     int      `json:&quot;id&quot;`<br>  Method string   `json:&quot;method&quot;`<br>  Params []string `json:&quot;params&quot;`<br> }{<br>  Id:     unSubscribeId,<br>  Method: &quot;UNSUBSCRIBE&quot;,<br>  Params: tradeTopics,<br> }<br><br> b, err := json.Marshal(message)<br> if err != nil {<br>  log.Fatal(&quot;Failed to JSON Encode trade topics&quot;)<br>  return err<br> }<br><br> err = conn.WriteMessage(websocket.TextMessage, b)<br><br> return nil<br>}<br><br>func SubScribeAndListen(topics []string) error {<br> conn,err := getConnection()<br> if err != nil {<br>  log.Fatal(&quot;Failed to get connection %s&quot;, err.Error())<br>  return err<br> }<br>  <br> // binance send&#39;s ping every few minutes and requires to ping back in order <br> // to notify active connections<br> conn.SetPongHandler(func(appData string) error {<br>  fmt.Println(&quot;Received pong:&quot;, appData)<br>  pingFrame := []byte{1, 2, 3, 4, 5}<br>  err := conn.WriteMessage(websocket.PingMessage, pingFrame)<br>  if err != nil {<br>   fmt.Println(err)<br>  }<br>  return nil<br> })<br><br> tradeTopics := make([]string, 0, len(topics))<br> for _, topic := range topics {<br>  tradeTopics = append(tradeTopics, topic+&quot;@&quot;+&quot;aggTrade&quot;)<br> }<br> log.Println(&quot;Listening to trades for &quot;, tradeTopics)<br> <br>message := RequestParams{<br>  Id:     subscribeId,<br>  Method: &quot;SUBSCRIBE&quot;,<br>  Params: tradeTopics,<br> }<br> //log.Println(message)<br> b, err := json.Marshal(message)<br> if err != nil {<br>  log.Fatal(&quot;Failed to JSON Encode trade topics&quot;)<br>  return err<br> }<br><br> err = conn.WriteMessage(websocket.TextMessage, b)<br> if err != nil {<br>  log.Fatal(&quot;Failed to subscribe to topics &quot; + err.Error())<br>  return err<br> }<br><br> defer conn.Close()<br> defer unsubscirbeOnClose(tradeTopics)<br><br> for {<br>  _, payload, err := conn.ReadMessage()<br>  if err != nil {<br>   fmt.Println(err)<br>   return err<br>  }<br><br>  trade := Ticker{}<br><br>  err = json.Unmarshal(payload, &amp;trade)<br>  if err != nil {<br>   fmt.Println(err)<br>   return err<br>  }<br>  log.Println(trade.Symbol, trade.Price, trade.Quantity)<br> }<br>}</pre><p><em>To all those who don’t know </em><strong><em>defer</em></strong><em>, it is used to move certain codes to the end of execution. Here close and unsubscribeOnClose will be called whenever this function exits.</em></p><p>Let’s run this code. Call SubscirbeAndListen to main.go</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;os&quot;<br> &quot;strings&quot;<br> &quot;github.com/joho/godotenv&quot;<br>  trades &quot;producer/trades&quot;  // &lt;==== add this<br>)<br><br>func main() {<br> err := godotenv.Load(&quot;../.env&quot;)<br> if err != nil {<br>  fmt.Print(&quot;Failed to load environment&quot;)<br> }<br> t := os.Getenv(&quot;TICKERS&quot;)<br> topics := strings.Split(t, &quot;,&quot;)<br> for i,topic := range topics {<br>  topics[i] = strings.Trim(strings.Trim(topic,&quot;\\&quot;),&quot;\&quot;&quot;) <br> }<br><br>trades.SubScribeAndListen( // &lt;==== add this<br>  topics,<br> )<br> <br>}</pre><p>Build and run the app. Run this code inside your app folder</p><pre>go build . &amp;&amp; ./producer</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rBYsnORfoXpXPN3RGWXS7A.png" /></figure><p>You might have similar logs on printing the data.</p><p><strong>publish.go</strong> Now comes the part where we connect and send data coming from web socket to kafka.</p><pre>package trades<br><br>import (<br> &quot;context&quot;<br> &quot;log&quot;<br> &quot;github.com/segmentio/kafka-go&quot;<br>)<br><br>var (<br> HOST string<br> PORT string<br>);<br><br><br>func LoadHostAndPort(host string, port string){<br> HOST = host<br> PORT = port<br>} <br><br>func Publish(t string, message kafka.Message, topic string) error {<br><br> messages := []kafka.Message{<br>  message,<br> }<br> <br> w := kafka.Writer{<br>  Addr: kafka.TCP(HOST + &quot;:&quot; + PORT), //127.0.0.1:9092 or kafka:9092 in docker<br>  Topic: topic,<br>  AllowAutoTopicCreation: true,<br> }<br> defer w.Close()<br><br> err := w.WriteMessages(context.Background(), messages...)<br> if err != nil {<br>  log.Println(&quot;Error writing messages to Kafka: &quot;, err.Error())<br>  return err<br> }<br><br> log.Println(&quot;Publish messages to Kafka on topic: &quot;, topic)<br> <br> return nil<br>}</pre><p>We will be using this function to publish to kafka. For this add a small change to listener.go.</p><pre>func SubScribeAndListen(topics []string) error {<br><br>  ...<br>  ...<br>  ...<br><br><br> for {<br>  _, payload, err := conn.ReadMessage()<br>  if err != nil {<br>   log.Println(err)<br>   return err<br>  }<br><br>  trade := Ticker{}<br><br>  err = json.Unmarshal(payload, &amp;trade)<br>  if err != nil {<br>   log.Println(err)<br>   return err<br>  }<br><br>  log.Println(trade.Symbol, trade.Price, trade.Quantity)<br>  go func() { // &lt;=== here<br>   convertAndPublishToKafka(trade)<br>  }() <br> }<br>}<br><br><br>// add this function<br>func convertAndPublishToKafka(t Ticker) { <br> bytes, err := json.Marshal(t)<br> if err != nil {<br>  log.Println(&quot;Error marshalling Ticker data&quot;, err.Error())<br> }<br><br> Publish(t.String(), kafka.Message{<br>  Key:   []byte(t.Symbol + &quot;-&quot; + strconv.Itoa(int(t.Time))),<br>  Value: bytes,<br> }, &quot;trades-&quot;+strings.ToLower(t.Symbol))<br>}</pre><p>Why use go func () { … }? People who are using go might already be familiar with Go routines, it is native support to asynchronous execution in Go.<br>Adding go before a function will convert and execute it to a different thread while still sharing the same addresses to objects. This gives the option to maximize performance, at a time many different tickers (BTC,ETH,MATIC…) might produce data. So instead of handling them and blocking the loop until we publish, we indicate go to start listening for next ticker and execute publish part in background.</p><p>This is enough to start producing ticker data through Kafka, after building and running the app you might see something similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WY-T1HSUCm86SMy6LI5svQ.png" /></figure><p>Here we complete our producer service. For any doubts please comment down below. I will be adding all the codes soon after completing the series. Stay Tuned.</p><h3>What’s Next?</h3><p>We will be taking up consumer service where we will be exposing a WebSocket API for Frotend in which data will be put on through Kafka listener.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a47e9b1bff8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making Real-Time Trading App with Golang, Kafka & Websockets: Introduction & Setup.]]></title>
            <link>https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-introduction-setup-ac2ae3a780da?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac2ae3a780da</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[trading]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 20:18:44 GMT</pubDate>
            <atom:updated>2023-09-27T11:01:02.086Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*8X2UT8VfLKn7dNeAS-UE9g.png" /></figure><p>As an engineer, we constantly wonder how different systems work behind the hood. Since I invest a bit of my earnings on a monthly basis, I have had a fair share of experiences with various trading apps. Trading View, Binance, and Zerodha Kite are some of them.</p><p>We will be dividing this blog into 4 Parts:<br><strong>1. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-introduction-setup-ac2ae3a780da"><strong>Introduction &amp; Setup</strong></a><strong><br>2. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-setting-up-kafka-with-golang-5a47e9b1bff8"><strong>Setting up Kafka with Golang</strong></a><strong>.<br>3. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-consumer-service-websockets-57b0a9f12e35"><strong>Consumer Service &amp; Websockets</strong></a><strong>.<br>4. </strong><a href="https://medium.com/@mohanishp1/making-real-time-trading-app-with-golang-kafka-websockets-charts-websockets-in-flutter-9e1ac36ff767"><strong>Frontend.</strong></a></p><h3>High-Level System Design.</h3><p>Before we proceed with the code part, let’s have a look at high-level what are we developing, and the tools we will be using.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*BEIniE3l8Y2y70b8FtKfZQ.png" /></figure><p>This article will guide you through the basics of making such a platform with Golang, Kafka, and Websockets. First things first, let’s justify the components we are using.</p><ol><li><strong>Golang: </strong>Golang directly compiles machine code and is simpler to write which makes it a good choice to make systems which required to run on low latency. Sure there can be a debate about choosing other languages, disadvantages of Go, and so on, but for now Go will get the work done.</li><li><strong>Kafka: </strong>A<strong> </strong>Question arises that since we are using web sockets why not use it between producer and consumer?. 2 major reasons. The first web socket is a bidirectional protocol used for facilitating data exchange. That’s it, it only does that, it never stores data. There can be multiple services that might do a timely analysis of financial data. Distributed architecture is another reason for Kafka. Stock market platforms are usually popular across various countries, which makes Kafka a solid choice over Websocket.</li><li><strong>Websocket: </strong>Now why do we need websocket? I just stated that Kafka is distributed. But here is the issue, it is by default designed to work in a multi-host environment. In Kafka, no one partition can be subscribed by more than one consumer. This means that if we open our app from 2 different tabs, both of them won’t receive the same message but instead will receive messages one by one. Also, not many client-side libraries are made for Kafka due to this reason, especially for the web. In mobile apps, it won’t be an issue since we only have one instance at a time running for a particular app. Using WebSocket makes it a good choice to enable it by default for all the platforms.</li><li><strong>Web App:</strong> We will be using Flutter to create the app with syncfusion charts. I won’t be giving in-depth attention to the front-end app. The purpose is to give a gist of how the overall flow is connected from data source to client.</li></ol><h3>Setup</h3><p><em>Note: From here I will be assuming that as a reader you will be familiar with the basics of the topics we are discussing as well as docker installed on your pc.</em></p><p>Let&#39;s get started. We will be using Binance’s <a href="https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md">open-source web socket connection</a> to feed our data source. The tickers we will be subscribing too:</p><pre>btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt</pre><p>Now that we have decided on our data source, it’s time to set up Kafka. <br>I will be using docker images from bitnami.</p><pre>version: &#39;3.7&#39;<br>services:<br>  zookeeper:<br>    env_file:<br>    - ./.env<br>    image: bitnami/zookeeper<br>    expose:<br>    - &quot;2181&quot;<br>    ports:<br>    - &quot;2181:2181&quot;<br>  <br>  kafka:<br>    image: bitnami/kafka<br>    env_file:<br>    - ./.env<br>    depends_on:<br>    - zookeeper<br>    ports:<br>    - &#39;9092:9092&#39;</pre><p>We have 2 take 2 services to begin with. Zookeeper and Kafka. For those who are unaware with zookeeper, it’s a centralized cluster management system developed by Apache. Zookeeper is usually used when distributed systems come into play, In the context of Kafka it answers some of the questions</p><ol><li>For a given topic &amp; partition which broker should handle the publish/subscribe?</li><li>How many nodes/server instances are available in the cluster?</li><li>Available topics, data retention settings, etc.</li></ol><p>While it is just a gist of it you can read more about it <a href="https://zookeeper.apache.org/">here</a>.</p><p><strong>Why are we using Zookeeper, since for testing and development we don’t need multiple nodes?</strong> <br>Kafka can run without a zookeeper, but it is not recommended by Apache to do so in production, so it’s best we start using it right away in similar fashion.</p><p>Here is the .env file</p><pre>KAFKA_HOST=kafka<br>KAFKA_PORT=9092<br><br>#Zookeeper<br>ALLOW_ANONYMOUS_LOGIN=yes<br>ZOO_PORT_NUMBER=2181<br><br><br>#Kafka<br>KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181<br>ALLOW_PLAINTEXT_LISTENER=yes<br>KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092<br><br>TICKERS=&quot;btcusdt,ethusdt,busdusdt,bnbusdt,ltcusdt,xrpusdt,maticusdt&quot;</pre><p>After this do:</p><pre>sudo docker-compose up -d</pre><p>This should be enough to set up Kafka &amp; Zookeeper in your local environment.</p><h3>What’s Next?</h3><p>We will be starting with the implementation part of producer service in Golang followed by consumer service and frontend app in flutter.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac2ae3a780da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing time-series queries.]]></title>
            <link>https://medium.com/@mohanishp1/optimizing-time-series-queries-b9fe7715ccea?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9fe7715ccea</guid>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Tue, 20 Jun 2023 04:29:31 GMT</pubDate>
            <atom:updated>2023-06-20T04:29:31.380Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_LgFLbFuBlzhys_H-fqIeA.png" /></figure><p>In my previous <a href="https://medium.com/@mohanishp1/chatgpt-ai-gpt-how-social-media-platforms-handle-hashtags-67c8c767f60c">post</a>, we reviewed how indexing helps optimize queries while dealing with a large number of tags. That covered one of the scenarios where indexing is used, time-series-based data is another one.</p><p>Data collected in a timely manner have various use cases. The stock market, IOT devices, Energy management. I have worked with IOT and energy management, I must tell you that it doesn’t take much time for your queries to go from 200ms to 2000ms+. Worse if there are multiple points data points every second.</p><p>Enough babbling….. show me the implementation. Okay cool, so here is some information: I am having 3 million rows of data specifically taken for the experiment, <a href="https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page">source</a>. As usual, I will be using PostgreSQL, since it is the one I am most familiar with.</p><h4>Table:</h4><pre>postgres=# \d trips;<br>                               Table &quot;public.trips&quot;<br>        Column         |           Type           | Collation | Nullable | Default <br>-----------------------+--------------------------+-----------+----------+---------<br> id                    | integer                  |           | not null | <br> vendorid              | bigint                   |           |          | <br> trip_start            | timestamp with time zone |           |          | <br> trip_end              | timestamp with time zone |           |          | <br> passenger_count       | double precision         |           |          | <br> trip_distance         | double precision         |           |          | </pre><pre><br>postgres=# select count(id) from trips;<br>  count  <br>---------<br> 3066766</pre><p>Of all the above columns only 2 of them are important for this article trip_start and trip_end . Have a closer look at data type of these 2 columns:timestamp with time zone .</p><p>Why so? let’s take an example. Considering this table, let’s say we want to query trip records that are started 2:00 PM today. Now actual issue can rise here, where if no timezone is specified database can return results 2:00 PM from another timezone. Timezone differences vary from country to country.</p><p>Let’s check how queries are performing if I search for a particular DateTime range.</p><pre>postgres=# explain analyze select * from trips where trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39; and trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;;<br>                                                                         QUERY PLAN                                                                          <br>-------------------------------------------------------------------------------------------------------------------------------------------------------------<br> Gather  (cost=1000.00..93344.89 rows=21456 width=150) (actual time=1.815..162.397 rows=26 loops=1)<br>   Workers Planned: 2<br>   Workers Launched: 2<br>   -&gt;  Parallel Seq Scan on trips  (cost=0.00..90199.29 rows=8940 width=150) (actual time=61.388..113.703 rows=9 loops=3)<br>         Filter: ((trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39;::timestamp with time zone) AND (trip_start &lt;= &#39;2023-01-01 00:00:00+00&#39;::timestamp with time zone))<br>         Rows Removed by Filter: 1022247<br> Planning Time: 0.320 ms<br> Execution Time: 162.469 ms<br>(8 rows)</pre><p>Here I have filtered trips that were started in the last month of 2022. It took the database roughly 160 ms to return results. This is definitely not bad, but we have to remember that we only have 3 million records, the execution time can shoot up where records are more than just a few million.</p><p>Let’s try to optimize a bit further. In my previous <a href="https://medium.com/@mohanishp1/indexing-a-blessing-in-vertical-scaling-e037c3063b05">article</a>, I have given a short intro to Btree indexes and how it helps optimize performance. We will index on trip_start column.</p><pre>postgres=# create index trip_start_index on trips (trip_start ASC);<br>CREATE INDEX<br><br>postgres=# \d trips;<br>                               Table &quot;public.trips&quot;<br>        Column         |           Type           | Collation | Nullable | Default <br>-----------------------+--------------------------+-----------+----------+---------<br> id                    | integer                  |           | not null | <br> vendorid              | bigint                   |           |          | <br> trip_start            | timestamp with time zone |           |          | <br> trip_end              | timestamp with time zone |           |          | <br> passenger_count       | double precision         |           |          | <br> trip_distance         | double precision         |           |          | <br> <br>Indexes:<br>    &quot;trips_pkey&quot; PRIMARY KEY, btree (id)<br>    &quot;trip_start_index&quot; btree (trip_start)</pre><p>In the above code you can see, trip_start_index is a tree index. By default, PostgreSQL uses Btree as index type.</p><p>What happens if we use the same query now?</p><pre>postgres=# explain analyze select * from trips where trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39; and trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;;<br>                                                                           QUERY PLAN                                                                           <br>----------------------------------------------------------------------------------------------------------------------------------------------------------------<br> Bitmap Heap Scan on trips  (cost=6.36..732.16 rows=188 width=150) (actual time=0.010..0.029 rows=25 loops=1)<br>   Recheck Cond: ((trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39;::timestamp with time zone) AND (trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;::timestamp with time zone))<br>   Heap Blocks: exact=20<br>   -&gt;  Bitmap Index Scan on trip_start_index  (cost=0.00..6.31 rows=188 width=0) (actual time=0.005..0.005 rows=25 loops=1)<br>         Index Cond: ((trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39;::timestamp with time zone) AND (trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;::timestamp with time zone))<br> Planning Time: 20.266 ms<br> Execution Time: 0.067 ms<br>(7 rows)</pre><p>160 ms to 0.067 ms this is an insane improvement over the last one. In major cases, we trade space to achieve good speed. The same happened in this case too.</p><pre>postgres=# select indexname, pg_size_pretty(pg_relation_size(indexname::regclass)) as size from pg_indexes where tablename = &#39;trips&#39;;<br>    indexname     | size  <br>------------------+-------<br> trips_pkey       | 66 MB<br> trip_start_index | 53 MB<br>(2 rows)</pre><p>We are adding additions of 53 Mb to make our queries faster.</p><p>What if I tell you there is also one more index which also lowers response time as well as very cheap in storage?</p><p>BRIN index! Have a detailed look over <a href="https://www.postgresql.org/message-id/20130614222805.GZ5491@eldon.alvh.no-ip.org">here</a> on how it stores data.</p><pre>postgres=# create index trip_start_index on trips using brin (trip_start);</pre><p>not using asc as brin doesn’t support sort order indexing. <br>Result:</p><pre>explain analyze select * from trips where trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39; and trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;;<br>                                                                              QUERY PLAN                                                                              <br>----------------------------------------------------------------------------------------------------------------------------------------------------------------------<br> Gather  (cost=1022.96..85319.83 rows=21455 width=150) (actual time=1.501..28.609 rows=25 loops=1)<br>   Workers Planned: 2<br>   Workers Launched: 2<br>   -&gt;  Parallel Bitmap Heap Scan on trips  (cost=22.96..82174.33 rows=8940 width=150) (actual time=3.683..11.867 rows=8 loops=3)<br>         Recheck Cond: ((trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39;::timestamp with time zone) AND (trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;::timestamp with time zone))<br>         Rows Removed by Index Recheck: 22008<br>         Heap Blocks: lossy=1223<br>         -&gt;  Bitmap Index Scan on trip_start_index  (cost=0.00..17.60 rows=275459 width=0) (actual time=0.667..0.667 rows=15360 loops=1)<br>               Index Cond: ((trip_start &gt;= &#39;2022-12-01 00:00:00+00&#39;::timestamp with time zone) AND (trip_start &lt; &#39;2023-01-01 00:00:00+00&#39;::timestamp with time zone))<br> Planning Time: 15.372 ms<br> Execution Time: 28.789 ms<br>(11 rows)</pre><p>Wait…. this query performs indeed slower compared to Btree indexes. But have a look at space consumed.</p><pre>postgres=# select indexname, pg_size_pretty(pg_relation_size(indexname::regclass)) as size from pg_indexes where tablename = &#39;trips&#39;;<br>    indexname     | size  <br>------------------+-------<br> trips_pkey       | 66 MB<br> trip_start_index | 32 kB<br>(2 rows)</pre><p><strong>32 Kb </strong>compared to <strong>53 MB</strong> in the tree index. 99.99% of space saving. Using BRIN can be much more favorable in conditions where data is growing at an exponential rate.</p><h4>Conclusion:</h4><p>I am not favoring any particular index. It’s up to the person to decide which one to pick and evaluate their query patterns. I would recommend sorting by DESC in btree index if latest time ranges are queried more frequently than earlier ones.</p><p>That was my 2 cents! Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9fe7715ccea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#Chatgpt #AI #gpt …: How social media platforms handle hashtags]]></title>
            <link>https://medium.com/@mohanishp1/chatgpt-ai-gpt-how-social-media-platforms-handle-hashtags-67c8c767f60c?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/67c8c767f60c</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Sun, 04 Jun 2023 16:01:08 GMT</pubDate>
            <atom:updated>2023-06-04T16:11:51.748Z</atom:updated>
            <content:encoded><![CDATA[<h3>#Chatgpt #AI #gpt….: How social media platforms handle tags</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t09INbutboN_A9cIk4P-1w.png" /></figure><p>In my previous <a href="https://medium.com/@mohanishp1/indexing-a-blessing-in-vertical-scaling-e037c3063b05">post</a>, I have shared a brief introduction to Indexing. Now it’s time to implement some stuff and check out the actual workings of some real-life examples. Our foremost example will take deep dive into how tags work on popular social media platforms. To add a bit more fun to it we will use trending tags such as #Chatgpt #AI and so on. While the exact implementation might not be the same for all the platforms, underlying concepts are commonly implemented with indexing.</p><p>I am much more familiar with PostgreSQL, hence will be using the same throughout articles in this series.</p><h3>Model:</h3><pre>#tags.py<br>class Tag():<br>    id = Column(Integer, primary_key=True)<br>    name = Column(String) #tags will be stored here.</pre><p>Wait… why there is no unique constraint on name? Because….</p><h3>Unique Constraint is itself an Unique Index:</h3><p>Yes, under the hood most sql platforms implement unique constraints as uniqiue index. Have a look at Postgres <a href="https://www.postgresql.org/docs/current/indexes-unique.html#:~:text=PostgreSQL%20automatically%20creates%20a%20unique,mechanism%20that%20enforces%20the%20constraint.">documentation</a> for the same:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XKU5CvQqJ7-Na371qEYuUA.png" /></figure><p>Moving forward… the reason to not add a unique constraint is that we can compare the result with and without indexing.</p><h3>Let’s cut to the chase:</h3><p>I have created roughly <strong>1 million</strong> unique tags.</p><pre>postgres=# select count(*) from tags;<br>  count  <br>---------<br> 1105568<br>(1 row)</pre><p>what’s the result if we have queried a tag without any indexing?</p><pre>postgres=# explain analyze select * from tags where name = &#39;#chatgpt&#39;;<br>                                                     QUERY PLAN                                                     <br>--------------------------------------------------------------------------------------------------------------------<br> Gather  (cost=1000.00..14835.27 rows=1 width=27) (actual time=56.066..58.581 rows=1 loops=1)<br>   Workers Planned: 2<br>   Workers Launched: 2<br>   -&gt;  Parallel Seq Scan on tags  (cost=0.00..13835.17 rows=1 width=27) (actual time=27.404..27.404 rows=0 loops=3)<br>         Filter: ((name)::text = &#39;#chatgpt&#39;::text)<br>         Rows Removed by Filter: 368530<br> Planning Time: 0.125 ms<br> Execution Time: 58.608 ms<br>(8 rows)</pre><p>Postgres directly went for a sequential scan as it had no other option.</p><p>Have a look when indexing (unique constraint) is added.</p><pre>postgres=# explain analyze select * from tags where name = &#39;#chatgpt&#39;;<br>                                                     QUERY PLAN                                                     <br>--------------------------------------------------------------------------------------------------------------------<br> Index Scan using &quot;Unique Tag&quot; on tags  (cost=0.43..8.45 rows=1 width=27) (actual time=0.121..0.126 rows=1 loops=1)<br>   Index Cond: ((name)::text = &#39;#chatgpt&#39;::text)<br> Planning Time: 0.212 ms<br> Execution Time: 0.173 ms<br>(4 rows)</pre><p>Compare to the previous query, this time Postgres used an index scan as we have added a unique constraint.</p><p>If we round numbers then we went from <strong>59 ms</strong> to <strong>1 ms</strong> ~= <strong>98.31%</strong> improvement.</p><h3>What’s the trade-off?</h3><p>In my previous post, I mentioned that in order to achieve better performance over time we usually trade space. How much did we trade this time?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YMHknGykAYBnq6Ob_OmdKQ.png" /></figure><p><strong>47 mb</strong> for 1 million tags and 98% improvement. I think we are in a good spot here. However, space will increase as the tags are added.</p><h3>What’s next?</h3><p>Given we can quickly find tags, we can fetch posts with related tags. In general case there are 3 models</p><pre>class Tag(Base):<br>    __tablename__ = &#39;tags&#39;<br>    id = Column(Integer, primary_key=True)<br>    name = Column(String, unique=True)<br><br>    def __repr__(self):<br>        return f&quot;Tag(id={self.id}, name=&#39;{self.name}&#39;)&quot;<br><br>class Post(Base):<br>    __tablename__ = &#39;posts&#39;<br>    id = Column(Integer, primary_key=True)<br>    content = Column(String)<br><br>    def __repr__(self):<br>        return f&quot;Post(id={self.id}, content=&#39;{self.content}&#39;)&quot;<br><br>class PostTag(Base):<br>    __tablename__ = &#39;post_tags&#39;<br>    id = Column(Integer, primary_key=True)<br>    post_id = Column(Integer, ForeignKey(&#39;posts.id&#39;)))<br>    tag_id = Column(Integer, ForeignKey(&#39;tags.id&#39;))<br><br>    def __repr__(self):<br>        return f&quot;PostTag(post_id={self.post_id}, tag_id={self.tag_id})&quot;</pre><h3>Is that it?</h3><p>Of course not, we have experimented with only a million tags. Real-life platforms like Instagram, LinkedIn, and such, have billions of active users. A million is the bare minimum for them. A lot of techniques were also added on top of this: Caching, Sharding &amp; Partitioning ( Horizontal Scaling).</p><p>At the same time, this is not the usual case for all companies and hence to avoid over-engineering efforts, Indexing is a good place to start with.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67c8c767f60c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Indexing: A blessing in vertical scaling]]></title>
            <link>https://medium.com/@mohanishp1/indexing-a-blessing-in-vertical-scaling-e037c3063b05?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e037c3063b05</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Mon, 15 May 2023 14:31:56 GMT</pubDate>
            <atom:updated>2023-05-15T14:31:56.075Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-YpBNPWQSiugaGHnUePHg.jpeg" /></figure><p>When it comes to managing large-scale databases, vertical scaling plays a crucial role in maintaining the performance of the system. However, as the size of the database grows, it becomes increasingly challenging to retrieve data efficiently, especially when it comes to complex queries. This is where Indexing comes into the picture.</p><p>You might have heard of Btree and B+ trees in advanced data structures, at that moment you might be wondering where we are gonna use them. Indexing is one such technique that extensively uses many such data structures Here is a nice tutorial if you aren’t familiar with Btrees: <a href="https://www.youtube.com/watch?v=aZjYr87r1b8&amp;ab_channel=AbdulBari">Btrees</a>.</p><p>What does Indexing do exactly? To be honest, you are better off doing in-depth indexing than even thinking about any kind of sharding or partitioning. Let’s assume you have some millions of rows, and as the table growing larger, your queries are becoming slower. Indexing helps improve data retrieval, by storing the correct locations of the exact data required rather than scanning the whole table for each query. This in turn reduces the number of times the disk is accessed. The process involves indexing or we can say storing the location of the most frequent column values, so that database knows already where to find the expected results.</p><p>One of the most common step used during optimization of time-based queries is to trade space in order to get a quicker response. In addition to actual table size there is space dedicated for indexing which is stored as Btree, B+ tree, GIN, BRIN, etc.</p><p>Indexing should be enough, to add an edge to your queries if done right. Regardless, if you are still in a situation where more performance is required you can consider sharding and partitioning.</p><p>Going forward I will be adding some examples where Indexing is used in real life and we will climb our way from vanila database to database with indexing and will compare the results. I will attach the links as when I add the articles.</p><p>Thanks for reading till here. I hope you find my writing resourceful 😄.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e037c3063b05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Highly available databases, as difficult and costly as it sounds.]]></title>
            <link>https://medium.com/@mohanishp1/highly-available-databases-as-difficult-and-costly-as-it-sounds-21a1ac8844da?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/21a1ac8844da</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[business-strategy]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Wed, 10 May 2023 15:03:32 GMT</pubDate>
            <atom:updated>2023-05-10T15:03:32.806Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*moKLvTHLg3iwosHhgtPm2Q.jpeg" /></figure><p>In software engineering, the terms highly available system, fault-tolerant systems, and 24/7 systems are used quite frequently while designing, and discussing software architectures. However, building and operating high-availability systems can be challenging and costly for several reasons.</p><p>After load balancing, distributed systems are an increasingly common technique to achieve the goal of this kind of system. Implementing this by own might become a nightmare for a developer. Fortunately, cloud providers like AWS, GCP, and Azure have come a long way and given us the option to go for ready-made solutions, even though costlier for businesses.</p><p>Let’s take examples of this in the context of Databases. Distributed databases mean stored data is replicated to multiple working nodes in a cluster.</p><h3>Data Replication</h3><p>Is everything replicated? No. Suppose if we have, N nodes in our server at least K nodes (this number is configurable) should have the replica. Thus, adding cost to pocket. A lot of components work together under the hood, networking, hardware, and software. In such an environment, data is usually loaded with techniques such as sharding. This also requires specialized skill sets.</p><h3>Consistency</h3><p>Additionally, consistency is also added to the total cost. Writing and maintaining database versions at every write takes time. There are various consistency techniques to opt for:</p><ol><li>Strong consistency: Data is only added if and only all the desired K nodes accept writes request. Resulting in immediate to all replicas.</li><li>Weak consistency: Data is not immediately replicated, and the user may not see the updated values.</li><li>Eventual consistency: Data is replicated eventually over time.</li></ol><p>Databases such as Amazon Dynamo DB and Casandra are based on eventual consistency. Not getting consistent data, may or may not result in faulty transactions.</p><h3>Skills</h3><p>Many distributed database management systems (DBMSs) are proprietary or require specialized skills to operate effectively, which can increase the cost of licensing or hiring skilled staff. Additionally, distributed DBMSs may require additional software components, such as containerization technologies or distributed queries.</p><p>The kind of companies that wants high availability is Financial institutes, Social Media, Ecommerce, and many more. The above components are just a few of them which directly contribute to business expenses.</p><p>Overall, while distributed databases offer many benefits in terms of scalability, fault-tolerance, and high availability, they can be costlier to operate than centralized databases due to the additional hardware, software, and networking infrastructure required to support them.</p><p>Thanks for reading. <br>You can follow me for such content 😃.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21a1ac8844da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Web Applications with Django and FastAPI: Combining the Best of Both Worlds]]></title>
            <link>https://medium.com/@mohanishp1/building-web-applications-with-django-and-fastapi-combining-the-best-of-both-worlds-1892719a8b9d?source=rss-e9e313a7551e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1892719a8b9d</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Mohanish Patel]]></dc:creator>
            <pubDate>Mon, 01 May 2023 04:28:27 GMT</pubDate>
            <atom:updated>2023-05-01T12:59:36.091Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_mVXTWlzkjNfm5GNOdEgA.jpeg" /></figure><p>In python ecosystem there are multiple frameworks available for making a robust backend. Django and FastAPI are the most popular ones. Django comes with many out-of-the-box features such as ORM, Middleware, Authentication, Admin Panel etc. FastAPI on the other hand comes with Async-ready support and is a super fast, lightweight framework to generate Rest APIs.</p><p>The aim of this article is to explore how to leverage power of both frameworks in production. We will first be setting up a simple Django App and then will move towards integrating FastAPI.</p><p><em>Note: This article assumes that you are familiar with both frameworks and SQLAlchemy and have used them in past projects.</em></p><h4><strong>Django Part</strong></h4><p>I will be assuming that you have base django project ready at your end, if not please keep it ready before going forward. Here is the tutorial for doing so <a href="https://realpython.com/django-setup/">Your First Steps With Django: Set Up a Django Project — Real Python</a>.</p><p>Firstly I will start basic Book app</p><pre>django-admin startapp book</pre><p>Add book app to installed apps</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HecZRxZzuVzbkaNrLeeqfw.png" /></figure><p>Adding models Author and Book</p><pre>from django.db import models<br><br>class Author(models.Model):<br>    name = models.CharField(max_length=200)<br><br>class Book(models.Model):<br>    title = models.CharField(max_length=200)<br>    author = models.ForeignKey(Author,on_delete=models.CASCADE)<br>    published_date = models.DateField()</pre><p>Making migrations and migrating changes to the database.</p><pre>python manage.py makemigrations &amp;&amp; python manage.py migrate</pre><p>You might be able to see both new tables added with prefixes as app_name in your database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*p7LRN7d68TXDZUYghsZ-og.png" /></figure><p>Make sure these two tables are configured. Now we will register these models in the admin panel and add some of my favorite books and their authors.</p><p>Screenshot of book samples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rUqcyZv1GeHI3VvjdUICXg.png" /></figure><p>authors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WYAqj_StuCj-Li1nOFZ6Ig.png" /></figure><p>Finally, we are done with setting up Django. Moving forward with the FastAPI setup</p><h4><strong>FastAPI setup</strong></h4><p>Unlike Django, FastAPI’s setup requires a bit of work. To people who are unfamiliar with FastAPI, I would recommend getting a bit more comfortable with it first and continuing the article. Adding some references from official documents:</p><ol><li><a href="https://fastapi.tiangolo.com/tutorial/first-steps/">First Steps — FastAPI (tiangolo.com)</a>.</li><li><a href="https://github.com/tiangolo/full-stack-fastapi-postgresql">GitHub — tiangolo/full-stack-fastapi-postgresql: Full stack, modern web application generator. Using FastAPI, PostgreSQL as database, Docker, automatic HTTPS and more.</a><br>( I use this project generation for FastAPI).</li></ol><p>Make sure you have the base setup ready for FastAPI and Sqlalchemy with Alembic.</p><p>Here’s how my project looks. I will be adding a link to the repo at the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*CiqYHYyytcpyv-F7XtbCXA.png" /></figure><p>Let’s start with the database (named as db) part. We will be creating base model and binding the database engine with a session.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*KuSuO7szjK_R5qanQzy3mg.png" /></figure><p>These are the files usually require to start with db connection. You can take this from the repo I have added in bottom.</p><p><strong>session.py</strong></p><pre>from sqlalchemy import create_engine<br>from sqlalchemy.orm import sessionmaker<br><br>from app.core.config import settings<br><br>engine = create_engine(settings.SQLALCHEMY_DATABASE_URI, pool_pre_ping=True)<br>SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)</pre><p>This is the usual setup for initializing session config, we will be initiating Metadata obj of sqlalchemy. Metadata class contains information on all the tables present in the given connection.</p><pre>from sqlalchemy import create_engine, MetaData<br>from sqlalchemy.orm import sessionmaker<br>from app.core.config import settings<br><br>engine = create_engine(settings.SQLALCHEMY_DATABASE_URI, pool_pre_ping=True)<br>SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)<br>metadata = MetaData(bind=engine)</pre><p>Now comes, the part you might be waiting for. Adding models! Have a look at the files I have created. Currently, both files are empty!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*ajq3utZOYXO_dupJci45bQ.png" /></figure><p>We will be importing metadata from app.db.session and pointing to existing tables that we have already created.</p><p><strong>books.py</strong></p><pre>from app.db.session import metadata, engine<br>from sqlalchemy import Table<br><br>book_table = Table(&#39;&lt;table_name&gt;&#39;,metadata,autoload_with=engine)</pre><p>replace &lt;table_name&gt; with the exact one you have in the database in our example I will be using book_book for the book table and book_author for the author table.</p><p>Now only part left to completely assign a model is to add a table in its config.</p><pre>from app.db.session import metadata, engine<br>from app.db.base import Base<br>from sqlalchemy import Table<br><br>book_table = Table(&#39;book_book&#39;,metadata,autoload_with=engine)<br>class Book(Base):<br>    __table__ = book_table</pre><p><strong>author.py</strong></p><pre>from app.db.session import metadata,engine<br>from app.db.base import Base<br>from sqlalchemy import Table<br><br>author_table = Table(&#39;book_author&#39;,metadata,autoload_with=engine)<br>class Author(Base):<br>    __table__ = author_table</pre><p>Done? No! One more important step is still left. I know this article got long but stay with me.</p><p>We need to tell alembic not to do any migration on these models, as Django is managing these entities. You can also do the reverse, but I think that is a more complex version.</p><p><strong>alembic/env.py</strong></p><pre>def run_migrations_offline() -&gt; None:<br>    &quot;&quot;&quot;Run migrations in &#39;offline&#39; mode.<br><br>    This configures the context with just a URL<br>    and not an Engine, though an Engine is acceptable<br>    here as well.  By skipping the Engine creation<br>    we don&#39;t even need a DBAPI to be available.<br><br>    Calls to context.execute() here emit the given string to the<br>    script output.<br><br>    &quot;&quot;&quot;<br>    url = config.get_main_option(&quot;sqlalchemy.url&quot;)<br>    context.configure(<br>        url=url,<br>        target_metadata=target_metadata,<br>        literal_binds=True,<br>        dialect_opts={&quot;paramstyle&quot;: &quot;named&quot;},<br>    )<br><br>    with context.begin_transaction():<br>        context.run_migrations()<br><br>def run_migrations_online() -&gt; None:<br>    &quot;&quot;&quot;Run migrations in &#39;online&#39; mode.<br><br>    In this scenario we need to create an Engine<br>    and associate a connection with the context.<br><br>    &quot;&quot;&quot;<br>    connectable = engine_from_config(<br>        config.get_section(config.config_ini_section, {}),<br>        prefix=&quot;sqlalchemy.&quot;,<br>        poolclass=pool.NullPool,<br>    )<br><br>    with connectable.connect() as connection:<br>        context.configure(<br>            connection=connection, target_metadata=target_metadata<br>        )<br><br>        with context.begin_transaction():<br>            context.run_migrations()</pre><p>These are 2 methods to run and make migrations in the case of alembic. You have to change this above code to the given below code.</p><pre>def include_object(object, name, type_, reflected, compare_to):<br>    if type_ == &quot;table&quot;  and reflected and compare_to is None:<br>        return False<br>    else:<br>        return True<br><br>def run_migrations_offline() -&gt; None:<br>    &quot;&quot;&quot;Run migrations in &#39;offline&#39; mode.<br><br>    This configures the context with just a URL<br>    and not an Engine, though an Engine is acceptable<br>    here as well.  By skipping the Engine creation<br>    we don&#39;t even need a DBAPI to be available.<br><br>    Calls to context.execute() here emit the given string to the<br>    script output.<br><br>    &quot;&quot;&quot;<br>    url = config.get_main_option(&quot;sqlalchemy.url&quot;)<br>    context.configure(<br>        url=url,<br>        target_metadata=target_metadata,<br>        literal_binds=True,<br>        include_object=include_object,<br>        dialect_opts={&quot;paramstyle&quot;: &quot;named&quot;},<br>    )<br><br>    with context.begin_transaction():<br>        context.run_migrations()<br><br>def run_migrations_online() -&gt; None:<br>    &quot;&quot;&quot;Run migrations in &#39;online&#39; mode.<br><br>    In this scenario we need to create an Engine<br>    and associate a connection with the context.<br><br>    &quot;&quot;&quot;<br>    connectable = engine_from_config(<br>        config.get_section(config.config_ini_section, {}),<br>        prefix=&quot;sqlalchemy.&quot;,<br>        poolclass=pool.NullPool,<br>    )<br><br>    with connectable.connect() as connection:<br>        context.configure(<br>            connection=connection, target_metadata=target_metadata,include_object=include_object,<br>        )<br><br>        with context.begin_transaction():<br>            context.run_migrations()</pre><p>include_object function as the name suggests compares the tables, sequences, and relationships to pick for migrations.</p><p>We are avoiding the models which are reflected i.e. that is taken from already initialized tables.</p><p>That’s it now you are ready to run your APIs.</p><pre>uvicorn app.main:app --reload</pre><p>Try apis on open api docs. You will get similar results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qn2EKVmLeBF9shhp1OfbSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9u8kATqcSTj8U29ST5wwBw.png" /></figure><p>We are done here, now you can make fast performant APIs with FastAPI and at the same time use the admin panel with Django.</p><p>Thanks for reading through.</p><p><a href="https://github.com/mohanish2504/django-fastapi-combo">Github repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1892719a8b9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>