<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jayant Jain on Medium]]></title>
        <description><![CDATA[Stories by Jayant Jain on Medium]]></description>
        <link>https://medium.com/@jayantjain100?source=rss-5e941e6d6a6e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*H__uxb4MrtqtbLR2AECN9w.jpeg</url>
            <title>Stories by Jayant Jain on Medium</title>
            <link>https://medium.com/@jayantjain100?source=rss-5e941e6d6a6e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:32:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jayantjain100/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Breaking Free from Work Guilt]]></title>
            <link>https://medium.com/@jayantjain100/breaking-free-from-work-guilt-694f18e36416?source=rss-5e941e6d6a6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/694f18e36416</guid>
            <category><![CDATA[habits-for-success]]></category>
            <category><![CDATA[time-tracking]]></category>
            <category><![CDATA[work-life-balance]]></category>
            <category><![CDATA[time-management]]></category>
            <dc:creator><![CDATA[Jayant Jain]]></dc:creator>
            <pubDate>Thu, 23 May 2024 21:27:33 GMT</pubDate>
            <atom:updated>2024-05-26T07:12:25.079Z</atom:updated>
            <content:encoded><![CDATA[<h4>Crafting a System for Work-Life Balance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6EGo78QR7G7Lxs8roevLVg.png" /><figcaption>A depiction of work-life balance, credit: ChatGPT</figcaption></figure><p><em>“How is it Sunday night already? I didn’t get anything done. Maybe I should have just gone to that party. Let me sleep early tonight, maybe this week will be different.”</em></p><p>Sound familiar? If you constantly battle work guilt and find yourself sacrificing weekends to catch up, this article is for you. Three years ago, fresh out of grad school, this was me. My days were a rollercoaster, bursts of productivity followed by stretches of minimal progress. The constant weight? Guilt. Every weekend, the cycle repeated. Cancel plans, mentally commit to work, then barely touch it. I was in a prison of my own making.</p><h3>Isn’t a Routine Enough?</h3><p>You might think a simple 9–to–5 schedule is the answer. After all, that’s what most of us have seen our parents do all their lives. The problem? Life. That unexpected oversleeping, the occasional personal errand, the coworker who drags you to an impromptu game of pool — the list goes on. Can a rigid schedule handle these inevitable disruptions?</p><p>In my experience (limited as it may be!), schedules work well for people with constraints: a child’s bedtime, crucial morning meetings, or a packed hobby schedule. For the rest of us, self-control can be a fickle friend. I, for one, knew constraints would be my saviour, not self-control. As a bachelor with a relaxed work culture and no other commitments, freedom backfired.</p><h3>Tracking your focus time</h3><p>A year in, I craved artificial constraints. I tried the 9–to–5 routine, but one slip-up — that late-night party, that long phone call — and I was back to square one. Committing to getting specific tasks done in a certain period also failed. Luckily, I stumbled upon this quote by James Clear:</p><blockquote>You do not rise to the level of your goals. You fall to the level of your systems.</blockquote><p>I realised that committing to specific times of the day or committing to completing certain tasks did not work for me. Instead, I had to <strong>commit to the amount of time I’d spend on my work</strong>. Before I get into the details, keep in mind that the system I am about to propose is not a rigid set of rules. You can tweak it to fit your needs, and over time, it can evolve as your work and life change.</p><p>Before we commit the amount of time, we need a way to track this time. I define a <strong><em>unit</em></strong> as 90 minutes of focus time (usually not done in one go). I always have a timer at my desk counting down such <em>units</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*enfjB5Xux8k5fhP6-EiqBA.jpeg" /><figcaption>You can see the timer on my desk. It is paused right now since I am not working. I am counting down from 90 minutes, which means I have completed 19m25s of work till now. The app I use is called <a href="https://play.google.com/store/apps/details?id=com.brunnock.bigtimer&amp;hl=en&amp;gl=US">Sean’s Big Timer</a>.</figcaption></figure><p>Anytime I pick up the phone, go to the loo, talk to a colleague, or grab a snack, I manually pause the timer! The aim is to do 3 <em>units</em> every day, that’s all. I don’t count meetings as part of this focus time.</p><h4>Wait, is this enough?</h4><p>On the surface, it might seem like a walk in the park. But trust me, quantify your current focus time; you might be surprised! Completing one <em>unit</em> (90 minutes) often takes me 2 hours of wall time. So, 3 <em>units</em> translate to roughly 6 hours of “working” for me. Add an hour each for lunch and meetings, and that’s my 8-hour workday.</p><h4>What if I’m unable to do 3 <em>units</em> on a particular day?</h4><p>Life doesn’t always cooperate. The beauty of this system is that it embraces the inevitable. We don’t enforce daily goals but aim for a <strong>weekly target of 15 <em>units</em></strong>. You might cram 4 <em>units</em> in one day and cruise through 2 on another, but by the end of the week, you hit 15 — no exceptions. The logic? We can’t control daily schedules, but across the week, we can catch up. Surprisingly (or unsurprisingly), in the 50+ weeks that I’ve followed this, I have never had a perfect week i.e. not a single week where I had exactly 3 units each day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TyJADbl54DQFWCdIuK2zEg.png" /><figcaption>I track my daily and weekly output at the top of my calendar. Light green means “successfully met the target”. The other colours highlight deviations e.g. red is for low output (≤ 1.5), orange is for “close but not enough” (2 or 2.5) and dark green is for overperforming (≥ 4). Similarly, I use purple to track work on holidays and weekends.</figcaption></figure><h4>What if I’m unable to do 15 units by Friday night?</h4><p>The weekend is your safety net. I realise that this is less than ideal, but the silver lining is that <strong>you have quantified how much you need to work</strong>. Say, you got 14 <em>units</em> during the weekdays, then you know that 1 <em>unit</em> is pending. You can easily make plans for Saturday and still cover that 1 <em>unit</em> on Sunday afternoon. If you have 5 <em>units</em> pending, you undoubtedly know that it would be tough to make plans for the weekend. With time you’ll get better at this, and work won’t spill into the weekend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vH4oKky-JtED0rRrc2l1Rg.png" /><figcaption>For this particular week I had to cover up on Saturday.</figcaption></figure><h4>Even after the weekend, what if I’m not at 15 units?</h4><p>We count each week from Monday morning to Sunday night. Not being able to complete 15 <em>units</em> in this period is something we want to avoid at all costs. Contrary to what you might expect, the pending <em>units</em>, if any, <strong>do not</strong> flow over to the next week. Every week is fresh, you don’t carry last week’s baggage to the next. Similarly, overperforming last week doesn’t mean you get to slack in the next. If you consistently miss your weekly target, then you might want to re-evaluate and tune the target. Personally, if I am unable to get 15 <em>units</em> by the end of Sunday, I add a vacation day in the past week, which is a huge pain given that vacation days are so limited and precious.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_teucxyhWel9euDTkLYOHQ.png" /><figcaption>I was OOO on Friday, therefore the weekly target was 12 instead of the usual 15. By Friday night, I was unable to meet the target. For some reason I couldn’t catch up on the weekend either. This is one of the rare examples where the weekly target remained red.</figcaption></figure><h4>What if some partial time is left, how do I account for that?</h4><p>The granularity of counting is 0.5 <em>units</em>. If you started the timer at 90 minutes and are left with 30 minutes now, it doesn’t count as 0.67 <em>units</em>, it is still 0.5 <em>units</em>. Rather, even if you had just 5 minutes left (i.e. you technically did 0.94 <em>units</em>) it gets counted as 0.5 <em>units</em>. We always round down! This, in my experience, is one of the most powerful things about this system. <strong>It is not tough to work once you establish the flow, it’s tough to fight the inertia when stuck or when starting a new task.</strong> Many times I have finished some task with 15 minutes left on the timer. I’d feel super lazy to pick up the next new task, and if it were up to me, I’d call it a day. Instead, with the system, I have already spent 30 minutes (towards the 45-minute <em>half-unit</em>) and if I don’t continue working, that would go down the drain. I would end up starting the new task, fighting that inertia with my greed, and sooner than you’d expect, I’d be back in a state of flow.</p><h4>What if there is a holiday in the week?</h4><p>Just count the effective working days and multiply with your daily aim. For example, if I took a half-day on Tuesday, and Friday was a national holiday, the effective working days would be 3.5 and the updated weekly target is now 3.5 * 3 = 10.5 <em>units</em>.</p><h4>Customisation</h4><p>As I mentioned before, you can customise various aspects of the system —</p><p><strong><em>Definition of a unit</em></strong><em><br></em>For my version of this system, meetings don’t count towards focus time. This is because it is usually easy to slack during meetings (at least group ones). Moreover, I wanted to incentivise myself to be more effective in communication. If something can be done async over text, assuming it’s not time-sensitive or blocking, why not? This was a priority for me, feel free to include or exclude avenues where you spend time as part of your job to what counts towards a <em>unit</em>.</p><p><strong><em>The weekly target </em></strong><em><br></em>Depending on how much work you usually need to do, or like to do, you can increase or decrease the daily target (&amp; hence the weekly target). The nature of my work changed slightly in the recent past, now I have more meetings and I had to reduce my target <em>units</em> to 2.5 daily (12.5 weekly).</p><p><strong><em>Penalties and Rewards</em></strong><em><br></em>If you think about it carefully, we’ve swapped the abstract goal of “getting my week’s work done” with “giving 15 <em>units</em> of focus time”. Since this is now a concrete number to be achieved, by changing the scoring function you can amplify or kill certain patterns and behaviours. For instance, I used to mindlessly check my phone throughout the day. To curb this, I added a 2-minute penalty for every unnecessary phone grab. Problem solved! Another challenge was that Fridays often turned into slack-fests, with work spilling into the weekend. To combat this, I introduced a 50% penalty on all weekend work. Basically, to make up 1 <em>unit</em> on the weekend, I’d have to slog through 2 <em>units</em>. Needless to say, this extinguished my Friday procrastination.</p><h4>Realisations and Tips</h4><p><strong>Freedom &gt; Constraints </strong><br>Ironically, this system is more freeing than constraining. Need to run errands on Wednesday? No issue! You can make up the lost <em>units</em> earlier in the week. The beauty lies in the <strong>quantification</strong>. No more ambiguity or room for doubt about my work output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n5xwdngFBs5G5D8WqwKBOw.png" /><figcaption>For this week, I had some errands to run on Thursday and Friday. The quantification helped me calculate the exact extra hours I had to put in for Mon-Tue-Wed.</figcaption></figure><p><strong>Identify Overwork &amp; Prevent Burnout</strong><br>There were times when I got super engrossed, pulling 4 or more <em>units</em> a day, pushing into my personal time. The system forced me to acknowledge this and make conscious choices about how I spent my time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ilUsE7kQsBdY2FwM4KFmow.png" /><figcaption>One of the heaviest weeks I have had.</figcaption></figure><p><strong>Re-use the system for your personal goals</strong><br>I recently started tracking personal weekly <em>units</em> (targeting just 2 <em>units</em>). This ensures I consistently spend time towards my personal goals, like reading books or learning an instrument.</p><p><strong>Consistency is key</strong><br>Missing a whole day (3 <em>units</em>) means an extra <em>half-unit</em> or so on every other workday to compensate. That translates to heavy workloads for several days in a row. Trust me, experiencing this makes you appreciate consistency! A <em>half-unit</em> here or there is fine, but missing a couple creates a lot of work pressure for your other days.</p><p><strong>Adjust your daily targets</strong><br>Instead of rigidly planning for 3 <em>units</em> a day, you can adjust based on your calendar. Many people have fewer meetings on Mondays and Fridays — perfect opportunities to knock out extra <em>units</em> and lighten your load on busier days.</p><p><strong>More granular targets</strong><br>Having a very loose mapping of <em>units</em> to the hours of the day also helps. I try to get the first <em>unit</em> by 1 pm, the next by 4 pm and the last by 7 pm. This is not strict in any way and as you can see, the times are also very relaxed. With this, I know if I’m on track for the day, and these loose time targets help with a pseudo sense of success, which is also a great motivator.</p><h3>Conclusion</h3><p>So, what was I doing wrong before employing the system? I was still putting in effort, but I wasn’t consistent at all. I’d only be working when my tasks were relatively easy and well-defined, trying to escape as soon as I hit a dead end. My deadlines were not very strict and I lacked the self-control to keep working when it became tough. I overestimated the amount of work I did and my productivity. The lack of quantification is what led to the feeling of working all the time but not getting stuff done.</p><p>This system started as a tracking mechanism and became one of the biggest motivators. I consistently complete my weekly goals with this system and enjoy my personal time guilt-free. The only thing that this system requires from you is honour and some patience to tune the parameters to your liking. I hope you found these ideas helpful! If you implement this system or develop your variation, please share your experiences in the comments below. Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=694f18e36416" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synchronising Video Streaming]]></title>
            <link>https://levelup.gitconnected.com/tired-of-making-to-do-list-applications-acce875fe617?source=rss-5e941e6d6a6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/acce875fe617</guid>
            <category><![CDATA[synchronization]]></category>
            <category><![CDATA[streaming-video]]></category>
            <category><![CDATA[teleparty]]></category>
            <category><![CDATA[youtube]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Jayant Jain]]></dc:creator>
            <pubDate>Tue, 29 Jun 2021 15:18:08 GMT</pubDate>
            <atom:updated>2021-06-30T15:03:55.380Z</atom:updated>
            <content:encoded><![CDATA[<h4>Something new for your first web development project</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CXvBeDoWtguE01XU31jddA.gif" /><figcaption>Demonstration: Video Synchroniser with four tabs open in Chrome. GIF by Author, Video Source: <a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ&amp;t=60s">Big Buck Bunny</a></figcaption></figure><p>In this article, I’ll discuss how I created my first web application to synchronise video streaming. Services like <a href="https://twoseven.xyz/">twoseven</a> and <a href="https://www.netflixparty.com/">Teleparty</a> enable users to stream movies and tv shows on popular platforms like Netflix while maintaining synchronisation with their friends. Being a regular user, I was highly fascinated and curious about how they worked. I created my version for streaming YouTube videos. I explain my findings on different algorithms, techniques and tricks I used to achieve this synchronised playback for multiple remote users. I also go over the nasty bugs I faced while figuring out my solution. (Here’s the <a href="https://github.com/jayantjain100/video-synchronisation">Git Repo</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*3lb8nG8nhaLAvhQRJdztUw.gif" /><figcaption>Demonstration: Interface and functionality. GIF by Author, Video Source: <a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ&amp;t=60s">Big Buck Bunny</a></figcaption></figure><p>I have a machine learning and competitive programming background. When I started this project, I did not know the difference between a web application’s backend and frontend. If you are new to this, just like I was, I have included <a href="#87ba">some useful links</a> to videos and articles to get you started. Regarding the tech stack, I created the frontend using Vanilla JS and the backend using Flask in Python. The majority of the discussion is independent of the mentioned tech stack. The best way to follow along is to take your time to read about the unfamiliar concepts; feel free to search for yourself or go through the links provided. Each section might not be relevant for you, and it is fine to read out of order or even skip a few. Let’s jump right in! 😃</p><ol><li><a href="#9d7c">Overview</a></li><li><a href="#4267">Communication</a></li><li><a href="#5d58">State Management</a></li><li><a href="#37d9">Clock Synchronisation</a></li><li><a href="#00ec">Asynchronous JS</a></li><li><a href="#b10d">Parallel Computing</a></li><li><a href="#7e88">Interesting Bugs I Found</a></li><li><a href="#f408">Disclaimer</a></li><li><a href="#6c4a">Further</a></li><li><a href="#87ba">Resources</a></li></ol><h3><strong>Overview</strong></h3><p>Whenever a user changes the state — plays, pauses or seeks the video, we want the exact change to happen for everyone connected in real-time. For this, we need someone to coordinate these updates among the users. The server is responsible for this state management and coordination. Whenever a user makes any change, the updated state is sent to the server, which propagates this new state to the rest of the clients, keeping everyone in sync.</p><h3>Communication</h3><p>The client(browser) can inform the server as soon as the user(you) makes any change by making a simple HTTP POST request. However, how the server propagates this updated state to the rest of the users is not trivial. <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Overview">HTTP</a> is a request-response protocol, which means that requests are initiated by the client and responded to by the server. The server cannot send information to a client whenever it wants, out of the box. So how do we solve this?</p><p><strong>Short Polling</strong></p><p>One way to circumvent this limitation is short polling. The idea is that the client will repeatedly make requests to the server at a fixed interval. So, for instance, in our application, the client could ask the server every 2 seconds whether or not anyone else has made a change.</p><p>This approach has two major problems for our use case —</p><ol><li>In the worst case, a change reaches another user 2 seconds late, disrupting the viewing experience.</li><li>It might seem that we can reduce this time period arbitrarily to propagate changes in the network quicker, but there is a limit to which we can increase the polling frequency. The higher the frequency, the more network and computing resources are consumed. Even if the state is unchanged, all clients keep making unnecessary, wasteful requests to the server, which is undesirable.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*lP8L6m6pQMETxiXDczpNdg.gif" /><figcaption>Demonstration: Delayed synchronisation with polling (Polling time period was 5s). GIF by Author, Video Source: <a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ&amp;t=60s">Big Buck Bunny</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*4c1ubV9ayrWWqFoMtm9hNw.gif" /><figcaption>Demonstration: Intermediate states missed with polling (Polling time period was 5s). GIF by Author, Video Source: <a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ&amp;t=60s">Big Buck Bunny</a></figcaption></figure><p>In both demonstrations above, you can see the lag due to polling in action. In the first, there is a significant delay in the state update reaching the other client. In the second demonstration, since the client polls every 5 seconds, it will just get the latest state and will not know if multiple changes happened in between (not a con, just an observation).</p><p><a href="https://en.wikipedia.org/wiki/Push_technology#Long_polling"><strong>Long Polling</strong></a></p><p>The server responds to a client request only when the requested resource has been updated from the last sent version. It keeps the request open and responds as soon as a change is made. This way, the client might have to wait for the server for a long time, hence the name. As soon as the client receives the response, usually, it immediately creates another request and this way, long polling emulates a <a href="https://en.wikipedia.org/wiki/Push_technology">server push</a>.</p><p><a href="https://en.wikipedia.org/wiki/WebSocket"><strong>WebSocket</strong></a></p><p>From the official IETF(Internet Engineering Task Force) RFC(Request for Comments), <a href="https://datatracker.ietf.org/doc/html/rfc6455">RFC 6455</a>,</p><blockquote>The WebSocket Protocol is designed to supersede existing bidirectional communication technologies that use HTTP as a transport layer to benefit from existing infrastructure (proxies, filtering, authentication). Such technologies were implemented as trade-offs between efficiency and reliability because HTTP was not initially meant to be used for bidirectional communication.</blockquote><p>This protocol lets the client-server establish a full-duplex channel, meaning that either can send messages whenever they’d like once the connection is established. The server does not require an open request from the client to send it information, as was the case with long polling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Sj8YTpHGgDI4XyUKdFeAkA.gif" /><figcaption>Demonstration: Real-time changes with WebSockets. GIF by Author, Video Source: <a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ&amp;t=60s">Big Buck Bunny</a></figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Server-sent_events"><strong>SSE (Server-Sent Events)</strong></a></p><p>From Wikipedia,</p><blockquote>Server-Sent Events is a server push technology enabling a client to receive automatic updates from a server via an HTTP connection, and describes how <strong>servers can</strong> <strong>initiate</strong> data transmission towards clients once an initial client connection has been established.</blockquote><p>Support for SSE is built into the latest HTTP version and is much simpler than using WebSockets, a separate dedicated protocol. One catch is that the server cannot send binary data to the client using SSE. Also, note that SSE is just from the server to the client. The client still makes its requests the same way, using plain HTTP.</p><p><strong>Comparison</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*YnmubCy-SbXU2UFmtCSuKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*yh5lymD1Jz6sD7JWcuemFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*OmgG0MWf5yZr7RlT9OWWmQ.png" /><figcaption>Illustration: How Short Polling, Long Polling and WebSockets would work in the same situation. Images by Author</figcaption></figure><p>There are many intricacies and practical aspects that need to be considered while choosing the protocol to use, e.g., the maximum number of connections, scalability etc. Since this is not the aim of the article, I have not discussed these. I have linked a couple of Stack Overflow answers and blog posts, if you are interested. I ended up using the WebSocket Protocol since I found it the simplest to use with real-time synchronisation (using the <a href="https://socket.io/">Socket.IO library</a> for the client and <a href="https://flask-socketio.readthedocs.io/en/latest/">Flask-SocketIO</a> at the server).</p><h3>State Management</h3><p>For now, assume that the clocks of all machines are in sync (<a href="#37d9">more on this later</a>). We need to store two obvious things: the current video timestamp and whether or not the video is playing. Two not so obvious things to store are the global timestamp of when the last change was made and the global timestamp at which the video timestamp was recorded. Both are essential for handling out of order packets and accounting for network lags, respectively.</p><p>For all the examples discussed here, assume that Client1 and Client2 are already in sync before the global timestamp t_0. At t_0, user1 pauses the video (could be any change), and shortly after, at t_1, user2 seeks the video. Since user2’s change happened later in time, that should be the final state, but these messages could reach the server out of order for several reasons. As you can see, if we do not record when the change actually happened at the user’s end, the server won’t ignore stale updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QK3qFgFsKFEbSNMidpzw2Q.png" /><figcaption>Illustration1: Out of order messages. Image by Author</figcaption></figure><p>It is essential to record the global time whenever the current timestamp of the video is noted. For example, consider the case where a user plays the video from an initially synchronised paused state. This message will reach the other users after some delay due to network latency, and they will always be behind the person who made the change. This is illustrated below. Note that there will be no problem if the change is from a running state to a paused state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObSYYq3QU0tFJiacjFOiXw.png" /><figcaption>Illustration2: Need for recording Global timestamps alongside relative video timestamps. Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*98vXElmPQoUP1ulGFBV7GQ.gif" /><figcaption>Demonstration: Effect of network lag when global times are not recorded (introduced artificial lag of 500ms at the server end). GIF by Author, Video Source: <a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ&amp;t=60s">Big Buck Bunny</a></figcaption></figure><p>The issue described above causes the clients to be out of sync. This is different from what we saw in the case of polling. If polling is done with global timestamps, then the synchronisation would be delayed, but when it kicks in, it would be correct.</p><h3>🕑Clock Synchronisation🕔</h3><p>It took me an extremely long while to discover this bug. In all of the state management discussion, we assumed that each system on the network had access to a global time which was the same for everyone. In reality, all time-libraries in any programming language use the system time to calculate the global time. This time is maintained by an actual hardware timer inside your computer system. Of course, there are many reasons for the times on two machines to be out of sync — the clocks may have drifted apart by a couple of seconds over a long time, or someone might have deliberately changed it. The point is that somehow we need to synchronise all clocks before starting; otherwise, we can’t expect the video synchroniser to work at all.</p><p>Since every client talks to the server, it seems like the best way for everyone to synchronise their time is to independently synchronise their global time with the server after establishing their connection.</p><p>By now, you might have realised that there is another fundamental issue. To synchronise with the server, first, you’ll ask its global time. Then, as soon as you get the response, you’ll compare it with your current global time and see the difference, which you’ll subsequently use to correct your time estimates. The issue is that some time passed by — from the point when the server measured its time, to the point where you actually received it and measured yours. This network lag will always be there. Therefore, we can never have a perfect synchronisation unless you physically keep your machine next to the server and set both times (or coordinate using some other pre-synchronised clock). So what’s the best we can do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6B0ia6Dhi7wyYetOEiuQCA.png" /><figcaption>Illustration3 — Clock Synchronisation. Image by Author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*I9Lj-ipSPBTd3jYwuCFStA.png" /><figcaption>Explanation of the Algorithm, <a href="https://drive.google.com/file/d/1jIkYK5FDB5TN1VXIDIGJ94w1VxvVfAKV/view?usp=sharing">click here</a> for a higher resolution. Image by Author.</figcaption></figure><p>This method, to find the time correction, was inspired by the clock synchronisation algorithms described <a href="https://en.wikipedia.org/wiki/Clock_synchronization">here</a>. On testing my approach with the server and client on different machines, I found the correction estimate to be off by less than 100ms. When I ran both the server and client on the same machine, the true correction is supposed to be 0 because they use the same clock. For this case, my estimate came out to be ~2ms.</p><h3>Asynchronous JavaScript</h3><p>Having worked with synchronous code all my college life, my first major conceptual error was treating my JS code to be synchronous. Even after watching videos about callbacks and <a href="https://www.youtube.com/watch?v=DHvZLI7Db8E&amp;ab_channel=WebDevSimplified">promises</a>, I had just copied the “fetch” and “dot then” syntax for making HTTP requests without fully understanding what a promise truly meant in JS. I found the <a href="https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous">MDN documentation</a> extremely helpful in clearing most of my misconceptions and doubts.</p><p>When we make an HTTP request with “fetch” in JS, it immediately returns with a promise. We don’t need to wait for the response to come back; we can continue our own processing. We can tell it what to do with the result, whenever it comes back, by specifying callback functions. Instead, if the fetch call were blocking, besides wasting precious time waiting for the call to return, the web interface would freeze, not register user clicks, etc. Note that the concept of promises and asynchrony is not limited to network calls. I strongly recommend watching <a href="https://www.youtube.com/watch?v=8aGhZQkoFbQ&amp;ab_channel=JSConf">Philip Robert’s crystal clear explanation of the Event Loop in JS</a> before proceeding in this section (you should be able to answer the question below after watching the video).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dd9813f1e59fc3007faead8fc9edcfe/href">https://medium.com/media/1dd9813f1e59fc3007faead8fc9edcfe/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*cSvJYU4FcbxLB5FwVwBo9Q.png" /><figcaption>The output of the snippet. Image by Author.</figcaption></figure><h3><strong>Parallel Computing</strong></h3><p>Similar to the need for JS to be asynchronous, the server can’t run just one process or thread. If that were the case, then it would handle only one client’s request at a time, and other requests coming in would get queued and would have to wait, or worse, get rejected.</p><p>Currently, I use global variables in my python script for state management. However, this is not thread-safe since I could have two threads that might simultaneously overwrite the state, causing unexpected behaviour. Also, if multiple processes are run, each would have its own copy of the state variables. The ideal solution is to use some database management software, which would handle all this for us, but it seemed overkill since I have a very minimal state that is not specific to any user. A simple solution would be to write the state to a file and use a locking mechanism while writing to this file. I believe this would work correctly with multiple threads/processes, though it might not be the most efficient way. As of now, I have used only the inbuilt development server provided by Flask, which runs one process with a single thread. That’s why this major issue has not come back to bite me. I plan to fix this in the <a href="#6c4a">next version of this project</a>, if any.</p><p>Note that the inbuilt development server by default runs with the parameter “host” set to “127.0.0.1”, which means that only you will be able to access the application. From <a href="https://flask.palletsprojects.com/en/1.1.x/quickstart/">Flask’s docs</a>-</p><blockquote>If you run the server you will notice that the server is only accessible from your own computer, not from any other in the network. This is the default because in debugging mode a user of the application can execute arbitrary Python code on your computer.</blockquote><p>Therefore, if you have the debugger disabled, or trust the users on your network, you can make the server publicly available by setting the “host” parameter to “0.0.0.0”.</p><h3>Interesting Bugs I Found 🐛</h3><p>I’ll discuss one implementation-specific(maybe?) issue that I could not solve properly until the end. I am using the HTML5 video player to display the video and let the user play/pause/seek the video. This element has <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#events">events</a> corresponding to these user actions, and I wrote an event handler(a function called whenever the event fires) for each event. Whenever a user makes a change, the corresponding event handler would get fired and inform the server about the changes, informing everyone else connected. Sounds good, right? The issue is that I could not find any reliable way to distinguish between user caused and script caused events. This means that whenever a client received a state update from the server, it would make the changes locally, and this would trigger the event handlers, which would again inform the server about these changes, causing an infinite loop. To add to this, when you try to seek an already playing video, it causes 3 events; first, it pauses, then seeks and then plays, causing 3 requests to go to the server when we needed to send only the last one. The fix I came up with was that once the server receives a request, it ignores requests from all other clients for some time (say 1 second). The expectation is that, within this time frame, the new state would reach everyone; each client would implement the changes and try to send their new state back to the server, which the server would ignore, bringing the redundant propagation to an end. This is not a perfect solution. One major issue here is that, if some actual user makes a genuine change during this interval, that would also get ignored.</p><p>The alternative solution is to create your own controls to play/pause/seek. This will give you better control over when to fire the video-player events, and when to call a state update to the server. In the current approach, both these got entangled, causing several issues. If you were to create your own controls, another interesting thing could be done. Currently, if you play the video at your end, and this status update message reaches your friend’s system 0.5s later, then their video would play from a timestamp 0.5s later than where it was paused (initially synchronised). This jump in time ensured synchrony, but made them miss out on a small part of the video. My friend came up with an interesting solution to this issue. When a user clicks on the play button, instead of instantly playing their video, the message will go to the server, and a global time would be decided for everyone to start playing at. This global time would be delayed enough for everyone to receive the server’s message, and be ready, but hopefully, small enough not to bug the user that actually played the video 😅.</p><h3>Disclaimer</h3><p>So…this project uses the python library <a href="https://pypi.org/project/pafy/">Pafy</a>, which converts a youtube URL to a direct streamable link using the <a href="https://github.com/ytdl-org/youtube-dl">youtube-dl</a> library in the background. Using the application to steam media might violate Youtube’s terms of service, and downloading copyrighted content from youtube using this library or my web application is definitely illegal ❌. This project is meant for learning 📚 , and I would greatly appreciate it if no one misused it. Thanks!</p><h3><strong>Further</strong></h3><p>I’ll mention a couple of things that I have in mind for the next version of this project. As of now, the web app does not handle the case when a user’s video starts buffering. Should the video pause for everyone, or should the user whose video buffered be skipped ahead to the point where everyone else is? After a state update, changes from other users are ignored for a short period of time, which can be avoided by <a href="#3c2f">implementing my own controls</a>. Support for uploading personal videos could be added. Functionality for independent rooms could be added so that distinct groups of people could use the service simultaneously, without disturbing each other. Similar to twoseven, a chatbox and functionality to connect video feed from each user’s webcam could be added. Instead of having a separate application, achieving this synchronisation functionality through a chrome plug-in similar to Teleparty would be something to explore. The next steps definitely involve deploying the web application to a cloud platform like AWS, and shifting from a development server to a production one. We will have to use a suitable database management system to make our state management thread-safe and process-safe while maintaining efficiency. Further, I’d like to investigate and write an article covering how Teleparty and twoseven work, and possibly other techniques to achieve the same feat.</p><h3><strong>Resources</strong></h3><ol><li><a href="https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/How_the_Web_works">How the web works — MDN</a></li><li><a href="https://www.youtube.com/watch?v=eCUeoPp-5do">Frontend and Backend Explained</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/JavaScript_basics">Introduction to Javascript — MDN</a> (Also see the Intro to HTML and CSS)</li><li><a href="https://www.youtube.com/playlist?list=PLzMcBGfZo4-n4vJJybUVV3Un_NFS5EOgX">Building Web Applications in Python using Flask — Youtube Playlist</a></li><li><a href="https://stackoverflow.com/a/28197906">Overview of Short Polling vs Long Polling vs WebSockets — Stack Overflow</a></li><li><a href="https://ably.com/blog/websockets-vs-long-polling">In-Depth WebSockets vs Long Polling — Blog Post</a></li><li><a href="https://www.youtube.com/watch?v=71hId_-Iwqc">What are Server-Sent Events — Conference talk</a></li><li><a href="https://www.youtube.com/watch?v=8aGhZQkoFbQ&amp;ab_channel=JSConf">Crystal clear explanation of the Event Loop in JS</a></li><li><a href="http://latentflip.com/loupe/?code=JC5vbignYnV0dG9uJywgJ2NsaWNrJywgZnVuY3Rpb24gb25DbGljaygpIHsKICAgIHNldFRpbWVvdXQoZnVuY3Rpb24gdGltZXIoKSB7CiAgICAgICAgY29uc29sZS5sb2coJ1lvdSBjbGlja2VkIHRoZSBidXR0b24hJyk7ICAgIAogICAgfSwgMjAwMCk7Cn0pOwoKY29uc29sZS5sb2coIkhpISIpOwoKc2V0VGltZW91dChmdW5jdGlvbiB0aW1lb3V0KCkgewogICAgY29uc29sZS5sb2coIkNsaWNrIHRoZSBidXR0b24hIik7Cn0sIDUwMDApOwoKY29uc29sZS5sb2coIldlbGNvbWUgdG8gbG91cGUuIik7!!!PGJ1dHRvbj5DbGljayBtZSE8L2J1dHRvbj4%3D">“Loupe” tool for observing the call stack</a></li><li><a href="https://stackoverflow.com/a/32825482">Why global variables are not thread-safe in Flask</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#events">HTML5 Video player Events</a></li><li><a href="https://github.com/jayantjain100/video-synchronisation"><strong>GitHub Repository</strong></a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acce875fe617" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/tired-of-making-to-do-list-applications-acce875fe617">Synchronising Video Streaming</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Augmented Reality DIY]]></title>
            <link>https://medium.com/swlh/augmented-reality-diy-3fc138274561?source=rss-5e941e6d6a6e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fc138274561</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[opencv-python]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Jayant Jain]]></dc:creator>
            <pubDate>Tue, 17 Dec 2019 21:05:27 GMT</pubDate>
            <atom:updated>2021-06-24T15:16:44.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3y2_VTQ_9qlUw7AK" /><figcaption>Photo by <a href="https://unsplash.com/@davidgrdm?utm_source=medium&amp;utm_medium=referral">David Grandmougin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Hi!! My name is Jayant, and what follows is a guide to make your first augmented reality program using Python and OpenCV. This project was an assignment I had in my Undergrad Computer Vision course, which I have broken down here, step by step.</p><p>After following this tutorial, hopefully, you’ll have something that looks like this or even better.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FWJLHM9LYNWs%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWJLHM9LYNWs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FWJLHM9LYNWs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5387fddcb1c32ff09cecfd4ae45c63f/href">https://medium.com/media/c5387fddcb1c32ff09cecfd4ae45c63f/href</a></iframe><p><em>Disclaimer: This will require a basic knowledge of python and OpenCV, so if you haven’t used OpenCV before I would recommend that you go through some introductory tutorial first. I have inserted code snippets for a better understanding of whats going on. It is recommended to go through the full blog first, understand everything and then try writing yourself with my </em><a href="https://github.com/jayantjain100/Augmented-Reality"><em>code</em></a><em> as a reference.</em></p><p>In this post, I shall cover-</p><ol><li>Aruco Markers &amp; Detection</li><li>Perspective Transformations</li><li>Augmenting the Object into our Reality</li><li>Possible Improvements</li></ol><h4>Aruco Markers</h4><p>As you can see in the video, the cut-out in my hand with small black and white squares is an Aruco Marker. This will be my target surface. By positioning the marker, I can control the location and movement of the fox.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*kkFsYp7ZI46w1AnacTYahg.png" /><figcaption>various aruco markers, source: <a href="https://mecaruco2.readthedocs.io/en/latest/notebooks_rst/Aruco/aruco_basics.html">https://mecaruco2.readthedocs.io/en/latest/notebooks_rst/Aruco/aruco_basics.html</a></figcaption></figure><p>If you look at one marker carefully, you can see that there are 64 subsquares (8 by 8) and are colored either black or white. The pattern is not entirely random. The design tries to ensure that a marker won’t be mistaken for some background element and vice-versa. For the same reason, we wouldn’t use a simple checkerboard as an aruco marker. I chose to use Aruco Markers as my target surface because their simple black and white pattern makes them super easy to detect and identify.</p><p>So given the original aruco pattern, how do I find it in my image?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wdKJLsu9eIiR2y-ze2IZiw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e6f51e40b3d5ebddea54cb738927482/href">https://medium.com/media/6e6f51e40b3d5ebddea54cb738927482/href</a></iframe><p>We find all contours in the original image (A contour is simply the boundary of an object in an image). Here I directly used the function cv2.findContours, but at the back end, it may be doing something as simple as finding the edges and then trying to connect them and get closed figures.</p><p>The cv2.approxPolyDP() function then uses the <a href="https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm">Douglas Peucker Algorithm</a> to “smoothen” the contours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*9WqeOouuqwufEdIR0ZpQiA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*o67l6fGtTCdJl_xB3H7Pzw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*hdAf-8-wJH6EGWgPEAimig.jpeg" /><figcaption>Different levels of smoothening</figcaption></figure><p>Note that even though our original pattern has a square boundary, the image of this marker may not look square. This is because of the perspective transformation that it undergoes. We’ll dive deeper into this ahead. An easy example would be to think of railway tracks. We know that they are parallel, but sometimes when you look at them along their direction, they seem to incline towards each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*ti6J5nPQOuCO7eA5K2YDgg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*ypRAKACu2-5diJtwLBeJhA.png" /><figcaption>source: <a href="https://stackoverflow.com/questions/12919398/perspective-transform-of-svg-paths-four-corner-distort">https://stackoverflow.com/questions/12919398/perspective-transform-of-svg-paths-four-corner-distort</a></figcaption></figure><p>Since we are looking for a Quadrilateral, we disregard all found contours with more than 4 points and continue our search. For the filtered out quadrilaterals, I find the bit-signatures by dividing each quadrilateral into an 8 x 8 grid and threshold the pixel value at the center.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3a20d3b54f746f84ca20f462eed5302/href">https://medium.com/media/f3a20d3b54f746f84ca20f462eed5302/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*LYbUPobih7XAp0dEaBNHJw.jpeg" /><figcaption>points of interest for a general convex quadrilateral</figcaption></figure><p>This bit-signature found can then be matched with the bit-signature of the original pattern. All four rotations of the original are considered, and the corresponding match gives information about the order of points, which is crucial in homography estimation, which will come ahead.</p><p>Yay!! We found our aruco marker in the image.</p><h4>Homography &amp; Perspective Transform</h4><p>I want to answer questions like if I had a real fox and I kept it on top of my marker, then what would the captured image look like?</p><p>So, given the unchanged image of me with the marker in my hand and the true Aruco pattern for reference, I wish to augment the fox into my reality by making edits to the image I have.</p><p>Before we do that, we need to learn about different coordinate systems and how they are related to one another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*uTDsperqa_n4nTHgMLTUyA.png" /></figure><p>The World Coordinate System(WCS) is the 3D Cartesian Coordinate system in which we describe our object of interest. It can be arbitrarily kept according to our convenience. The Camera Coordinate system(CCS) is also a 3D Cartesian Coordinate system which has its axes according to the camera&#39;s orientation.</p><p>A translation and rotation are enough to transform world coordinates into camera coordinates. The jump from world coordinates to camera coordinates is governed by the following equation-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*Pq2ZzDX_PSF3kuqhCh9x2w.png" /></figure><p>Next, we have the intrinsic matrix (A), which is a projection from the 3d camera coordinates(X, Y, Z) to 2d pixel coordinates(u,v). It’s enough to know that the intrinsic matrix is dependent purely on the camera, not on its orientation or other stuff. It takes care of the focal length of the camera, aspect ratio, etc. For now, say we have the intrinsic matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*M1KJZK4LlzXf4qYGI9rMwA.png" /><figcaption>The equation for a perspective transformation, from WCS to Pixel Coordinates</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XnTDmgs8rD9IYc-3kAUvog.png" /><figcaption>Point P has World Coordinates (U1, V1, 0) and Pixel Coordinates (u1, v1)</figcaption></figure><p>Our aim is to find the parameters of this perspective equation so that we can place our object in the WCS and find its pixel coordinates, which will give us our desired image.</p><p>Till now, we’ve located our Aruco Marker in the image. Using its corners, we have four correspondences <em>(a pair (P, P*) where P is in the WCS and P* is the corresponding point in the Pixel Coordinates)</em><strong><em>. </em></strong>If we try to solve for the parameters, 4 points suffice to find the homography.</p><p>Now, what is a Homography? A Homography is a degenerate form of the perspective equation. Using the homography H, we can transform points belonging to a specific planar surface; in this case, the plane W = 0. Note that in our perspective equation for all 4 points on the marker, the z component(W) is 0, and therefore we can’t get any information about r3. But we do know that r3 should be the cross product of r1 and r2(orthogonal axes). Using this, we find r3 in the extended_RT function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f805c4e308f968f85e7cb75cff2df03/href">https://medium.com/media/3f805c4e308f968f85e7cb75cff2df03/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*9N0aCoEa301tApc8sCbjsQ.png" /></figure><p>Coming back to the intrinsic matrix, OpenCV has <a href="https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_calib3d/py_calibration/py_calibration.html">direct functions</a> to help you calibrate your web camera. You just need to follow their documentation. I will just briefly give you an idea of what you’ll have to do.</p><p>So it describes various types of distortions in images and gives the correction equations which have certain parameters.</p><blockquote>To find all these parameters, what we have to do is to provide some sample images of a well defined pattern (eg, chess board). We find some specific points in it ( square corners in chess board). We know its coordinates in real world space and we know its coordinates in image. With these data, some mathematical problem is solved in background to get the distortion coefficients.</blockquote><p>Given these sample images, it finds the distortion coefficient along with the intrinsic matrix(our aim).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/defeb1671559e2812412ce3871d980ca/href">https://medium.com/media/defeb1671559e2812412ce3871d980ca/href</a></iframe><p>The problem here is that you have to take a lot of photos of a <a href="https://github.com/opencv/opencv/blob/master/doc/pattern.png">checkerboard</a>.</p><p>If you are lazy, then there is another way. You can try to approximate it smartly. The matrix has the form-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*F3tgZ4DrT04rWeyhV-fu-A.png" /></figure><p>c<em>x</em> and c<em>y</em> are supposed to be half the resolution of the webcam. So for me, the resolution was 1280x640, which gives c<em>x</em>= 640 and c<em>y</em>= 320. The focal length can be approximated following this <a href="https://www.learnopencv.com/approximate-focal-length-for-webcams-and-cell-phone-cameras/">post</a>. The exact value of f is not crucial to this application and therefore a smart guess could work.</p><p>So, after all this hassle we have the perspective transformation matrix.</p><p>But before transforming points to the pixel coordinate systems, we first have to decide the world coordinates of the animated fox.</p><h4>Rendering</h4><p>I got the fox from the website <a href="https://clara.io/library">clara.io</a>, which has a wide range of 3D objects that you can choose from. We will be dealing with the <a href="http://www.cs.cmu.edu/~mbz/personal/graphics/obj.html">.obj</a> format. It is a very simple way to describe a three-dimensional object. All we need to know is that it lists all the vertex coordinates, texture coordinates, and face information.</p><p>What I’ve done is not the best(or even correct) way to render this image. There are many things that need to be taken into consideration, but I took some shortcuts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be8ee82f9146cccb97371a438da2bb7c/href">https://medium.com/media/be8ee82f9146cccb97371a438da2bb7c/href</a></iframe><p>The basic idea is that we find world coordinates for all the vertices. For that, we just have to shift the object to the center of the aruco marker, by shifting each vertex. Project all the vertices to their pixel coordinates and then draw all the 2d faces on top of the original image.</p><p>For color, there was an accompanying texture file and <a href="https://discoverthreejs.com/book/first-steps/textures-intro/#uv-mapping-explained">texture coordinates</a> for each face. In the image, you can see each triangular face mapped to the corresponding texture. I have directly picked up the color (by using pixel intensity at the centroid of the triangle). Since, while rendering no ordering of faces is kept in mind, the fox appears transparent. No consideration of lighting has been taken either. Maybe all of this can be solved using some library, but for now, this is what we have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8MOUCEMA80ndn6dqZCnDg.png" /></figure><p><em>Tip:</em> <em>When dealing with images in OpenCV in python, please take care of pixel coordinates and locations in the image. What I mean to say is that intensity at point(x, y) in the image is img[y][x] and not img[x][y]. Such bugs waste a lot of time and are extremely frustrating.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/1*2MrFUkoYU4fZech4a6NiZg.png" /></figure><p><em>For example, in this 100x100 image, a circle was drawn at (20,50) but the pixel value at image[20][50] is black and image[50][20] is red</em></p><h4>Summary</h4><p>This is how it will all come together-</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bdad22090d538b7f425f588c9986583/href">https://medium.com/media/4bdad22090d538b7f425f588c9986583/href</a></iframe><p>Before running the main file we have hardcoded the intrinsic matrix which was found either by calibration or smart guessing.</p><p>At each time step, we fetch a frame from the webcam. We try to find the aruco marker in the frame by finding all contours, filtering out the quadrilaterals and then matching their bit-signatures. If we find the aruco marker in our frame, we use its corners to estimate a homography from the world coordinates to the pixel coordinates. This homography is extended to get the full perspective matrix. We shift our object vertices to the center of the marker and using the perspective transform, find its 2D projection in our image.</p><h4>Further</h4><p>One major improvement, that I’m working on currently would be not to detect the marker each time, but rather track it. What I mean to say that If I know that the marker was at some position currently, then I can exploit this information when I’m looking for it at the next time step, instead of searching for it all over again from scratch. This is precisely what tracking is. This will address the instability in the video that we have right now and since it will take less processing the simulation would be more responsive. Using the Lucas-Kande algorithm for Sparse Optical Flow should solve our problem, and I plan to cover that in the next part.</p><p>Some other issues that I’d like to solve are to have a more realistic rendering.</p><p>Another way to create this augmented reality program would be to use any marker (not Aruco, something random like your name written on a piece of paper) and use SIFT feature descriptors. An advantage would be that it would work even if some part of the marker is occluded. The Aruco method fails if a part is outside because it’s impossible to detect the contour though I found that method slower and more unstable.</p><p>I would like to thank Juan Gallostra Acín, whose <a href="https://bitesofcode.wordpress.com/2017/09/12/augmented-reality-with-python-and-opencv-part-1/">blog </a>really helped me get started with this project and helped me write my first version. He has used SIFT descriptors for feature matching and homography estimation. You should check out his blog too.</p><p>I would also like to thank my project partner Manuj Trehan without whom I couldn’t have completed this project.</p><h4>Links</h4><ol><li><a href="https://bitesofcode.wordpress.com/2017/09/12/augmented-reality-with-python-and-opencv-part-1/">Juan Gallostra Acín’s blog</a></li><li><a href="https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm">Douglas Peucker Algorithm</a></li><li><a href="http://www.cs.cmu.edu/~mbz/personal/graphics/obj.html">.obj file description</a></li><li><a href="https://clara.io/view/1a03ac6b-d6b5-4c2d-9f1a-c80068311396">3D fox used</a></li><li><a href="https://github.com/jayantjain100/Augmented-Reality"><strong>GitHub repository</strong></a></li><li><a href="https://discoverthreejs.com/book/first-steps/textures-intro/#uv-mapping-explained">UV Texture Mapping</a></li><li><a href="https://github.com/opencv/opencv/blob/master/doc/pattern.png">Original Image of Checkerboard</a></li><li><a href="https://www.learnopencv.com/approximate-focal-length-for-webcams-and-cell-phone-cameras/">Approximating the focal length of a camera</a></li></ol><p><em>PS: This is the first time I have made a tutorial. If you found any part unclear or incorrect, feel free to leave a comment or write to me at jayantjain100@gmail.com so that I can improve these posts. Thank you.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fc138274561" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/augmented-reality-diy-3fc138274561">Augmented Reality DIY</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>