<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Faraz khan on Medium]]></title>
        <description><![CDATA[Stories by Faraz khan on Medium]]></description>
        <link>https://medium.com/@farazrkhan?source=rss-68c8f91bfc0e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TMVYPQ3JGwbS66Zt8bYcsQ.jpeg</url>
            <title>Stories by Faraz khan on Medium</title>
            <link>https://medium.com/@farazrkhan?source=rss-68c8f91bfc0e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 03:26:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@farazrkhan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A complete drive by wire implementation using OSCC on the Kia Soul EV]]></title>
            <link>https://medium.com/opencaret/a-complete-drive-by-wire-implementation-using-oscc-on-the-kia-soul-ev-55664e9e10fc?source=rss-68c8f91bfc0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/55664e9e10fc</guid>
            <category><![CDATA[autonomous-vehicles]]></category>
            <category><![CDATA[arduino]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Faraz khan]]></dc:creator>
            <pubDate>Tue, 05 Jun 2018 21:42:59 GMT</pubDate>
            <atom:updated>2018-10-22T17:41:17.286Z</atom:updated>
            <content:encoded><![CDATA[<p>This was step 1 to building my highway autopilot system. You can find more information about the entire project <a href="https://frk2.github.io/opencaret/">here</a></p><p>A month ago I started down this road of implementing polysync’s <a href="https://github.com/PolySync/oscc">OSCC</a> for myself in order to get drive by wire control of my car. I had the option of buying Polysync’s drivekit— but then I realized that since my main aim is to learn here my time would be better spent trying to install their opensource car control setup on my own.</p><p>The polysync folks have done some kick ass work and opensourced not only the software but even the hardware schematics and board designs! This helps folks like me a TON!</p><p>While the hardware and software part as it relates to the boards is 100% complete — there’s some detail missing on how to actually get this to work in a car. This blog and my wiki will document this process so anyone can get started. The cost of this kit ends up being less than $500 with all parts and shipping considered. The only permanent modification you have to make is with the steering torque sensor connector.</p><blockquote>NOTE: This is not a replacement for Polysync’s drivekit. The drivekit is a hardened high quality kit using proper wiring and harnesses. If you are a startup please support them by buying their kit and save yourself a lot of time :) Its the cheapest DBW Kit on the planet. If you are tinkerer like me read on!</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_MvwWrbWzgU%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_MvwWrbWzgU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_MvwWrbWzgU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4fad2bc3e4077e9a7dee44eb7480ed9/href">https://medium.com/media/d4fad2bc3e4077e9a7dee44eb7480ed9/href</a></iframe><p>This is a quick overview of what I did. If you want actionable detail head over to the <a href="https://github.com/frk2/opencaret/wiki/Drive-by-wire-Installation">project wiki</a> which is in the process of being populated.</p><h3>Hardware</h3><p>Things you’ll need:</p><ul><li>OSCC Boards</li><li>Accelerator, Brake and Steering connectors / mating connectors</li><li>CAN to USB adapter like <a href="https://www.carloop.io/">Carloop</a> or <a href="http://canable.io/">Canable</a></li><li>OBD-2 Connector to tap for power and car can</li><li>Power connectors and signal (4-way) connectors</li><li>Xbox controller</li><li>A ODB2 “professional” code reader which can reset your brake/accel/steering faults. I used <a href="https://www.amazon.com/gp/product/B00UJV3EA8/ref=oh_aui_detailpage_o03_s00?ie=UTF8&amp;psc=1">this one</a> — its the cheapest that actually works even though the car isn’t strictly supported.</li></ul><h3>Putting the boards together</h3><p>If you take a look in the <a href="https://github.com/PolySync/oscc/tree/master/hardware/boards">boards</a> directory in the repo you’ll find a number of eagle board files and schematics. Explained as follows:</p><ul><li><strong>gateway</strong>: You need 1 of these, this interfaces primarily with the vehicle CAN bus to get you stuff like vehicle speed, steering angle, brake pressure, etc and drives a debug LCD display. I was lazy so I didn’t bother hooking this up till the end.</li><li><strong>sensor_interface</strong>: This is the analog sensor spoofing module and you need 2 of these (1 for accelerator, 1 for steering).</li><li><strong>vehicle_control_module:</strong> This is designed to handle brakes. Its basically the same thing as the sensor_interface except it also drives 1 additional relay (for brake lights)</li></ul><p>Now to actually getting the boards made: just use <a href="https://macrofab.com/">Macrofab</a>! They are a bit pricey ($50–60 per board) but they will produce extremely high quality boards, solder all the minuscule SMD components on there and test for weird connectivity issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K8DzB3CMlmLUOdChkBINKg.jpeg" /><figcaption>Boards in their android sheild enclosures (from amazon)</figcaption></figure><h3>BOM (Parts)</h3><p>Macrofab automatically extracts schematics and device info from the board files but you still have to search for the correct part number which was a bit frustrating. You can use <a href="https://docs.google.com/spreadsheets/d/17XYKf66g3194c76FXC2jgOLxLVx6NxTcLJDDG6sa3j4/edit#gid=120139621">this BOM</a> to save you time if you are ordering from macrofab.</p><p>I had to order SOME elements directly from mouser since macrofab either didn’t have them OR it was adding significantly to the timeline. Its upto you how you wish to do this. Soldering all but the SMD components by hand isn’t that hard.</p><h3>Controlling with my XBOX controller</h3><p>I used a canable USB to CAN convertor to get my laptop to talk can to the boards. OSCC assumes that you have a separate can bus for just the boards. Luckily due to how simple CAN is, this basically means connecting all the cans together :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Vb2HrKOHYKF2M8zlesgtg.jpeg" /></figure><p>Hope you like my fluorescent lime e-stop canister! If you have a crappy 3-d printer like I do, let me know and i’ll send you a modified STL file without the extrusions on the top — my printer doesn’t like those at all.</p><h3>Whats next?</h3><p>Now that theres a car that’s drive by wire capable I’m going to jump on the software/sensor side. Next step sensor wise is to get them installed:</p><ul><li>Camera: Mid/Rearview mirror mounted ZED Stereo camera</li><li>Radar: Get a nice mount made for the toyota radar, maybe even mount it inside the bumper on the chassis if possible.</li></ul><p>I’m not going to bother with an actual compute/CPU on the car just yet — instead try to iterate quickly simply using my laptop. Things that need to be decided:</p><ul><li>Whether to use ROS/ROS2 or go with a barebones ZeroMQ approach. I’m leaning towards ROS/ROS2</li><li>Whether we even need a stereo RGBD Camera.</li></ul><h3>Join In and test stuff out on this car!</h3><p>The purpose of this project is to create a community that can churn out amazing autopilot software. Whether you want to do this for yourself or simply join in to test stuff out on the car you are welcome to do so! <a href="https://join.slack.com/t/opencaret/shared_invite/enQtMzU1OTQ2NjY0MTgyLTlhY2JmYTlkYTg4ZGIyNDYzMjFhNWMxNjlmZGFiYTI1MTIyZjA0ODNhMzI5ZDUxYTQ3MWFjYWQ0YzQxZGE1ZjA">Join our Slack</a> channel and let me know what you have in mind!</p><p>If you want to build this yourself <a href="https://github.com/frk2/opencaret/wiki">check out the wiki</a> for details for info on the build process.</p><p>This is all based out of my house but soon moving over to <a href="http://www.circuitlaunch.com/">Circuitlaunch</a> in Oakland starting July so we’ll have a place to hang out and test stuff. <a href="https://www.google.com/maps/place/Alameda+Point,+Alameda,+CA+94501/@37.7899242,-122.3340486,14.53z/data=!4m5!3m4!1s0x808f80671978e30f:0x7de780340ce11b82!8m2!3d37.7860582!4d-122.3181623">Alameda Point</a> next door is a great safe public place to test this stuff out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qN8MVnsMlkadECEJ-RJl0w.jpeg" /><figcaption>Testing the DBW at alameda point. Lighter was used to make heat shrink tubing in car!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55664e9e10fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opencaret/a-complete-drive-by-wire-implementation-using-oscc-on-the-kia-soul-ev-55664e9e10fc">A complete drive by wire implementation using OSCC on the Kia Soul EV</a> was originally published in <a href="https://medium.com/opencaret">Opencaret</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working throttle and brake!]]></title>
            <link>https://medium.com/opencaret/working-throttle-and-brake-ee2914ac2393?source=rss-68c8f91bfc0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee2914ac2393</guid>
            <category><![CDATA[self-driving-cars]]></category>
            <dc:creator><![CDATA[Faraz khan]]></dc:creator>
            <pubDate>Tue, 29 May 2018 18:17:55 GMT</pubDate>
            <atom:updated>2018-05-29T18:17:55.151Z</atom:updated>
            <content:encoded><![CDATA[<p>I was able to order mating connectors from ballenger motorsports and design a pretty decent Y connector to plug into my throttle position sensor, brake position sensor and the ECU. It ACTUALLY WORKS! You have no idea how good it feels to be able to see something come to life like this!</p><p>The only thing left now is steering. For that I need to cut open the stock wiring :\</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3U3ohWdvlbkrYvcPslGwyA.jpeg" /><figcaption>My Y connector ends. One end plugs into the APS, the other into the stock connector</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4epJpP4IhxdXaTAyPuQ6w.jpeg" /><figcaption>Pretty proud of my wiring work here</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee2914ac2393" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opencaret/working-throttle-and-brake-ee2914ac2393">Working throttle and brake!</a> was originally published in <a href="https://medium.com/opencaret">Opencaret</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OSCC Boards are here!]]></title>
            <link>https://medium.com/opencaret/oscc-boards-are-here-48156fcbb746?source=rss-68c8f91bfc0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/48156fcbb746</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <dc:creator><![CDATA[Faraz khan]]></dc:creator>
            <pubDate>Tue, 29 May 2018 17:42:12 GMT</pubDate>
            <atom:updated>2018-05-29T17:42:12.194Z</atom:updated>
            <content:encoded><![CDATA[<p>Thank you, Polysync!</p><p>For making these opensource/openhardware. Huge step for people like us trying to hack into their cars. Now to install all the components macrofab won’t do and get these ready for action!</p><p>For more info check out: <a href="https://github.com/PolySync/oscc">https://github.com/PolySync/oscc</a></p><p>Now I need to find out how to tap all the gas/brake and steering sensors on my car. With that done it should be as easy as commanding the gateway to go :)</p><p>Also need to find a large parking lot somewhere!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UaYPF6js6eUYp6qpPOiXVQ.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48156fcbb746" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opencaret/oscc-boards-are-here-48156fcbb746">OSCC Boards are here!</a> was originally published in <a href="https://medium.com/opencaret">Opencaret</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Working Toyota Radar]]></title>
            <link>https://medium.com/opencaret/working-toyota-radar-784a1f8cc16a?source=rss-68c8f91bfc0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/784a1f8cc16a</guid>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[radar]]></category>
            <dc:creator><![CDATA[Faraz khan]]></dc:creator>
            <pubDate>Tue, 29 May 2018 17:11:18 GMT</pubDate>
            <atom:updated>2018-05-29T17:11:18.614Z</atom:updated>
            <content:encoded><![CDATA[<p>Since im waiting on the macrofab order for Polysync OSCC to come through I thought I’ll try and get the radar to actually spit out real information. I thought this would be as easy as connecting the radar and deciphering its CAN messages.</p><p>Ofcourse it was way harder than that! Turns out the radar has to be turned ON by a sequence of CAN bus messages from the DSU. Comma.ai’s openpilot handles this situation so I started going through all there code and sending all possible messages I could think of. In the end I gave up and realized I can simply sniff out all the messages that openpilot sends on the CAN bus to the radar! This actually worked! I was able to separate out the messages openpilot was sending and upon sending those from my python script the radar came to life!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*irCiJhHAKSwUOtLYyHi9sw.jpeg" /></figure><p>If you want to do this yourself- take a look at my <a href="https://github.com/frk2/toyoyta_radar_control_can">Toyota Radar Can control project</a></p><p>Now to actually mount this thing on the car 💭</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=784a1f8cc16a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opencaret/working-toyota-radar-784a1f8cc16a">Working Toyota Radar</a> was originally published in <a href="https://medium.com/opencaret">Opencaret</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepCat: A CNN based status light detector for the Litter Robot III]]></title>
            <link>https://medium.com/@farazrkhan/deepcat-a-cnn-based-status-light-detector-for-the-litter-robot-iii-6b4746d92785?source=rss-68c8f91bfc0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b4746d92785</guid>
            <category><![CDATA[cats]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[poop]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Faraz khan]]></dc:creator>
            <pubDate>Fri, 04 Aug 2017 03:55:23 GMT</pubDate>
            <atom:updated>2017-08-04T04:50:44.255Z</atom:updated>
            <content:encoded><![CDATA[<p>I have a <a href="https://www.litter-robot.com/">litter robot</a> which is shoved into a far corner of the house which only Mr. Cat frequents. So much to the cat’s disgust, it ends up blinking blue (which signifies full litter bin) for many days without nobody noticing. The idea was to build something that can detect blinking blue (among other) status lights in a non-intrusive way.</p><h4>Cool so whats the ‘NN’ part?</h4><p>I understand that there are much easier ways of solving this problem:</p><ul><li>I could have opened the litter robot control panel and tapped into the LEDs and fed them to a photon/raspberry pi.</li><li>I could have mounted a raspberry pi with a camera pointed in a known way and then used openCV to look for the blue/red/yellow or the absence thereof.</li></ul><p>OR</p><p>I can use a Neural net to do all of this. This would be less intrusive than tapping into LEDs and (hopefully) generalize better than the openCV approach so I won’t be extremely limited in where I place the camera.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MGxj8zqhbFqx-8PrQDVRpA.jpeg" /><figcaption>Said cat aka ‘Kitty’ wearing a bandana cuz why not?</figcaption></figure><p>In the end I have a classifier uses a fine-tuned SqueezeNet that does _fairly_ well on images of varying brightness and registers ~ 95% test accuracy on images taken from different cameras *if* I stay within some defined boundaries (particular range of distance away, all 3 lights showing in the picture, etc). In here I discuss what I tried and what went wrong and hope it helps someone :) Code’s at: <a href="https://github.com/frk2/DeepCat">https://github.com/frk2/DeepCat</a></p><p>Please feel free to reach me out on <a href="https://twitter.com/faraz_r_khan">twitter</a> if you have a suggestion on how I can improve this or if you feel Im doing something completely wrong. I’m a noob when it comes to neural nets so any help/advice is welcome!</p><h4>Data Collection</h4><p>This was by far the hardest part and the one I’m going to go into most depth into as everything else is pretty standard as far as neural nets are concerned. What I have available as train_data_orig.tar in the repository. I collected by recording a video from various angles and distances, using my phone camera and then the raspberry pi camera. I moved the cameras around to generate offsets in the image as a method to increasing the number of data points. Images like the following were generated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pIBfHn7gBxaa4buZSzDPHQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_jDCxV5nvVMBIUjb8CWTCg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QCaRY9Q3gS44F0dT4QLkUg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CKMMXgA6QbIc637WX3H34w.jpeg" /></figure><p>So 4 classes, ~ 2500 images per class. A decent amount of data augmentation was done using the keras ImageDataGenerator which happily adds shear, zoom, translate and rotations to your images. Very handy!</p><p>It was extremely easy to overfit the data — probably due to the fact that there was very little unique data I was feeding into the model. Videos produced a lot of non unique frames. If I collect images from far away a lot of room elements get included and it seemed like it was harder for the model to converge and generalize. By trial and error over a few days, I found that images which capture all the lights in the center seemed to work the best.</p><h4>The Model</h4><p>I started out by training my own based on <a href="https://devblogs.nvidia.com/parallelforall/deep-learning-self-driving-cars/">NVidia’s end to end deep learning </a>paper but it was really hard to make it generalize with the small amount of unique data that I had. Things changed drastically when I started using a Keras Squeezenetport which was pretrained on Imagenet. Thanks to this guy <a href="https://github.com/rcmalli/keras-squeezenet">https://github.com/rcmalli/keras-squeezenet</a>.</p><p>Ive been toying with two different techniques for this. In the first I removed the last softmax layer of the model and added my own 4-class Dense layer:</p><pre>model = SqueezeNet()<br>x = Dense(4, activation=’softmax’)(model.layers[-2].output)<br>model = Model(model.inputs, x)</pre><p>This gives me ~90% accuracy. In the second I took out the entire convnet classifier from Squeezenet:</p><pre>model = SqueezeNet()<br>x = Convolution2D(4, (1, 1), padding=’same’, name=’conv11&#39;)(model.layers[-5].output)<br>x = Activation(‘relu’, name=’relu_conv10&#39;)(x)<br>x = GlobalAveragePooling2D()(x)<br>x = Activation(‘softmax’)(x)<br>model = Model(model.inputs, x)</pre><p>Give me 95% Test accuracy. Seems to initially work better.</p><p>This was inspired by the fact that Squeezenet has the last convnet generally the size of classes. I’m not sure if this is a sensible approach but seems to work. Does it make sense to use weights of a pretrained model and yank an entire layer this way? Please tell me!</p><h4>The Accuracy Function</h4><p>I dont know if its my Keras version but I always end up gettting an accuracy of 100% using evaluate_generator when in reality my accuracy was near 0! I found out that apparently keras calculates accuracy in mysterious ways so the most reliable way was to do it yourself using the following snippet:</p><pre>def evaluate_model(model, generator):<br> from sklearn.metrics import accuracy_score<br> steps = 0<br> scores = []<br> for i in generator:<br>   y_pred = model.predict(x=i[0])<br>   y_pred = y_pred &gt; 0.6<br>   scores.append(accuracy_score(i[1], y_pred))<br>   steps += 1<br>   if (steps &gt; 29):<br>     break<br> print(“Model score is: {}”.format(sum(scores)/len(scores)))</pre><p>So given a model and a test generator, this guy will calculate your prediction and compare to the truth. The classification is chosen if &gt;60% of the softmax probability is in your favor. Saved my life. Before this everything called accuracy was a lie! (Or I dont understand how keras accuracy works)</p><h4>What worked</h4><ul><li>Using a pre-trained Squeezenet! I realized that folks who participiated in the Nexar challenger for traffic light detection were doing that — and my problem is basically super similar so why not.</li><li>Using your own accuracy function!</li><li>Balancing was important since for some I kept producing way more blue and yellow frames than ‘none’ or red frames. I had to manually get rid of blue/yellow frames to bring it to par. I got rid of ‘bad’ images like ones where one of the lights was cut off etc.</li><li>Checkpointing. It was super crazy easy to start overfitting, sometimes even after the first epoch! Going back to earlier models that showed high test accuracy was always way superior than a later overfitted model.</li></ul><h4>What didn’t work</h4><p>Well a lot of stuff. Here goes:</p><ul><li>Using a different color space: I thought maybe I could extract the S channel from HLS and intuitively that looks like it might work but in reality it makes accuracy worse</li><li>Training your own model. I tried many different models but they all sucked in that they overfit to the extreme.</li><li>Using color balancing to negate the effects of illumination. I tried using equalize_adapthist on the training set and all it did was slow the entire thing down. This was to combat training data gathered in sunlight vs testing data gather later in the day in warm LED light. Is this possible?</li><li>Cropping the image to focus always on the lights. This worked, except it super duper overfit the model to expect images in exactly that distance, captured at exactly that resolution.</li><li>Augmenting yourself. Too painful, just use ImageDataGenerator! Its free!</li></ul><h4>And the rest</h4><p>I deployed everything to my raspberry pi and it was fairly simple to get everything (including tensorflow with keras) going. The raspberry pi basically takes 30 pictures in quick succession and runs a prediction on all of them. So</p><p>blue,blue,blue,none,none,none,blue,blue,blue</p><p>becomes:</p><p>(&#39;blue&#39;,3) , (&#39;none&#39;,3), (&#39;blue&#39;,3)</p><p>That pattern signifies blinking — if true the raspberry pi will go online and tweet to the world about its bin being full. If you are curious/super bored feel free to follow <a href="https://twitter.com/kitty_litterbot">Kitty LitterBot’s twitter account</a> and you’ll be made aware of every single one of kitty’s bowel movements 😆</p><p><a href="https://github.com/frk2/DeepCat">Check it out at Github</a>.</p><p>Its been working pretty well! The model has generalized enough that I can keep the camera this close or even like 3–4 feet away. Works during low light as well — but starts to fail when its darker. Here’s my contraption uglily glued to the litter robot with a flex camera mount thing I got on amazon. The cat doesn’t seem to mind too much (so far!):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wl2FgqrLpbh9rFdfSeTcmw.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b4746d92785" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Calls: Is it you or is it me?]]></title>
            <link>https://medium.com/several-people-are-coding/calls-is-it-you-or-is-it-me-f5d36749e8ed?source=rss-68c8f91bfc0e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5d36749e8ed</guid>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[webrtc]]></category>
            <dc:creator><![CDATA[Faraz khan]]></dc:creator>
            <pubDate>Wed, 11 May 2016 15:01:13 GMT</pubDate>
            <atom:updated>2016-05-11T18:52:48.018Z</atom:updated>
            <content:encoded><![CDATA[<p>Slack Calls are now in beta, on Mac, Windows, iOS, Android and Chrome. If you haven’t given it a try yet, please do (and let us know how it goes)! <a href="https://get.slack.help/hc/en-us/articles/216951758">Our help center article on Calls</a> has more details on the feature.</p><p>We wanted to answer the age-old question that we have all asked each other during a voice call when quality degrades: “Is it my connection or yours?”. We wanted a UI that the user can <em>trust.</em> So we decided to build in the logic to answer that question for you. It’s much better than a generic “Having network issues” warning.</p><p>We utilize a selective forwarding architecture to better support group calls. In this architecture, participants communicate only with the media server and are unaware of the state of each other’s network. In this post, we’ll talk about how we analyze real-time control protocol (RTCP) feedback from each of the users to generate correct feedback to all users about their individual uplink / downlink quality.</p><p>We use <a href="https://github.com/meetecho/janus-gateway">Janus</a> as our selective forwarding unit (SFU) and <a href="http://www.webrtc.org/">WebRTC</a> for voice chat. WebRTC provides us with standardized RTCP feedback but Janus doesn’t analyze it out of the box. This post is about us adding this support. Most code excerpts that you see are from Janus alongside some of our customizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wr4nkQe8rWO3iKyvItTZPw.png" /><figcaption>SFU in operation. Notice that Susy’s voice (in yellow) is broadcasted as-is to Mike and Raja through the server</figcaption></figure><h3>A little about SFUs</h3><p>An SFU’s job is to broadcast audio which it receives on the real-time protocol (RTP) layer from each participant to all other participants (this type of architecture is also known as a router server). This is in contrast to a Multipoint Control Unit, or MCU, which mixes the audio from each participant to maintain a single client/server stream. <a href="https://webrtchacks.com/webrtc-beyond-one-one/">More details on the differences between these approaches here</a>.</p><p>In addition to RTP, there is also the RTCP layer (Real-time Control Protocol), which is used to relay back information about the connection. The topology offers us an advantage: since the server sits in the middle, it can separate out the two legs of the call (upload and download). We can use this to pinpoint a problematic link.</p><h3>How we do it (TL;DR version)</h3><p>We analyze the RTCP Receiver Reports (RR) that all the receivers of a particular stream send back (which denotes <em>total loss</em>) while simultaneously calculating the upload loss of said stream from the sender. Given this, we figure out the <em>realistic loss</em> (loss caused <em>only</em> by the downlink from the server) on each of the downloads. We use ALL the <em>realistic losses </em>gathered by the process and convey them as custom Slack messages called <em>download_link_quality </em>and <em>upload_link_quality</em>.</p><p>Now we’ll show you how to analyze the RTCP RR packets that WebRTC receivers send back and use it to figure out the quality of each leg of the call.</p><h3>Analyzing RTCP RR Packets</h3><p>According to the <a href="https://www.ietf.org/rfc/rfc3550.txt">RFC</a>, the RTCP RR packet looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*lhcQR593xsd1CXIXjkqwrg.png" /></figure><p>For simplicity, we assume that all RTCP-RR packets only have a single report block within them (called non-compound RTCP). Generally, they can have many of these RR Report blocks (for multiple streams) but for this example we assume there is only one. Also we assume you are familiar with packet parsing and that code is removed for brevity. To learn more about RTP/RTCP parsing check out section A-2 of the <a href="https://www.ietf.org/rfc/rfc3550.txt">RFC</a>.</p><p>The important part here is “fraction lost” which is a ratio (out of 255) of packets lost divided by the number of packets transmitted in this period.</p><p>Okay enough talk! Time to see some code.</p><h3>Download loss</h3><p>To calculate the actual number lost we need to know how many packets the receiver has seen so far. The <em>ehsnr</em> (Extended high sequence number) holds the 32-bit sequence number which the receiver has last seen before this report was sent.</p><pre>uint32_t packets_received = ehsnr — receiver-&gt;last_reported_seq_num;</pre><pre>receiver-&gt;last_reported_seq_num = ehsnr;</pre><pre>double packets_seemed_missing = (double)packets_received *<br>    fraction_lost / 255;</pre><p>The above gets us the actual number of missing packets. We must remember that we have just counted the <em>total amount </em>that the receiver has seen missing. It’s unclear whether this is due to <em>download</em> loss on the receiver’s side or due to the sender’s <em>upload</em> loss. Since we maintain the sequence number spacing from the original sender’s stream, the loss reported here is total loss (sender’s upload loss + receiver’s download loss). We must go deeper.</p><p>We calculate <em>pkts_missed_due_to_upload_loss_since_last_rr</em> by calculating how many sequence numbers have been missed on the upload side of this stream (code not shown for simplicity).</p><pre>double loss = packets_seemed_missing — <br>   sender-&gt;pkts_missed_due_to_upload_loss_since_last_rr;</pre><pre>double realistic_loss = 0;</pre><pre>if(loss &gt; 0)<br>   realistic_loss = loss;</pre><pre>slack_download_loss = realistic_loss / packets_received * 255.0f;</pre><p>For download loss &gt; 20% we would transmit an event to the receiver which would make their UI give the appropriate warning. As an example if Faraz is calling Pavel and is suffering from download loss &gt; 20% he’d see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*ElyOZgc0T2quqiLezJ1elw.png" /><figcaption>Faraz’s side indicating to him that his download is bad.</figcaption></figure><p>This state is broadcasted to all other users so everyone knows that Faraz’s download is bad and he might not be able to understand what is being said. So during this call Pavel’s side would correctly say:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/313/0*KfTey4hIX0N_2lK9." /><figcaption><em>Pavel’s side with the correct warning</em></figcaption></figure><p>Why 20%? For two reasons. Firstly the audio codec WebRTC uses (Opus) is able to deal with packet loss &lt; 20% elegantly so people may not even notice it. Opus does this using forward error correction (FEC) which is an ingenious way of encoding information about the previous packets into current ones (so they can be recovered if lost). <a href="http://www.ietf.org/proceedings/80/slides/codec-4.pdf">There’s evidence</a> that indicates that user perception of audio quality does not degrade significantly at loss levels of up to 20%.</p><h3>Upload loss</h3><p>This is the easy one and is done without using RTCP. Every RTP packet that you receive has a sequence number. We simply maintain a list of all sequence numbers received to date. Every 500ms, we run a loop that calculates <em>fraction loss</em> as:</p><pre>int num_packets = g_list_length(sender-&gt;seq_list_since_last_rr);</pre><pre>uint16_t exp_since_last = (packet-&gt;seq_number — <br>    sender-&gt;seq_number_at_last_rr);</pre><pre>int missing = exp_since_last — num_packets;</pre><pre>upload_frac_lost = (uint8_t)(256.0f * missing / exp_since_last);</pre><p>As with download loss, this is communicated to the sender and warns him appropriately above 20%. With the same call if Faraz is suffering upload instead of download loss this time around Pavel would see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*dLCsl-lVBlhuC4J6mZyGxA@2x.png" /><figcaption>Pavel sees me as having upload problems on his side</figcaption></figure><h3>So how much does all this help?</h3><p>Though we do slightly more with the fraction loss we have calculated here (like pack it into a RTCP RR message back to the sender to increase FEC) sometimes there is little you can do to help. So we do the best we can and put up <em>accurate</em> UI warnings alerting users as to why they are experiencing the problem. We have found that this helps users understand the problem instead of “Having network issues” which would leave them more in the dark.</p><p>Combating the variability of the internet is a never-ending battle in real-time communication. We can’t guarantee perfect audio quality every time, but hopefully with changes such as these it makes your call experience a bit more pleasant, and your conversations more productive.</p><h4>Learning more</h4><p>We could not have built Slack Calls without amazing open source software, standardized protocols, and freely available research. Here are some resources to learn more about the technologies mentioned in this post:</p><ul><li><a href="https://webrtc.org/">WebRTC</a></li><li><a href="https://janus.conf.meetecho.com/">Janus</a></li><li><a href="https://www.ietf.org/rfc/rfc3550.txt">RTCP/RTP</a></li></ul><p><em>We’re working hard to build a good team here at Slack, and, as you can see, have many big, interesting challenges! If they sound exciting to you, head on over to </em><a href="https://slack.com/jobs/dept/engineering"><em>https://slack.com/jobs/dept/engineering</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5d36749e8ed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/several-people-are-coding/calls-is-it-you-or-is-it-me-f5d36749e8ed">Calls: Is it you or is it me?</a> was originally published in <a href="https://medium.com/several-people-are-coding">Several People Are Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>